[WIP] dat/cam/sim/dsd/zon/val tool preview

[skwasjer]

Alright. Here and there I've been posting some comments and every so often a screenshot of what I'm working on. One of the bigger problems in modding today as you know, is working with the native game files. There are some tools out there that help right now, but I wanted to go further.

My aim is to provide you with a tool that will allow you to edit all files that are based on the dat-structure (both SH3 and SH4). I've taken all the knowledge that others have gathered, and debugged the file format myself as well for countless hours, and am trying to create a tool that is easy to use, and combines features of several other tools, and more... Especially, the 'more' part.

While I realise the tool is similar to what has been done before (aka Pack3D, mini tweaker, etc), I do hope to add something new. Below is a list of some (not all) things I either have finished or have planned.

Supported:

• Full dat support. My file reader/writer reads 100% (!) of all SH3/4 files, without errors, and it can also write them back without problems. This doesn't mean I also know what every single byte means, but the reader/writer doesn't mind.
  Also, for characteristics and object properties, I've got a flexible parser which is not based on file offsets. So, no more headaches with creating tweak files. The only thing that sometimes must be predefined is the datatype of a specific property. In most cases, it will even detect it. This makes it patch proof, mod proof, you name it, it will always read any file and write it back...
  It reads the biggest files (interiors, 33MB) on my 3 year old pc in about 8-9 seconds! Small files are as fast as you can blink... (cheesy, i know :p)
• Rich GUI, which allows you to edit all the data that is known to me or anyone else (meaning, if you got info, share, so I can embed it). Not everything is built in yet, but I'm working on it.
• Multiple files can be opened (it will open in a new window). This is to support copy/paste and drag&drop in the future.
• Two types of views for chunks: List, Nested. I want to add another view (grouped), but this is for later.
• Options for editing id's, offsets in decimal/hex, rotations in radians/degrees, etc.
• Import/export of tga's, dds's.
• Import/export of a raw chunk. Can be used to move chunks over to other files.
• Preview function for tga's and dds's.
• Explorer style navigable links for id's. Click an id, if the id exists in the file, it will navigate to it.
• Explorer style history of visited nodes. Back/forward buttons allow you to track back to previously selected nodes.
• Rebuild the index chunk with all id's in the file.
• List of all id's in the file (by selecting the index chunk). Id's and offsets can be copied to clipboard. Later, maybe even in all the game folders.
• Support for placement chunk (cities/harbors)
• Move chunk up/down in list. Only one chunk at a time.
• Revert changes option (entire reload, not undo/redo)
• Recent file list
• One (easy) easteregg... First to post a screenie wins...

Planned or work in progress:

• Import/export of 3D models (with multiple channels, normals, materials).
• Preview for 3D models.
• Property editor for all object characteristics/properties (the property parser is done, but the UI is still a todo)
• Cloning, Copy&paste
• Add/insert new chunk planned.
• Support for the more exotic chunks (sound, animation, etc), but this will be for later... Nevertheless, the file will be read successfully, even if the type of chunk isn't known.

It may sound like a lot of work (it is actually), but I'm enjoying it. If you've got questions or feature requests, please post them.

Release date: soon, probably beginning of august. This will be an ALPHA!

Source code: I may release it eventually when I feel like it. Thing is, it is partially based on corporate code (my own company), so I may need to strip it a bit by then.

[edit] screenshots removed

[tater]

Wow.

[WernerSobe]

very good idea cant wait to try it.

Im sure it will work. When you look at these files they all start with a message "Created and Modified using Kashmir (c) by Ubisoft."

Looks like Ubisoft uses their own editor called Kashmir and all that files can be modified by that. So far ive been modifying all my mods manualy using hexeditor and floating point converter. I can tell you id realy want to have such tool in hands ^^.

[CaptainCox]

This is BIG! MAN!
Is it sure you don't have a copy of the SDK or maybe wrote the dam thing :p

Will you be able to import/export TMAP?...looks like it, but maybe you could elaborate a bit on it.

[skwasjer]

Originally Posted by CaptainCox

Is it sure you don't have a copy of the SDK or maybe wrote the dam thing :p

Will you be able to import/export TMAP?...looks like it, but maybe you could elaborate a bit on it.

No SDK here In all seriousness, I've learned also from what others have done. I just took it a step further, and discovered more interesting data. That and the fact that I have access to resources (as a pro-developer) enables me to whip it up in a nice interface.

And yes, you'll be able to import/export TMAP data. If you look at the "Model" screenshot, you'll see the model I selected contains two texture channels. The only problem here is still that there aren't many 3D file formats that support multiple UV sets, so I'll probably resort to exporting a model multiple times, each time with a different UV set. There are some formats, but import filters for 3D suites are limited or non-existend for those formats. The obj format is a no go in that respect. This is still in research, but I'm making some progress here...

My aim now, is to get the tool ready, at least as a read/analyze tool, and for modifying basic data. Then, with subsequent updates, the goodies get added...

[CaptainCox]

Brilliant man ABSOLUTLY BRILLIANT!. This will for sure be a big breakthrough in modding SH3/4. Keep her steady as she goes!

[Anvart]

I would like, that you have made a correct hierarchical tree of the interconnected objects for the best understanding of communications between objects ...
Example:
pic.1. Druck_LM1 is root object (Type4, name = Type8), and Druck_LM1 is material (Type2, name = Type8) and cannot be root node ... But i wish to see names Type4 & Type2 ...
Common_Druck.tga is base bitmap for Druck_LM1 material ...
Gato_CR_LM1_LightMap.tga is lightmap for Druck_LM1 (submaterial Type13)...
...
And terminology... for example Controller (Type 10), Controller Parameters (Type 6)...
Offset must to be decimal ...
IDs (always in hex) must to be without "0x" ...
And many other things ...

[Bando]

Great stuff. Hope you'll be able to upload soon. Looks good. Bando

[Seeadler]

Good job

Is it possible to read also the harbour.dat files in a correct way so that one can change the harbour model tiles with new models?

[skwasjer]

Originally Posted by Anvart

I would like, that you have made a correct hierarchical tree of the interconnected objects for the best understanding of communications between objects ...
Example:
pic.1. Druck_LM1 is material (name of material - Type8) and cannot be root node ...
Common_Druck.tga is base bitmap for Druck_LM1 material ...
Gato_CR_LM1_LightMap.tga is lightmap for Druck_LM1 ...
...
And terminology... for example Controller (Type 10), Controller Parameters (Type 6)...
And many other things ...

The tree is as correct as it can be.

I won't put materials under the nodes they are used by, because a material is often used by more nodes on the same level. I want to keep the tree as clean as possible, and not have a material added to multiple parents... I've tried your suggestion a couple of weeks ago, and when viewing for instance the Gato interior, this became a huge mess.
There are other ways to show relationships (and I have some things planned), like navigable id's. So, for nodes that reference materials, you'll get a list of id's on the right that you can click. You'll have to trust me on this for now and see for yourself when I release it.

As far as terminology, this is debatable. I think some terminology used in the community was bad, and decided to use some new. 'Controller' is a bit non-descriptive if you ask me, but that's just me. I don't know, maybe if others complain to me as well

Originally Posted by Seeadler

Good job

Is it possible to read also the harbour.dat files in a correct way so that one can change the harbour model tiles with new models?

Harbor_kit.dat is read without problems... Also, all files under \Terrain\Locations are read, and chunk type 11 (Placement) is editable.

[sergbuto]

Good to hear. It is about time someone takes on the project like this.

I agree with Anvart. Having the tree similar to that in the Datconvert tool would be quite convenient.

thank you for this, it is a wonderful tool it looks like.

i will ask now that you write the instructions for absolute nubs and dummies like me who cant follow instructions without getting lost.


since it will not be needed anymore, i will stop converting all of the sh3 particles files to sh4, only did 10 so far anyway. if anyone wants them, use my filefront link.

[skwasjer]

Originally Posted by sergbuto

Good to hear. It is about time someone takes on the project like this.

I agree with Anvart. Having the tree similar to that in the Datconvert tool would be quite convenient.

The tool you are referring to, is only showing nodes (chunk 4) and not materials. My tool shows 'every' chunk.

As far as the tree itself, best wait for the release and see how logical/natural my implementation is...

[denis_469]

Wow! Excellent mod would be

[Seeadler]

Originally Posted by skwasjer

Harbor_kit.dat is read without problems... Also, all files under \Terrain\Locations are read, and chunk type 11 (Placement) is editable.

nicely, then there is a chance that I can reopen my historical harbour project which I laid down long ago