[WIP] dat/cam/sim/dsd/zon/val tool preview

[skwasjer]

Alright. Here and there I've been posting some comments and every so often a screenshot of what I'm working on. One of the bigger problems in modding today as you know, is working with the native game files. There are some tools out there that help right now, but I wanted to go further.

My aim is to provide you with a tool that will allow you to edit all files that are based on the dat-structure (both SH3 and SH4). I've taken all the knowledge that others have gathered, and debugged the file format myself as well for countless hours, and am trying to create a tool that is easy to use, and combines features of several other tools, and more... Especially, the 'more' part.

While I realise the tool is similar to what has been done before (aka Pack3D, mini tweaker, etc), I do hope to add something new. Below is a list of some (not all) things I either have finished or have planned.

Supported:

Full dat support. My file reader/writer reads 100% (!) of all SH3/4 files, without errors, and it can also write them back without problems. This doesn't mean I also know what every single byte means, but the reader/writer doesn't mind.
Also, for characteristics and object properties, I've got a flexible parser which is not based on file offsets. So, no more headaches with creating tweak files. The only thing that sometimes must be predefined is the datatype of a specific property. In most cases, it will even detect it. This makes it patch proof, mod proof, you name it, it will always read any file and write it back...
It reads the biggest files (interiors, 33MB) on my 3 year old pc in about 8-9 seconds! Small files are as fast as you can blink... (cheesy, i know :p)
Rich GUI, which allows you to edit all the data that is known to me or anyone else (meaning, if you got info, share, so I can embed it). Not everything is built in yet, but I'm working on it.
Multiple files can be opened (it will open in a new window). This is to support copy/paste and drag&drop in the future.
Two types of views for chunks: List, Nested. I want to add another view (grouped), but this is for later.
Options for editing id's, offsets in decimal/hex, rotations in radians/degrees, etc.
Import/export of tga's, dds's.
Import/export of a raw chunk. Can be used to move chunks over to other files.
Preview function for tga's and dds's.
Explorer style navigable links for id's. Click an id, if the id exists in the file, it will navigate to it.
Explorer style history of visited nodes. Back/forward buttons allow you to track back to previously selected nodes.
Rebuild the index chunk with all id's in the file.
List of all id's in the file (by selecting the index chunk). Id's and offsets can be copied to clipboard. Later, maybe even in all the game folders.
Support for placement chunk (cities/harbors)
Move chunk up/down in list. Only one chunk at a time.
Revert changes option (entire reload, not undo/redo)
Recent file list
One (easy) easteregg... First to post a screenie wins...Planned or work in progress:

Import/export of 3D models (with multiple channels, normals, materials).
Preview for 3D models.
Property editor for all object characteristics/properties (the property parser is done, but the UI is still a todo)
Cloning, Copy&paste
Add/insert new chunk planned.
Support for the more exotic chunks (sound, animation, etc), but this will be for later... Nevertheless, the file will be read successfully, even if the type of chunk isn't known.It may sound like a lot of work (it is actually), but I'm enjoying it. If you've got questions or feature requests, please post them.

Release date: soon, probably beginning of august. This will be an ALPHA!

Source code: I may release it eventually when I feel like it. Thing is, it is partially based on corporate code (my own company), so I may need to strip it a bit by then.

[edit] screenshots removed

[tater]

Wow.

[WernerSobe]

very good idea cant wait to try it.

Im sure it will work. When you look at these files they all start with a message "Created and Modified using Kashmir (c) by Ubisoft."

Looks like Ubisoft uses their own editor called Kashmir and all that files can be modified by that. So far ive been modifying all my mods manualy using hexeditor and floating point converter. I can tell you id realy want to have such tool in hands ^^.

[CaptainCox]

This is BIG! MAN!:up:
Is it sure you don't have a copy of the SDK or maybe wrote the dam thing :p

Will you be able to import/export TMAP?...looks like it, but maybe you could elaborate a bit on it.

[skwasjer]

Is it sure you don't have a copy of the SDK or maybe wrote the dam thing :p

Will you be able to import/export TMAP?...looks like it, but maybe you could elaborate a bit on it.

No SDK here :) In all seriousness, I've learned also from what others have done. I just took it a step further, and discovered more interesting data. That and the fact that I have access to resources (as a pro-developer) enables me to whip it up in a nice interface.

And yes, you'll be able to import/export TMAP data. If you look at the "Model" screenshot, you'll see the model I selected contains two texture channels. The only problem here is still that there aren't many 3D file formats that support multiple UV sets, so I'll probably resort to exporting a model multiple times, each time with a different UV set. There are some formats, but import filters for 3D suites are limited or non-existend for those formats. The obj format is a no go in that respect. This is still in research, but I'm making some progress here...

My aim now, is to get the tool ready, at least as a read/analyze tool, and for modifying basic data. Then, with subsequent updates, the goodies get added... :)

[CaptainCox]

Brilliant man ABSOLUTLY BRILLIANT!. This will for sure be a big breakthrough in modding SH3/4. Keep her steady as she goes!

[Anvart]

:up: :up: :up:
I would like, that you have made a correct hierarchical tree of the interconnected objects for the best understanding of communications between objects ...
Example:
pic.1. Druck_LM1 is root object (Type4, name = Type8), and Druck_LM1 is material (Type2, name = Type8) and cannot be root node ... But i wish to see names Type4 & Type2 ...
Common_Druck.tga is base bitmap for Druck_LM1 material ...
Gato_CR_LM1_LightMap.tga is lightmap for Druck_LM1 (submaterial Type13)...
...
And terminology... for example Controller (Type 10), Controller Parameters (Type 6)...
Offset must to be decimal ...
IDs (always in hex) must to be without "0x" ...
And many other things ...

[Bando]

Great stuff. Hope you'll be able to upload soon. Looks good. Bando

[Seeadler]

Good job:up:

Is it possible to read also the harbour.dat files in a correct way so that one can change the harbour model tiles with new models?

[skwasjer]

I would like, that you have made a correct hierarchical tree of the interconnected objects for the best understanding of communications between objects ...
Example:
pic.1. Druck_LM1 is material (name of material - Type8) and cannot be root node ...
Common_Druck.tga is base bitmap for Druck_LM1 material ...
Gato_CR_LM1_LightMap.tga is lightmap for Druck_LM1 ...
...
And terminology... for example Controller (Type 10), Controller Parameters (Type 6)...
And many other things ...

The tree is as correct as it can be.

I won't put materials under the nodes they are used by, because a material is often used by more nodes on the same level. I want to keep the tree as clean as possible, and not have a material added to multiple parents... I've tried your suggestion a couple of weeks ago, and when viewing for instance the Gato interior, this became a huge mess.
There are other ways to show relationships (and I have some things planned), like navigable id's. So, for nodes that reference materials, you'll get a list of id's on the right that you can click. You'll have to trust me on this for now and see for yourself when I release it.

As far as terminology, this is debatable. I think some terminology used in the community was bad, and decided to use some new. 'Controller' is a bit non-descriptive if you ask me, but that's just me. I don't know, maybe if others complain to me as well ;)


Good job:up:

Is it possible to read also the harbour.dat files in a correct way so that one can change the harbour model tiles with new models?


Harbor_kit.dat is read without problems... Also, all files under \Terrain\Locations are read, and chunk type 11 (Placement) is editable.

[sergbuto]

Good to hear. It is about time someone takes on the project like this. :up:

I agree with Anvart. Having the tree similar to that in the Datconvert tool would be quite convenient.

[Webster]

thank you for this, it is a wonderful tool it looks like.

i will ask now that you write the instructions for absolute nubs and dummies like me who cant follow instructions without getting lost.


since it will not be needed anymore, i will stop converting all of the sh3 particles files to sh4, only did 10 so far anyway. if anyone wants them, use my filefront link.

[skwasjer]

Good to hear. It is about time someone takes on the project like this. :up:

I agree with Anvart. Having the tree similar to that in the Datconvert tool would be quite convenient.
The tool you are referring to, is only showing nodes (chunk 4) and not materials. My tool shows 'every' chunk.

As far as the tree itself, best wait for the release and see how logical/natural my implementation is...

[denis_469]

Wow! Excellent mod would be:up: :yep:

[Seeadler]

Harbor_kit.dat is read without problems... Also, all files under \Terrain\Locations are read, and chunk type 11 (Placement) is editable.
nicely, then there is a chance that I can reopen my historical harbour project which I laid down long ago:smug:

[Uber Gruber]

@skwasjer (http://www.subsim.com/radioroom/member.php?u=230811)

Are you develping this using eclipse ? Just curious...

[skwasjer]

@skwasjer (http://www.subsim.com/radioroom/member.php?u=230811)

Are you develping this using eclipse ? Just curious...
Nope, it is coded in C# using .NET 2. Nevertheless, once source is released, it is very good possible to port it to java. Also, the file reader/writer comes in a single stand alone binary (as will other file formats, like 3DS/ASE) which can also be used for other projects if one chooses to do so...


i will ask now that you write the instructions for absolute nubs and dummies like me who cant follow instructions without getting lost.


since it will not be needed anymore, i will stop converting all of the sh3 particles files to sh4, only did 10 so far anyway. if anyone wants them, use my filefront link.

A manual surely is a must. There are already some do's and dont's that I know of. But I also believe that I alone can't write a proper manual. I prefer some help with this. I'm by no means a hardcode modder, just a humble coder, so we need some guys to step up and help out with this.

As far as editting properties, this may not make first release, so it can still take weeks before you can use my tool to edit them. That said, if you want to mod particles today, you still need to hexedit them...


Thanks for the positive notes guys. This community is giving me warm feelings :sunny:

[Anvart]

The tree is as correct as it can be.

I won't put materials under the nodes they are used by, because
...
As far as terminology, this is debatable. I think some terminology used in the community was bad, and decided to use some new. 'Controller' is a bit non-descriptive if you ask me, but that's just me. I don't know, maybe if others complain to me as well ;)
...

:D
1) The treelike hierarchy of objects is the most convenient kind of data display in SH4. For the best readership you can divide a tree into some levels (steps, screens, tabs...)...
2) Terminology is not discussed, it is DevsTeam terminology.
Look strings in SHControllers.act, SHSim.act and other ...

[Anvart]

Wow! Excellent mod would be:up: :yep:
It will be the tool ...

[skwasjer]

:D
1) The treelike hierarchy of objects is the most convenient kind of data display in SH4. For the best readership you can divide a tree into some levels (steps, screens, tabs...)...
2) Terminology is not discussed, it is DevsTeam terminology.
Look strings in SHControllers.act, SHSim.act and other ...

I think you should look again at my first pic. It includes a full tree. Only I kept materials on root level because they are only referenced chunks. They don't actually have a parent, it is the other way around. Same for 3D models, however, these never are referenced by more than one (the node link), so I decided to put it under the node that references it.

All other screenshots use my other view mode 'List' which is basically a list of 'order-of-appearance', no tree. Then there's even a 3rd view (no screenshot), which basically groups all types in seperate folders...

In fact, the tree you see in the first pic is better than any other tool that shows a tree of the dat-structure.

To give you a better idea of my other approach using navigable id's, I've created another screenshot:

http://sh4.bodeman.nl/ge/t/Material.jpg (http://sh4.bodeman.nl/ge/Material.jpg)

To sum up, I understand what you mean (I guess), but I think this is the best approach for now...

[haegemon]

Your job is the answer to our prays. :up:

Will you pretend to be posible to add new meshes and nodes inside *.dat?
I'll explain a little , in SH4 every plane had a predefined nodes for its load. In the stockgame itself devs made a mess, coding extense loadout cfgs while dat files are limited to few bombs in few pillons. So the planes should be reworked from scratch to finally use the famous airdepthcharges.

Anyway I give you a 10 for trying to make a step in the developement of "the tool". Nobody has worked such effort yet. And give you my vote too, to make this thread sticky :rock:

[tater]

The ability to add or fix nodes... wow, that would be cool.

[skwasjer]

Will you pretend to be posible to add new meshes and nodes inside *.dat?

Yes, to a certain extend. At least some types of nodes, that are fully known. Otherwise, it will be cloning which is the safe approach, and is quite easy for me to do actually. I may even include automatic id generation (to prevent duplicates/collisions).

The ability to add or fix nodes...
Depends. If a chunk is broken in a sense that it breaks parsing rules, then you're in no luck probably. Of course, you can always try to open it. Adding new nodes, see above.

[sergbuto]

Good to hear. It is about time someone takes on the project like this. :up:

I agree with Anvart. Having the tree similar to that in the Datconvert tool would be quite convenient.
The tool you are referring to, is only showing nodes (chunk 4) and not materials. My tool shows 'every' chunk.
For those nodes the tool shows, it does the best job so far. To get an overview of the 3D model and make an initial test after the model modification, I still prefer to use Datconvert, not Pack3D.

Naturally, it is up to you as an author to decide about the tree order, I am just speaking about my experiences as an user of different tools for SH3/SH4.

[CaptainCox]

@sergbuto where would one find Datconvert, never heard of it.

Cheers!

[Anvart]

@sergbuto where would one find Datconvert, never heard of it.

Cheers!
Mike, you can to download datconvert.zip and datconvertsrc.zip (source code) from my FileFront page ...
But Pack3D has greater functionality ...

[Anvart]

:D
1) The treelike hierarchy of objects is the most convenient kind of data display in SH4. For the best readership you can divide a tree into some levels (steps, screens, tabs...)...
2) Terminology is not discussed, it is DevsTeam terminology.
Look strings in SHControllers.act, SHSim.act and other ...

I think you should look again at my first pic. ...
I think this is the best approach for now...
I have tried to write my opinion, but you can do that you want...

[haegemon]

We better wait to see the results. If the tool is as powerful as expected, then I think the way how it shows data has little importance. ;)

[skwasjer]

Anvart, your opinion is appreciated. But it's fair to have different opinions too. Just wait and see, I'm sure you'll like it when you can actually see/try for yourself. :)

[Anvart]

Anvart, your opinion is appreciated. But it's fair to have different opinions too. Just wait and see, I'm sure you'll like it when you can actually see/try for yourself. :)
I look forward ...:up:

[sergbuto]

@sergbuto where would one find Datconvert, never heard of it.

Cheers!
This program was made long time ago and it was a great help and improvement from pure hexediting at that time. It is good to find out that it can be downloaded from Anvart's site.

However, I do not mind if the upcoming tool eliminates the need to use Datconvert. ;)

[CaptainCox]

Cheers, Anvart actually PMed me shortly after I posted. Have not had the time yet to look at it :p. Will do so ASAP.

Thanks!

[Webster]

ok guys back to the original post here, when is this tool going to be ready for downloading?

patch 1.3 is out and its time to start modding.

[skwasjer]

Probably two weeks time. Maybe sooner, maybe later... A bit vague, but I want to make sure some features make first release...

[Rockin Robbins]

You guys are all amazing! Please refrain from killing each other until your astounding mods made possible by this opaque (to me) wizardry are published.:rotfl:

[skwasjer]

I've been a bit delayed. I just got a brand new pc a week ago, so I'm still in progress of installation of all my apps. Additionally, I've been playing SH4 on this new rig, to enjoy the ubergraphics (my previous pc couldn't handle it much). I happened to run into a TF with the Yamato by chance. And she sank... :)

On topic: I got another feature idea that can be usefull (by reading the Raise/lower periscope mod/fix topic):
- Export and import support for an entire chunk to/from a raw file. This way, you can export working chunks from older mods or even SH3, and insert them anywhere in new/other files anywhere in the structure. Those chunks remain unmodified (until you decide to edit them of course). This will be of great use when updating older mods, or when duplicating dat-files for new game content.

Anyway, hopefully alpha release around Aug. 1st.

[Lagger123987]

Nice to hear, looking good so far!

[Bando]

August 1st?

Good news indeed.:up:
I'll be waiting Skwasjer.
Great news:up:

[Anvart]

...
On topic: I got another feature idea that can be usefull (by reading the Raise/lower periscope mod/fix topic):
- Export and import support for an entire chunk to/from a raw file. This way, you can export working chunks from older mods or even SH3, and insert them anywhere in new/other files anywhere in the structure. Those chunks remain unmodified (until you decide to edit them of course). This will be of great use when updating older mods, or when duplicating dat-files for new game content.

Anyway, hopefully alpha release around Aug. 1st.
!!!
Yes, necessarily!

[skwasjer]

I'll update the first post later today with new status/screenshots...

For now, I need help with two things:

A name for the tool. One that's easy to remember and googleable (iow, not too many hits). As far as I'm concerned, you all decide on the name, it don't really matter to me. Just spam ideas, and vote what you like...
A logo/icon for it (must somewhat depend on previous bullet). Talented artists are welcome to hop in. I'm a complete tard when it comes to creating artwork. Icons preferred: 96x96, 48x48, 32x32 and 16x16, all in 32 and 16-bit color. Not allowed: modified SH icons.Please post or PM if you want to help out and end up in the credits :)

[Taurolas]

skwasjer.

Dude if you manage to pull this off and it looks like you pretty much have that aspect of it in the bag, i'd say that this would have to be one of the greatest single applications for helping modding or file tweaking within the SH game series due to it encompassing several different formats.

i'd also like to pay my respects to all you modders and Apps guys who have taken the time to learn understand and fix the problems that have dogged SH3 / SH4 from day 1 without you i think the SH series would have been doomed from the start.

you guys make these games what they are today.

[tater]

Silenthunterman like this:
http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/talks/2004-01-techlinks/leatherman-flair.jpg

:)

That or somethign can opener or swiss army knife related.

SH4 Multitool?

tater

[skwasjer]

SH4 Multitool?
Hehe, good idea :)

Funny things went through my mind the past weeks... One of them:

sh|die

| = pipeline = is a programming symbol for 'or'

So it would read 'sh or die', where die = dat importer exporter. Not so googleable though... I'm such a tard, haha:arrgh!::sunny:


PS: it's not limited to SH4 ;)

[skwasjer]

skwasjer.

Dude if you manage to pull this off and it looks like you pretty much have that aspect of it in the bag, i'd say that this would have to be one of the greatest single applications for helping modding or file tweaking within the SH game series due to it encompassing several different formats.

i'd also like to pay my respects to all you modders and Apps guys who have taken the time to learn understand and fix the problems that have dogged SH3 / SH4 from day 1 without you i think the SH series would have been doomed from the start.

you guys make these games what they are today.
Lets hope so :) I can only hope it will enable more and better mods to come out, if the modders don't need to dive in so deep anymore, and can use all their time to create more quality stuff :up:

[tater]

I'm stoked about this tool.

Some of the stuff I want to mess with is still a little arcane for me.

[CaptainCox]

Had a bit of time so...

http://img374.imageshack.us/img374/6184/ideaswj8.jpg

Will think about it a bit more...weekend and holiday coming up so :p

[Anvart]

SHreBuilder ...
SHrePacker ...
SHinSider ...
SHreAssembler ...
SHPrepare ...
SHAdjuster ...
SHManager ...
SHTweaker ...
SHTurnover ...
SHTurnscrew ...:lol:

[skwasjer]

Very cool guys, keep 'm coming...

I've already updated my first post with new specs/todo's. New screenshots will follow later...

PS: efforts count towards the creditlist (maybe this will help getting more attention!)

[Bando]

Gamod(der)
GameTool
GameSol(ution)
(The) Modder
HeavenlyEdit

[Webster]

I'll update the first post later today with new status/screenshots...

For now, I need help with two things:

A name for the tool. One that's easy to remember and googleable (iow, not too many hits). As far as I'm concerned, you all decide on the name, it don't really matter to me. Just spam ideas, and vote what you like...
A logo/icon for it (must somewhat depend on previous bullet). Talented artists are welcome to hop in. I'm a complete tard when it comes to creating artwork. Icons preferred: 96x96, 48x48, 32x32 and 16x16, all in 32 and 16-bit color. Not allowed: modified SH icons.Please post or PM if you want to help out and end up in the credits :)


well i like to keep things simple so they are easy to remember and any changes made later should not make the name not fit the same later on.

my suggestion is: "Skwasjer's Mod Tool" or "Skwasjer's 3D Mod Tool" or maybe "Skwasjer's SH4 Mod Tool"

and for the logo, i like the look and style of Captain Cox's first icon top left of picture. something in that look would fit very well, although orange and grey would not be my favorite color choice.

EDIT: if you want to call it "porter" then please call it "teleporter" instead, it just sounds better than "porter". but maybe just because i am startrek fan lol.

[maikarant]

I have a vision... ;)

I suddenly saw it right before my eyes. The logo - a Popeyesque lower arm tattoed with a stylized can-opener. The name:

Das Tool

(in honor of U96.) ;) And since I'm not sufficiently bushy-tailed while using Paint.Net or something in the middle of the night, I'll leave it right there, hehe.

[tater]

http://i8.photobucket.com/albums/a50/passion2sail/Issac.jpg
MODTASTIC!

[Jace11]

SHIVTOOL

[skwasjer]

Proud to show you this one screeny:

Click me (http://sh4.skwas.net/PropEditor.jpg)

As you can see work is progressing on the editor for properties. In the alpha release, you will only get a view mode only. It will be improved though compared to this screenshot. I will be so kind to give you exact file offsets of each property (as you select it), so while waiting for a full read/write editor (which will still take a couple of weeks extra), you can just find the damn offset in your hex editor. Good enough for now, no?

[Bando]

Awesome.
Can't wait to see the version where one can edit and save.
This looks very very promising
Thanks for the peek:up:

[DivingDuck]

Moin skwas,

good to see you´re proceeding with your tool. I was sidetracked the last weeks by my buoy mod and didn´t keep track. Let´s hope you´ll finish it soon. You know, I´m one of those who would really need your tool.

Regards,
DD

[Anachronous]

Wow thats great. Look forward to giving it a run.

[WilhelmTell]

SHBuilder
TheBuilder
SHEase
EaseSH4
SHPreparateur

[kapitan_zur_see]

Great project!
This is going to be top notch!
Keep up this very good work :up::up:

[CaptainCox]

Some new graphic ideas based on the name
NODE 3D
SH NODE
SH NODE 3D
http://img96.imageshack.us/img96/6452/prop4ed7.jpg

http://img525.imageshack.us/img525/5435/prop5ry4.jpg

http://img224.imageshack.us/img224/1244/prop2cl4.jpg

http://img96.imageshack.us/img96/8894/prop1mv1.jpg

http://img53.imageshack.us/img53/8097/prop3yi3.jpg

Colors and stuff can of course be what ever, just like the black, orange, gray at the mo :p

[Mush Martin]

I like the first one best CC :up:

[skwasjer]

Thanks all, I've decided on a name. I will not tell yet though ;)

Thanks for the efforts, it is noted...

[corleonedk]

Holy crap its gonna be a big steep forward for the moders awesome work:up:

[donut]

Holy crap its gonna be a big steep forward for the moders awesome work:up:
The talent will have a tool,that may make SHIV all it can be.:sunny:

[Jace11]

Nylon
Polyester


get it...?

[skwasjer]

:) release is immenent. I'm packaging everything atm.

Note again, it's gonna be an alpha. By no means do I recommend using this for any public mod. It's not a question whether you will find a bug, but when and which one (the choice is yours... :88)) Nevertheless, I want to put it out now, so I can start gathering feedback and bugreports. This baby is already 850 KB or 33000 lines in code source files, and growing, and so it will have errors ;)

I'll get back here asap (cpl hours)

[longam]

post removed by me