about Crash, zero-sized 3d-objects, index and shadows in Dat-files

[Hebe Vollmaus]

Hi there,
for some reason i would like to stop modding and try to play again a Campaign from 1939 on to the end in 1945.
But my game does not allow this! :doh: In the middle of 1942 i stumbled over a Ship that crashes the game to desktop. Modding may be an endless-loop. LOL
I created a Training-Mission with that ship only to be shure the reason is this ship. Whenever i hit the target with my heavy-shells the game crashed. So with the help of "Silent 3ditor" (S3D) looked into its Dat-file. Found a bunch of zero-sized 3D-objects. Filled these 3Ds with something real. And the game nolonger crashes.
Really annoying, since the zero-sized 3D-objects allone should not be the problem, as there are some other files around which use these things for good. For example for fake-funnels or rudders.
I expanded mySH3-TooL to detect these Zero3Ds. On the way i found five Dat-files which S3D cannot open. Three of them had bad index-lenght. So added this to detect also indexes to my tool. What can i say... Almost all Dat-files in my SH3-install had bad indexes. Checked the original unmodified SH3 files - all are ok, and there are no zero-sized 3D-objects. As my game-install runs otherwise good, these indexes seems to be useless to the game.
For the long story short, i deleted all indexes i can find. Also allmost all Shadows (SHD, DynamicShadowCast, DynamicShadow), and useless (unfinished/forgotten) zero-sized 3D-object. The game looks the same, as it does not use DynamicShadows anyway.
SH3\data is a little more then 4GB. Saves little less then 60MB. Maybe not worth doing so. But now i have 3800 unused 3D-objects less.

Greetings.

[FUBAR295]

Hebe,


Interesting post. It may not help with the size but the loading time may be a bit faster, hopefully. There is a lot of junk and repetitive files taking up memory and adding to load times. I think Fifi and AlexOne have done a great job getting rid of unneeded stuff.


I use your tool, it's great! :up:


Good hunting,
FUBAR295

[Fifi]

Yes, tons of unnecessary or not working things everywhere in this game :yep:
Actually revising MFM ships, and had surprises with their 3D model… don’t know where they come from, but plenty weird things too :o

:Kaleun_Salute:

[Mister_M]

Hi there,
for some reason i would like to stop modding and try to play again a Campaign from 1939 on to the end in 1945.
But my game does not allow this! :doh: In the middle of 1942 i stumbled over a Ship that crashes the game to desktop. Modding may be an endless-loop. LOL
I created a Training-Mission with that ship only to be shure the reason is this ship. Whenever i hit the target with my heavy-shells the game crashed. So with the help of "Silent 3ditor" (S3D) looked into its Dat-file. Found a bunch of zero-sized 3D-objects. Filled these 3Ds with something real. And the game nolonger crashes.
Really annoying, since the zero-sized 3D-objects allone should not be the problem, as there are some other files around which use these things for good. For example for fake-funnels or rudders.
I expanded mySH3-TooL to detect these Zero3Ds. On the way i found five Dat-files which S3D cannot open. Three of them had bad index-lenght. So added this to detect also indexes to my tool. What can i say... Almost all Dat-files in my SH3-install had bad indexes. Checked the original unmodified SH3 files - all are ok, and there are no zero-sized 3D-objects. As my game-install runs otherwise good, these indexes seems to be useless to the game.
For the long story short, i deleted all indexes i can find. Also allmost all Shadows (SHD, DynamicShadowCast, DynamicShadow), and useless (unfinished/forgotten) zero-sized 3D-object. The game looks the same, as it does not use DynamicShadows anyway.
SH3\data is a little more then 4GB. Saves little less then 60MB. Maybe not worth doing so. But now i have 3800 unused 3D-objects less.

Greetings.

Hi Hebe Vollmaus,

1. Is it possible to automatically remove all the useless SHD nodes with your tool? :hmm2:

2. What is a "bad index"?


Yes, tons of unnecessary or not working things everywhere in this game :yep:
Actually revising MFM ships, and had surprises with their 3D model… don’t know where they come from, but plenty weird things too :o

Le Diable est dans les détails... (the Devil is in the detail)...

[propbeanie]

I think he is referring to the S3D index:

https://i.imgur.com/Xrfeh8t.jpg


As usual though, I could well be wrong... lol - Some of what Hebe found is from the hex-editing days. Silent3Ditor will not let you save a dat file without first updating the index, which it does automatically when you save, or you can initiate it yourself. S3D also does ID checks and will let you know if there are conflicts there. None of that functionality is available with hex-editing. That is up to the user. The empty 3D objects are also part of the hex-editing, as are a few other little "tricks" that hex-editing people employed to "mark" their files.

As for "automatically" removing such items, it is possible, but would involve some deep programming skills, which are hard to find these days in the "hobby" world. Hebe might be able to pull it off, but he has other, more important chores to do with his mySH3-Tool first... :arrgh!: - which is something every modder should have in their toolbox, and use on a regular basis...
:salute:

[Hebe Vollmaus]

Hi,
yes. Today we may be a very little bit amused. But 2007 was another world. No S3D. Only Hex-edit. And as for that i have high respect for these Modders in the old days.
The "Best of" i have seen until today is the "Air_br.dat". A landed Sunderland-Aircraft was made out of a small Torpedoboat. A lot of unseen Ballast were left in.

My tool is not able to remove these Nodes.
To do this i need to know all things what can be in a Dat-file, which i do not know. And as this functionality already exist in S3D i do not see, as of today, that i like to re-invent the wheel.
I think S3D can be "persuaded" to do this automatically, but only with a selfmade program that will telecommand S3D.

The index is what propbeanie has shown. But i would not call it "S3D index". Its a Dat-index.
I have two sorts of "Bad index". One are these with a wrong Lenght-entry in the index-Header. This will stop S3D from Loading. And a second with a Node (ID) within the index, that shows a wrong file-position. S3D always adds a comment into a Dat-file and therefore will be all internal Node-positions also re-positioned in this index.

S3D is very good. Now i have used a function that i have never used before: "Save in Mod". Very helpful! :yep:

Greetings.

[Kpt. Lehmann]

Good stuff, Hebe. Thank you.
We are continuously finding and getting rid of extraneous bits too.
It is nice to have more elements to look for. Every little bit counts. :up:

[Jeff-Groves]

Old information about dat structure from around 2005.
Don't recall who posted it back then and it's never been updated to current knowledge.



SH III DAT FILE FORMAT

Skip the first 20 bytes they seem to be always the same.
Next 4 bytes are a long with the header string lenght.
Next is the author name and the "using Kashmir" stuff, total lenght as read above.

Next, file is made of "chunks". All chunks are like this:
ChunkType 4 bytes
Chunk Subtype/Version 4 bytes
chunk_size long (4 bytes)
Data (chunk_size bytes)


Chunk types and internal structure of each one:

type 0: EMPTY chunk
it's just 12 "00" bytes, so type, subtype and size are 0

type 1: 3D chunk
after the Chunk type, subtype/ver and size, format of the data part is like follows:
chunk_id: 8 bytes //this is used to reference this part inside the file. References usually found inside NODE (type 4) chunks
???????: 1 byte //don't know about this one
nverts: 4 byte long //number of vertices
vertex data: nverts * 3 floats(X,Y,Z)// these are vertices, 3 floats per vertex
ntri: 4 byte long //number of triangles;
triangle data: there are ntri triangles and their texture mapping, each one is as follows
vertex1: unsigned 2-byte int
vertex2: unsigned 2-byte int
vertex3: unsigned 2-byte int
texture coord1: unsigned 2-byte int
texture coord2: unsigned 2-byte int
texture coord3: unsigned 2-byte int
????????????: one byte
ntxcoords: 4 byte long //number of texture coordinates
texture coords data: ntxcoords * 2 floats // texture coordinates are 2D

some objects end here, but some have "TMAP" and a bunch of floats after this, dunno what is that yet so I skip it

type 2: TEXTURE REFERENCE?
after the Chunk type, subtype/ver and size, format of the data part is like follows:
chunk_id: 8 bytes //this is used to reference this part inside the file. References usually found inside NODE (type 4) chunks
then a bunch of bytes that I haven't figured yet, usually ending with a texture filename.

type 3: TGA FILE
these are the TGA files, just packed after the type, subtype and chunk size.

type 4: NODE (these tell us how parts are related to each other, in a tree-like hierarchy)
after the Chunk type, subtype/ver and size, format of the data part is like follows:
chunk_id: 8 bytes //this is used to reference this node inside the file. References usually found inside NODE (type 4) chunks
parent_id: 8 bytes //reference to the "parent" node. root nodes have 00 00 00 00 00 00 00 00 as parent
3dpart_id: 8 bytes //reference to the 3d part managed by this node, 00 00 00 00 00 00 00 00 if this is a "dummy" node with no visible 3d part
????????: 1 byte, seems to be always 00.
????????: 1 byte, seems to be related to the number of different type 2 nodes referenced from this one.
x offset: 4 bytes float //x offset from parent node
y offset: 4 bytes float //y offset from parent node
z offset: 4 bytes float //z offset from parent node

then comes the type 2 chunk references and some stuff I don't yet understand

type 6: DATA?
these are children of type 10 chunks, they seem to hold effects, commands, interactions, etc...
first 8 bytes are the reference to the type 10 parent. rest is strings and numerical values.

type 8: LABEL
these are used to label other nodes
format is:
8 byte parent chunk reference
string with label

type 10: DATA?
these seem to group type 6 chunks and attach them to type 4 nodes
first 8 bytes: id of this chunk
second 8 bytes: id of parent node
then comes some data, usually strings and numbers.

type 102: SCALE?
these come usually near the end of the files, they are a bunch of floats, probably some global scaling, have to test it

type 1001: INDEX
this one seems to be an index, there are chunk ID's and the corresponding file postition, might have to update this if some chunk has its size altered, but some tries suggest this doesnñr affect loading.



There are more chunk types, haven't had time to check them

[Mister_M]

And what is the unknown node that we sometimes see at the end of a .dat file in S3D? :hmmm:

[Jeff-Groves]

That would be the Type 102 node.

No one that I know of has the exact answer to what it is.

As I stated, that is very old information I posted about the structure.

[Jeff-Groves]

Now, in a pretty much stock SH3 with GWX 3?
There's 177 files with SHD objects out of 189 folders.
Iceburgs don't have them as an example.

Finding EVERY zero sized 3D objects is gonna be a treat also with S3D.


So by using S3D you'd have to open every dat and find then hand delete things.
I'd do a script for 010 to search those files out myself.
Have fun!
:haha:

Oh. Out of those 177 files? There 3333 occurrences of SHD_

[Jeff-Groves]

If someone sends me a link to a Unit with a zero sized object?
I'll look at it and tell you how to find them with a quick search in 010
I'm not looking for them myself.

It's not like I didn't talk about this stuff years ago.
:nope:

Ya'll should be happy there's still someone that can do/knows things about Hex Editing files still around.

[Hebe Vollmaus]

Thanks for the information.
I also have saved this before, and some more tips from ref and Anvart years ago.
About the index he states: "Any indexation serves for increase
speeds of objects search in a file!!!".

Looks like i have to repair my broken indexes and see if the game loads faster.




as an example
GWX v3.0 NBC_Deutschland.dat
iD:191740028802A047 S3D:0x47a0028802401719 size:21 (0x15)

S3D says on Model info:
Vertex coordinates: 0 Number of triangles: 0
Vertex indexes: 0 Texture coordinates: 0
Vertex normals: No Texture sets/channels: 1

[Jeff-Groves]

I don't know if they load faster with a proper index. Never tested for that myself.

I can say with a proper index? I can program a search based on a given ID since the pointer is there.
I'd guess one could use Autoit and do a script for that to use S3D to open files from a list then save as that should rebuild the stock index.

I'll look at the NBC_Deutschland.dat and give you a quick way to find things like that. Now I only use Sweetscape 010 and script with that.
Give me the Node number.

ref and Anvart were a couple that taught me a lot also! DivingDuck was another!

[Jeff-Groves]

There are 86 zero type objects in 11 Sea Units in GWX 3.
This was a simple search and I did not write a script for it as yet.

Name of file and the position in the file.
To convert the Hex positions below use calculator in programmer mode.
(If I have to explain that? Your in over your head already!)
Open the dat, rebuild the Index, then find the number. Double click and you'll go to the 3D part.

NBC_Deutschland.dat 4CEA2Ah
NBC_Deutschland.dat 5C3745h
NBC_GrafSpee.dat 4CEA2Ah
NBC_GrafSpee.dat 5C3745h
NCA_Lutzow.dat 2AE8EEh
NCA_Lutzow.dat 2E1C6Ch
NCA_Lutzow.dat 2E1D0Ah
NCA_Lutzow.dat 2E84F1h
NCA_Lutzow.dat 2E858Dh
NCA_Lutzow.dat 2E86A3h
NCA_Lutzow.dat 2E873Fh
NCA_Lutzow.dat 3E0F2Fh
NCA_Lutzow.dat 3E1285h
NCA_Scheer.dat 4CE961h
NCA_Scheer.dat 5C367Ch
NCHT.dat F866Dh
NCHT.dat F8F30h
NCV_USSEssexCV9.dat 2807D4h
NCV_USSEssexCV9.dat 2811F4h
NCV_USSEssexCV9.dat 28128Fh
NCV_USSEssexCV9.dat 2813ACh
NCV_USSEssexCV9.dat 281447h
NCV_USSEssexCV9.dat 281564h
NCV_USSEssexCV9.dat 281607h
NCV_USSEssexCV9.dat 2816A4h
NCV_USSEssexCV9.dat 2817C1h
NCV_USSEssexCV9.dat 281864h
NCV_USSEssexCV9.dat 281981h
NCV_USSEssexCV9.dat 281A1Ch
NCV_USSEssexCV9.dat 281B39h
NCV_USSEssexCV9.dat 281BD4h
NCV_USSEssexCV9.dat 281CF1h
NCV_USSEssexCV9.dat 281D94h
NCV_USSEssexCV9.dat 281E31h
NDD_Type36A.dat 168619h
NDD_Type36A.dat 1686B6h
NDD_Type36A.dat 1687CDh
NDD_Type36A.dat 16886Ah
NDE_FTboot.dat 1C2484h
NDE_FTboot.dat 1C2521h
NGRA_.dat 15D465h
NGRA_.dat 15D892h
NKLK_.dat 1C242Ah
NKLK_.dat 1CCF9Fh
NKLK_.dat 1CD46Ch
NMAS.dat 4F1806h
NMAS.dat 4F1897h
NMAS.dat 4F19F9h
NMAS.dat 4F440Ah
NMAS.dat 4F44F8h
NMAS.dat 4F4662h
NMAS.dat 4F47D9h
NMAS.dat 4F4873h
NMAS.dat 4F4904h
NMAS.dat 4F4A17h
NMAS.dat 4F4AA7h
NMAS.dat 4F4C08h
NMAS.dat 4F4D1Ah
NMAS.dat 4F4DACh
NMAS.dat 4F4F0Fh
NMAS.dat 4F5023h
NMAS.dat 4F50B5h
NMAS.dat 4F5218h
NMAS.dat 4F532Ch
NMAS.dat 4F53BEh
NMAS.dat 4F5521h
NMAS.dat 4F5635h
NMAS.dat 4F56C7h
NMAS.dat 4F582Ah
NMAS.dat 4F593Eh
NMAS.dat 4F59D0h
NMAS.dat 4F5B33h
NMAS.dat 4FA6BDh
NMAS.dat 4FF09Ch
NMAS.dat 4FF1B2h
NMAS.dat 4FF246h
NMAS.dat 4FF3ABh
NMAS.dat 4FF4C1h
NMAS.dat 4FF555h
NMAS.dat 4FF6BAh
NMAS.dat 4FF7D0h
NMAS.dat 4FF861h
NMAS.dat 4FF9C3h
NMAS.dat 4FFAD6h
NMAS.dat 525B6Ch
NMAS.dat 525DD0h

[Jeff-Groves]

Note!!! IF the files have been messed with in anyway from original GWX 3 files?
The addresses above will not be correct.

That I class as NMFP!

[Hebe Vollmaus]

Again thanks for the info!


The interested reader here may already know, but it should be noted that the SHD-node may have a material connected. This may also deleted if it is not otherwise related also to something else. I think this would be relevant for some automatic Deletion.
For such Delete-process i think it is not possible to use a sort of "Mouse-klick - Push-Button" tool what can be found on Internet.
To telecommand S3D it is needed to know what S3D will show after a <STRG-F> + <DEL> Command. Since sometimes there are two SHD-Controllers after another. The tool should be able to read-out this info and react accordingly.

AutoIt is a thing i have never used, so cant tell anything about.



The Deutschland is an example what i call a Zero-sized 3D-object, from what i think all of you already have it on Harddisk. The Node from posted iD in S3D is 161. Below 162 "NBC_Dtland_cabluri02". Hex File position 0x4CEA2A. The iD posted can be found over the Function "Edit/Find..." in S3D. It is the real iD reversed plus 0x in front of.
My private tool is ready to find this, but the whole thing is not qualified for the public yet (maybe never - lets see).

My "Crash"-Ship may be unknown/no longer used today. It was the "Fast Transport" NTR3, made out of the TroopShip TR (NTR), with a second funnel added and some Lifeboats removed. Not really removed but "Zeroit" out. :doh:

[Jeff-Groves]

Most of the things that need deleted as you've pointed out?
I've talked about years ago. I don't think the ship builders now days put that trash in. At least I hope not!

S3D can not be Autoit run to find and delete the stuff. Sadly it does not have the ability to be run that way. Trust me! I know! :03:

A stand alone program could be done to scan and remove trash with a single pass of ALL files in a given folder and all sub folders.

I have a few scripts for 010 that do things like that but never wrote a full Trash remover script. Problem is not everyone will buy 010 so why bother?

[Hebe Vollmaus]

Nov 2018, 8 pages
"Shall we talk about file sizes in mods"
https://www.subsim.com/radioroom//showthread.php?t=239249

[Jeff-Groves]

Good spotting Mate!

If you check new Units by Vondos and others? You can clearly see they paid attention to that thread.

It's only old Units built LONG ago that suffer the problems of zero 3D models and so on.

I wish We had known all that stuff back years before that thread!

As to just the size of files on disk? I hate that S3D pounds his header stuff into the Units.