Shall we talk about file sizes in mods?

Jeff-Groves · 11-25-18, 11:40 PM

Re-using 3D meshes in a file.
I'll provide the example.
http://www.mediafire.com/file/4afldw...sell2.dat/file

Look at this dat with S3D. Then look at how the actual 3D meshes were done. Notice there are very few actual meshes used to complete the Sea Forts. The Game engine renders the needed parts as we placed them. No need for extra meshes!
At 2+ MB as is? Doing everything as separate meshes would place it at over 9 or 10 MB!
(Yes. They suffer the dreaded Dynamic shadow stuff! I know better now.)

Jeff-Groves · 11-26-18, 12:07 AM

One point I failed to mention about the dynamicshadows stuff.
In the early days one could clone a Unit but We did not have S3D to easily remove or add things.
It was mostly done with a hex editor and other very basic tools.

There's very few left here that can work the files with a hex editor.
In most cases you don't need one now. But one can still do things S3D and other Tools can not do!

Fifi · 11-26-18, 03:15 AM

It seems a pandora box you just opened

Of course community would be very grateful if you could manage to reduce mega mods loading time and probably avoid some CTD linked to them.

I’ve tried almost all the super mods present here on subsim, and the longer/heavier to load on my « old » gaming machine are in this order:

- GWX Rework Rus (always CTD on launching career - at least 8mn load)
- WAC5 (often CTD on loading save - almost 8mn load)
- TWOS (never CTD - around 3mn load)
- LSH (very few CTD - around 2mn load)
- GWX3 (never CTD - around 2mn load)
- NYGM (never CTD - around 1mn load)

Also get some occasional stuttering on TC with WAC5 and TWOS, very few with LSH and GWX, almost never with NYGM, but that’s an other story...

Sounds have a huge impact on loading times too. Probably N°1 hit

And WAC5 has a nice sound folder...

Blackswan1940 · 11-26-18, 09:47 AM

Quote:

Originally Posted by Fifi

- TWOS (never CTD - around 3mn load)

Since when is WOS available for SH3?
I thought it is a mod for SH5!?!?

Fifi · 11-26-18, 10:41 AM

Quote:

Originally Posted by Blackswan1940

Since when is WOS available for SH3?
I thought it is a mod for SH5!?!?

Was talking about mega mods i’m Using on my computer

Not only SH3.
And i know Jeff is also SH5 user

ETR3(SS) · 11-26-18, 02:13 PM

I started doing this in SH4 with some of the mods I released once I got a better understanding of how the game worked. One of the added unintended effects was better mod compatibility because less files were used. Case in point is the UnitParts files for all the different Conning Towers in SH4. If you only used a single CT, you could combine all the files pertaining to the CT and hull into one. So not only is removing unused nodes beneficial, but removing redundant file structure is as well.

ETR3(SS) · 11-26-18, 02:20 PM

And on the subject of useless entries...any ideas what the dmy_soclu entries are for? Soclu translates to pedestal in Romanian and the nodes seem to be positioned along the deck.

Anvar1061 · 11-26-18, 03:26 AM

EINFACH GENIAL

propbeanie · 11-26-18, 08:43 AM

Quote:

Originally Posted by Fifi

It seems a pandora box you just opened

Of course community would be very grateful if you could manage to reduce mega mods loading time and probably avoid some CTD linked to them.

I’ve tried almost all the super mods present here on subsim, and the longer/heavier to load on my « old » gaming machine are in this order:

- GWX Rework Rus (always CTD on launching career - at least 8mn load)
- WAC5 (often CTD on loading save - almost 8mn load)
- TWOS (never CTD - around 3mn load)
- LSH (very few CTD - around 2mn load)
- GWX3 (never CTD - around 2mn load)
- NYGM (never CTD - around 1mn load)

Also get some occasional stuttering on TC with WAC5 and TWOS, very few with LSH and GWX, almost never with NYGM, but that’s an other story...

Sounds have a huge impact on loading times too. Probably N°1 hit

And WAC5 has a nice sound folder...

Ahhh but Fifi, this would help ~close~ the lid on Pandora's Box...

Hitman · 11-26-18, 09:09 AM

Jeff you have raised one of the most important points ever here. So many years we have been following the inertia and ignoring how the snowball became ever bigger while rolling. There is no doubt the 3D part is a huge hit for our processors - having gone to much more modern PCs since 2005 has masked it but still on old game engines like SH3 becomes noticeable. Yes, shared resources in the 3D part are key to solve many problems but we have a formidable number of scattered units around, some in super mods, others in single unit mods/addons

Therefore the best way to do an optimization that works for everyone would probably be to list the units and proceed in a collaborative effort to slowly make our way through it. I guess that IABL merchant fleet mod is one of the most spread around mods, but I have no idea if he already did that optimization.

Jeff-Groves · 11-26-18, 04:57 PM

Quote:

Originally Posted by Hitman

Jeff you have raised one of the most important points ever here. So many years we have been following the inertia and ignoring how the snowball became ever bigger while rolling. There is no doubt the 3D part is a huge hit for our processors - having gone to much more modern PCs since 2005 has masked it but still on old game engines like SH3 becomes noticeable. Yes, shared resources in the 3D part are key to solve many problems but we have a formidable number of scattered units around, some in super mods, others in single unit mods/addons

Therefore the best way to do an optimization that works for everyone would probably be to list the units and proceed in a collaborative effort to slowly make our way through it. I guess that IABL merchant fleet mod is one of the most spread around mods, but I have no idea if he already did that optimization.

I believe a program written to scan every dat file and remove the useless stuff would be the way to go. Then anyone could download it, run it, and carry on without the hassle of reinstalling and so forth. No MFM has not been optimized. It has 125 Units with the stuff.

Jeff-Groves · 11-26-18, 05:08 PM

For converting SH3 units to SH4? There is a program that will do all the hard work for you.
It converts the unit completely! only thing it does not do is create the AO mappings.
All dynamic shadow stuff is ripped from the file along with the 3D models,
Unifiedrender is installed, textures are converted to dds, val and other files fixed also.

So it NEARLY readies SH3 units for SH5 also!

Jeff-Groves · 11-26-18, 05:56 PM

I just scanned the complete data folder of GWX.
I found 244 files with dynamicshadow.

My scan lists the names and locations of every file found also.
I'm adding to the script to give me exact locations in each file.
From there I should be able to build up to removing things.

areo16 · 11-27-18, 01:50 AM

Quote:

Originally Posted by Jeff-Groves

I believe a program written to scan every dat file and remove the useless stuff would be the way to go. Then anyone could download it, run it, and carry on without the hassle of reinstalling and so forth. No MFM has not been optimized. It has 125 Units with the stuff.

We have S3D to open the .dat file and show its objects and relationships but I don't think we have a .dat decompiler that can output some sort of file format that is readable to such a script.

Jeff-Groves · 11-27-18, 03:03 PM

Quote:

Originally Posted by areo16

We have S3D to open the .dat file and show its objects and relationships but I don't think we have a .dat decompiler that can output some sort of file format that is readable to such a script.

Don't need to decompile the dats.
I can read them in Hex as many of the old modders did from the start.
Some of us still know the format of the dats.

Here's a little sample:

SH III DAT FILE FORMAT
Skip the first 20 bytes they seem to be always the same.
Next 4 bytes are a long with the header string lenght.
Next is the author name and the "using Kashmir" stuff, total lenght as read above.
Next, file is made of "chunks". All chunks are like this:
ChunkType 4 bytes
Chunk Subtype/Version 4 bytes
chunk_size long (4 bytes)
Data (chunk_size bytes)

Chunk types and internal structure of each one:
type 0: EMPTY chunk
it's just 12 "00" bytes, so type, subtype and size are 0
type 1: 3D chunk
after the Chunk type, subtype/ver and size, format of the data part is like follows:
chunk_id: 8 bytes //this is used to reference this part inside the file. References usually found inside NODE (type 4) chunks

010 can be scripted to do what ever one wants to do to the dats in that case.
S3D is a fine Tool! Many of us old modders contributed information during it's development.
But it has it's limits. Working with a great Hex editor excides what S3D can do.

11-26-18, 03:15 AM	#3
Fifi Navy Seal Join Date: Dec 2012 Location: France Posts: 6,094 Downloads: 466 Uploads: 0	It seems a pandora box you just opened Of course community would be very grateful if you could manage to reduce mega mods loading time and probably avoid some CTD linked to them. I’ve tried almost all the super mods present here on subsim, and the longer/heavier to load on my « old » gaming machine are in this order: - GWX Rework Rus (always CTD on launching career - at least 8mn load) - WAC5 (often CTD on loading save - almost 8mn load) - TWOS (never CTD - around 3mn load) - LSH (very few CTD - around 2mn load) - GWX3 (never CTD - around 2mn load) - NYGM (never CTD - around 1mn load) Also get some occasional stuttering on TC with WAC5 and TWOS, very few with LSH and GWX, almost never with NYGM, but that’s an other story... Sounds have a huge impact on loading times too. Probably N°1 hit And WAC5 has a nice sound folder... __________________ https://www.mediafire.com/file/rfw2h...N_V7.8.7z/file

11-26-18, 02:13 PM	#6
ETR3(SS) Ocean Warrior Join Date: Dec 2007 Location: Between test depth and periscope depth Posts: 3,021 Downloads: 175 Uploads: 16	I started doing this in SH4 with some of the mods I released once I got a better understanding of how the game worked. One of the added unintended effects was better mod compatibility because less files were used. Case in point is the UnitParts files for all the different Conning Towers in SH4. If you only used a single CT, you could combine all the files pertaining to the CT and hull into one. So not only is removing unused nodes beneficial, but removing redundant file structure is as well. __________________ USS Kentucky SSBN 737 (G) Comms Div 2003-2006 Qualified 19 November 03 Yes I was really on a submarine.

11-26-18, 02:20 PM	#7
ETR3(SS) Ocean Warrior Join Date: Dec 2007 Location: Between test depth and periscope depth Posts: 3,021 Downloads: 175 Uploads: 16	And on the subject of useless entries...any ideas what the dmy_soclu entries are for? Soclu translates to pedestal in Romanian and the nodes seem to be positioned along the deck. __________________ USS Kentucky SSBN 737 (G) Comms Div 2003-2006 Qualified 19 November 03 Yes I was really on a submarine.

11-26-18, 09:09 AM	#10
Hitman Pacific Aces Dev Team Join Date: Sep 2002 Location: Spain Posts: 6,109 Downloads: 109 Uploads: 2	Jeff you have raised one of the most important points ever here. So many years we have been following the inertia and ignoring how the snowball became ever bigger while rolling. There is no doubt the 3D part is a huge hit for our processors - having gone to much more modern PCs since 2005 has masked it but still on old game engines like SH3 becomes noticeable. Yes, shared resources in the 3D part are key to solve many problems but we have a formidable number of scattered units around, some in super mods, others in single unit mods/addons Therefore the best way to do an optimization that works for everyone would probably be to list the units and proceed in a collaborative effort to slowly make our way through it. I guess that IABL merchant fleet mod is one of the most spread around mods, but I have no idea if he already did that optimization. __________________ One day I will return to sea ...

11-25-18, 11:40 PM	#1
Jeff-Groves GLOBAL MODDING TERRORIST Join Date: Sep 2014 Posts: 5,655 Downloads: 137 Uploads: 0	Re-using 3D meshes in a file. I'll provide the example. http://www.mediafire.com/file/4afldw...sell2.dat/file Look at this dat with S3D. Then look at how the actual 3D meshes were done. Notice there are very few actual meshes used to complete the Sea Forts. The Game engine renders the needed parts as we placed them. No need for extra meshes! At 2+ MB as is? Doing everything as separate meshes would place it at over 9 or 10 MB! (Yes. They suffer the dreaded Dynamic shadow stuff! I know better now.)

11-26-18, 12:07 AM	#2
Jeff-Groves GLOBAL MODDING TERRORIST Join Date: Sep 2014 Posts: 5,655 Downloads: 137 Uploads: 0	One point I failed to mention about the dynamicshadows stuff. In the early days one could clone a Unit but We did not have S3D to easily remove or add things. It was mostly done with a hex editor and other very basic tools. There's very few left here that can work the files with a hex editor. In most cases you don't need one now. But one can still do things S3D and other Tools can not do!

11-26-18, 03:26 AM	#8
Anvar1061 Ocean Warrior Join Date: May 2012 Location: In the sea, on land and above Posts: 3,487 Downloads: 878 Uploads: 0	EINFACH GENIAL

11-26-18, 05:08 PM	#12
Jeff-Groves GLOBAL MODDING TERRORIST Join Date: Sep 2014 Posts: 5,655 Downloads: 137 Uploads: 0	For converting SH3 units to SH4? There is a program that will do all the hard work for you. It converts the unit completely! only thing it does not do is create the AO mappings. All dynamic shadow stuff is ripped from the file along with the 3D models, Unifiedrender is installed, textures are converted to dds, val and other files fixed also. So it NEARLY readies SH3 units for SH5 also!

11-26-18, 05:56 PM	#13
Jeff-Groves GLOBAL MODDING TERRORIST Join Date: Sep 2014 Posts: 5,655 Downloads: 137 Uploads: 0	I just scanned the complete data folder of GWX. I found 244 files with dynamicshadow. My scan lists the names and locations of every file found also. I'm adding to the script to give me exact locations in each file. From there I should be able to build up to removing things.