Shall we talk about file sizes in mods?

Jeff-Groves · 11-25-18, 07:53 PM

So removing such things is an advantage to any mod.
Hans pointed out it is a large amount of work IF done the old way by hand and editing every file one by one.

What if we run a program to scan the complete Sea folder and do all the work for us?
Would that be of interest?

propbeanie · 11-25-18, 10:14 PM

Yes.

Jeff-Groves · 11-25-18, 11:06 PM

A few minutes scripting for 010 I was able to scan the sea folder of major mods for dynamicshadows.
All Mods were extracted but not installed to a game folder so it's the actual count of that Mod's Sea units.

I'll list from low to high.
Note that the mods might be over writing stock units along with adding units or some units from MFM may already be included in Major mods.
This count ONLY looks at dats.

Stock SH3 - 48 units
NYGM - 32 units
MFM - 125 units
WAC - 162 units
GWX - 181 units (Knocked on the back door to extract)
LSH - 320 units (Provided a base SH3 folder with SH3 files and no other folders to extract)

I could scan for reflection stuff but the count would be the same I suspect.
Reflections are needed but many times the added 3D mesh may not be needed. I've 2 thoughts on that not proven.

So it's a massive amount of work to edit by hand!
Not to mention Units converted to SH4 and possibly slipped into SH5!

Jeff-Groves · 11-25-18, 11:40 PM

Re-using 3D meshes in a file.
I'll provide the example.
http://www.mediafire.com/file/4afldw...sell2.dat/file

Look at this dat with S3D. Then look at how the actual 3D meshes were done. Notice there are very few actual meshes used to complete the Sea Forts. The Game engine renders the needed parts as we placed them. No need for extra meshes!
At 2+ MB as is? Doing everything as separate meshes would place it at over 9 or 10 MB!
(Yes. They suffer the dreaded Dynamic shadow stuff! I know better now.)

Jeff-Groves · 11-26-18, 12:07 AM

One point I failed to mention about the dynamicshadows stuff.
In the early days one could clone a Unit but We did not have S3D to easily remove or add things.
It was mostly done with a hex editor and other very basic tools.

There's very few left here that can work the files with a hex editor.
In most cases you don't need one now. But one can still do things S3D and other Tools can not do!

Fifi · 11-26-18, 03:15 AM

It seems a pandora box you just opened

Of course community would be very grateful if you could manage to reduce mega mods loading time and probably avoid some CTD linked to them.

I’ve tried almost all the super mods present here on subsim, and the longer/heavier to load on my « old » gaming machine are in this order:

- GWX Rework Rus (always CTD on launching career - at least 8mn load)
- WAC5 (often CTD on loading save - almost 8mn load)
- TWOS (never CTD - around 3mn load)
- LSH (very few CTD - around 2mn load)
- GWX3 (never CTD - around 2mn load)
- NYGM (never CTD - around 1mn load)

Also get some occasional stuttering on TC with WAC5 and TWOS, very few with LSH and GWX, almost never with NYGM, but that’s an other story...

Sounds have a huge impact on loading times too. Probably N°1 hit

And WAC5 has a nice sound folder...

Blackswan1940 · 11-26-18, 09:47 AM

Quote:

Originally Posted by Fifi

- TWOS (never CTD - around 3mn load)

Since when is WOS available for SH3?
I thought it is a mod for SH5!?!?

Anvar1061 · 11-26-18, 03:26 AM

EINFACH GENIAL

propbeanie · 11-26-18, 08:43 AM

Quote:

Originally Posted by Fifi

It seems a pandora box you just opened

Of course community would be very grateful if you could manage to reduce mega mods loading time and probably avoid some CTD linked to them.

I’ve tried almost all the super mods present here on subsim, and the longer/heavier to load on my « old » gaming machine are in this order:

- GWX Rework Rus (always CTD on launching career - at least 8mn load)
- WAC5 (often CTD on loading save - almost 8mn load)
- TWOS (never CTD - around 3mn load)
- LSH (very few CTD - around 2mn load)
- GWX3 (never CTD - around 2mn load)
- NYGM (never CTD - around 1mn load)

Also get some occasional stuttering on TC with WAC5 and TWOS, very few with LSH and GWX, almost never with NYGM, but that’s an other story...

Sounds have a huge impact on loading times too. Probably N°1 hit

And WAC5 has a nice sound folder...

Ahhh but Fifi, this would help ~close~ the lid on Pandora's Box...

Hitman · 11-26-18, 09:09 AM

Jeff you have raised one of the most important points ever here. So many years we have been following the inertia and ignoring how the snowball became ever bigger while rolling. There is no doubt the 3D part is a huge hit for our processors - having gone to much more modern PCs since 2005 has masked it but still on old game engines like SH3 becomes noticeable. Yes, shared resources in the 3D part are key to solve many problems but we have a formidable number of scattered units around, some in super mods, others in single unit mods/addons

Therefore the best way to do an optimization that works for everyone would probably be to list the units and proceed in a collaborative effort to slowly make our way through it. I guess that IABL merchant fleet mod is one of the most spread around mods, but I have no idea if he already did that optimization.

areo16 · 11-28-18, 02:31 AM

Quote:

Originally Posted by Jeff-Groves

Re-using 3D meshes in a file.
I'll provide the example.
http://www.mediafire.com/file/4afldw...sell2.dat/file

Look at this dat with S3D. Then look at how the actual 3D meshes were done. Notice there are very few actual meshes used to complete the Sea Forts. The Game engine renders the needed parts as we placed them. No need for extra meshes!
At 2+ MB as is? Doing everything as separate meshes would place it at over 9 or 10 MB!
(Yes. They suffer the dreaded Dynamic shadow stuff! I know better now.)

Thanks for the offsets for the .dat file in hex very helpful.

So I understand the optimized approach you are talking about here, you are still copying the same model (obj) over and over again in S3D?

Isn't it possible to just reference the same model in S3D and just go about it that way, wouldn't that save space?

I guess, from a hex point of view, what edits would we want to make to these files to make them more optimized. Looks like looking for duplicate data chunks for models (type 1) is step 1, and then possibly deleting the duplicates and re-reference the deleted duplicate data chunks with the original model reference id?

@EDIT

Yes in S3D it is possible to reference the same model in multiple nodes and this works just fine. It also decreases file space.

Is it possible to reference a model ID that is in another .dat file?

Dirk Gently · 11-28-18, 06:23 AM

Revelutionary find! A tool like that would not only open up more possibilities in terms of heavier expansion mods but also to make very high quality lightweight mods for non-gaming laptops.

Extraordinary find and plans, guys.

Jeff-Groves · 11-28-18, 10:44 AM

Quote:

Originally Posted by areo16

Thanks for the offsets for the .dat file in hex very helpful.

So I understand the optimized approach you are talking about here, you are still copying the same model (obj) over and over again in S3D?

Isn't it possible to just reference the same model in S3D and just go about it that way, wouldn't that save space?

I guess, from a hex point of view, what edits would we want to make to these files to make them more optimized. Looks like looking for duplicate data chunks for models (type 1) is step 1, and then possibly deleting the duplicates and re-reference the deleted duplicate data chunks with the original model reference id?

@EDIT

Yes in S3D it is possible to reference the same model in multiple nodes and this works just fine. It also decreases file space.

Is it possible to reference a model ID that is in another .dat file?

The Sea Forts were built in a program much like S3D. It's called Jammu.
ref never released it because he didn't feel it was good enough (Suffered from some annoying bugs now and then) and did not want to support it nor anyone that may use it. It actually pre-dates S3D by several years!
I still use it as it has the ability to mass export ALL 3D objects and UV mappings in one click! (Then it crashes)

You can use the Placement node as one way to use objects from a different dat. Look at how Harbors and Cities are done.

On the format of dats? Look at one in S3D with the List view. That show how the dat is actually coded from node 1 to the end of file.
The Tree view makes it easier for you to understand the relationship of each node

s7rikeback · 11-30-18, 07:16 PM

Quote:

Originally Posted by Jeff-Groves

So removing such things is an advantage to any mod.
Hans pointed out it is a large amount of work IF done the old way by hand and editing every file one by one.

What if we run a program to scan the complete Sea folder and do all the work for us?
Would that be of interest?

Hell yeah!!!

11-25-18, 10:14 PM	#2
propbeanie CTD - it's not just a job Join Date: May 2016 Location: One hour from Music City USA! Posts: 9,808 Downloads: 444 Uploads: 2	Yes. __________________ "...and bollocks to the naysayer/s" - Jimbuna

11-25-18, 11:06 PM	#3
Jeff-Groves Village Idiot Join Date: Sep 2014 Posts: 5,397 Downloads: 131 Uploads: 0	A few minutes scripting for 010 I was able to scan the sea folder of major mods for dynamicshadows. All Mods were extracted but not installed to a game folder so it's the actual count of that Mod's Sea units. I'll list from low to high. Note that the mods might be over writing stock units along with adding units or some units from MFM may already be included in Major mods. This count ONLY looks at dats. Stock SH3 - 48 units NYGM - 32 units MFM - 125 units WAC - 162 units GWX - 181 units (Knocked on the back door to extract) LSH - 320 units (Provided a base SH3 folder with SH3 files and no other folders to extract) I could scan for reflection stuff but the count would be the same I suspect. Reflections are needed but many times the added 3D mesh may not be needed. I've 2 thoughts on that not proven. So it's a massive amount of work to edit by hand! Not to mention Units converted to SH4 and possibly slipped into SH5! Last edited by Jeff-Groves; 11-25-18 at 11:25 PM.

11-26-18, 03:15 AM	#6
Fifi Navy Seal Join Date: Dec 2012 Location: France Posts: 5,801 Downloads: 457 Uploads: 0	It seems a pandora box you just opened Of course community would be very grateful if you could manage to reduce mega mods loading time and probably avoid some CTD linked to them. I’ve tried almost all the super mods present here on subsim, and the longer/heavier to load on my « old » gaming machine are in this order: - GWX Rework Rus (always CTD on launching career - at least 8mn load) - WAC5 (often CTD on loading save - almost 8mn load) - TWOS (never CTD - around 3mn load) - LSH (very few CTD - around 2mn load) - GWX3 (never CTD - around 2mn load) - NYGM (never CTD - around 1mn load) Also get some occasional stuttering on TC with WAC5 and TWOS, very few with LSH and GWX, almost never with NYGM, but that’s an other story... Sounds have a huge impact on loading times too. Probably N°1 hit And WAC5 has a nice sound folder... __________________ https://www.mediafire.com/file/8axze...N_V6.1.7z/file

11-26-18, 09:09 AM	#10
Hitman Pacific Aces Dev Team Join Date: Sep 2002 Location: Spain Posts: 6,099 Downloads: 109 Uploads: 2	Jeff you have raised one of the most important points ever here. So many years we have been following the inertia and ignoring how the snowball became ever bigger while rolling. There is no doubt the 3D part is a huge hit for our processors - having gone to much more modern PCs since 2005 has masked it but still on old game engines like SH3 becomes noticeable. Yes, shared resources in the 3D part are key to solve many problems but we have a formidable number of scattered units around, some in super mods, others in single unit mods/addons Therefore the best way to do an optimization that works for everyone would probably be to list the units and proceed in a collaborative effort to slowly make our way through it. I guess that IABL merchant fleet mod is one of the most spread around mods, but I have no idea if he already did that optimization. __________________ One day I will return to sea ...

11-25-18, 07:53 PM	#1
Jeff-Groves Village Idiot Join Date: Sep 2014 Posts: 5,397 Downloads: 131 Uploads: 0	So removing such things is an advantage to any mod. Hans pointed out it is a large amount of work IF done the old way by hand and editing every file one by one. What if we run a program to scan the complete Sea folder and do all the work for us? Would that be of interest?

11-25-18, 11:40 PM	#4
Jeff-Groves Village Idiot Join Date: Sep 2014 Posts: 5,397 Downloads: 131 Uploads: 0	Re-using 3D meshes in a file. I'll provide the example. http://www.mediafire.com/file/4afldw...sell2.dat/file Look at this dat with S3D. Then look at how the actual 3D meshes were done. Notice there are very few actual meshes used to complete the Sea Forts. The Game engine renders the needed parts as we placed them. No need for extra meshes! At 2+ MB as is? Doing everything as separate meshes would place it at over 9 or 10 MB! (Yes. They suffer the dreaded Dynamic shadow stuff! I know better now.)

11-26-18, 12:07 AM	#5
Jeff-Groves Village Idiot Join Date: Sep 2014 Posts: 5,397 Downloads: 131 Uploads: 0	One point I failed to mention about the dynamicshadows stuff. In the early days one could clone a Unit but We did not have S3D to easily remove or add things. It was mostly done with a hex editor and other very basic tools. There's very few left here that can work the files with a hex editor. In most cases you don't need one now. But one can still do things S3D and other Tools can not do!

11-26-18, 03:26 AM	#8
Anvar1061 Ocean Warrior Join Date: May 2012 Location: In the sea, on land and above Posts: 3,354 Downloads: 850 Uploads: 0	EINFACH GENIAL

11-28-18, 06:23 AM	#12
Dirk Gently Bosun Join Date: Nov 2018 Location: Norway Posts: 65 Downloads: 61 Uploads: 0	Revelutionary find! A tool like that would not only open up more possibilities in terms of heavier expansion mods but also to make very high quality lightweight mods for non-gaming laptops. Extraordinary find and plans, guys.