Shall we talk about file sizes in mods?

[areo16]

I think this is a damn good idea. Reason for many crashes too is not just id conflicts but hitting the 4gb limit for the 32 bit application which sh3 is (with 4gb patch).

[areo16]

Originally Posted by Jeff-Groves

I believe a program written to scan every dat file and remove the useless stuff would be the way to go. Then anyone could download it, run it, and carry on without the hassle of reinstalling and so forth. No MFM has not been optimized. It has 125 Units with the stuff.

We have S3D to open the .dat file and show its objects and relationships but I don't think we have a .dat decompiler that can output some sort of file format that is readable to such a script.

Originally Posted by Jeff-Groves

Water Reflections in most cases is the same.
No need for a separate 3D mesh.

It's been a long time since I tweaked with SH3/SH4 files, but I have to disagree here.

The water reflections don't need the full poly model to be displayed. A low-poly model can be safely used (not actually noticeable in the game) and lessen the charge on the GPU.

And I eagerly await for that program to convert SH3 models into SH4. I remember it was a painful, time consuming job to do.

[bigboywooly]

Originally Posted by Jeff-Groves

BBW,
Take a look at the NBB_Nevada in the SH4 sea folder.
(I can send it to you if you don't have SH4)
You might recall I tested this just after SH4 came out as We were putting the BuV_Dock1 and BuV_Dock2 into GWX.

Hmmm, have SH4. Will take a look

[BigWalleye]

Some thoughts from an old programmer:

Once, a very, very long time ago, I worked on the operating system for a computer which had exactly 4096 words of random access memory. That's not a misprint! Not K. Not M. Four thousand ninety-six 18-bit words. (A beer for anyone who can correctly identify that machine!) Every instruction had to be carefully chosen. Every status bit had to be considered for maximum added functionality. For the size of the code, it required a huge investment in programmer effort.

Later, I worked on a real-time virtual OS which, for sound physical reasons, was based on magneitc drum storage. The latency time of the slowly-turning drum meant that data and executable storage had to be carefully optimized to be under the heads when needed. There was software to help, but any significant change to data structures meant that the entire drum had to be remapped. Again, massive amounts of effort to overcome hardware limitations.

Throughout the history of computing, huge amounts of human programmer effort have been used to compensate for the limitations of the existing hardware. I'm sure Ada Lovelace had the same problems.

When SH3 was developed, the 3D models taxed the generally available hardware to the point where load times were barely tolerable. As SH3 grew through mods, much more extensive modelling made use of improving machine capabilities, and load times stayed barely tolerable. So we are today.

I am sure it would be possible, through large amounts of modder effort, to optimize the 3D models with standard elements. This would likely make a significant improvement in load times, especially for older hardware. (Would it also add significant render times and stress older CPUs and GPUs?) But is this the most effective use of the time and talents of a small modding community? The Polish Engineer sez: "If it ain't broke, don't try to fix it."

There are still many possible improvements to SH3 waiting to be done. In reverse enginnering, hsie did wonderful things, but much more is possible. (Compare to what TDW has done for SH5.) While the skill set for 3D modelling is not the same, still I suspect there are many useful projects to be done. (How about a complete interior?) I suggest that there are many projects which could use modder effort more effectively than cleaning up old inefficient models which do run adequately on current machines.

Truth in advertising: I am not an active modder, except for personal consumption. FWIW, this is just the opinon of a long-time software developer. YMMV.

And for all those who labor to make this old horse better and better every year, THANK YOU. Much love, and I will be grateful to use whatever you care to contribute to the community. God bless!

[Jeff-Groves]

Originally Posted by areo16

We have S3D to open the .dat file and show its objects and relationships but I don't think we have a .dat decompiler that can output some sort of file format that is readable to such a script.

Don't need to decompile the dats.
I can read them in Hex as many of the old modders did from the start.
Some of us still know the format of the dats.
Here's a little sample:


SH III DAT FILE FORMAT
Skip the first 20 bytes they seem to be always the same.
Next 4 bytes are a long with the header string lenght.
Next is the author name and the "using Kashmir" stuff, total lenght as read above.
Next, file is made of "chunks". All chunks are like this:
ChunkType 4 bytes
Chunk Subtype/Version 4 bytes
chunk_size long (4 bytes)
Data (chunk_size bytes)

Chunk types and internal structure of each one:
type 0: EMPTY chunk
it's just 12 "00" bytes, so type, subtype and size are 0
type 1: 3D chunk
after the Chunk type, subtype/ver and size, format of the data part is like follows:
chunk_id: 8 bytes //this is used to reference this part inside the file. References usually found inside NODE (type 4) chunks


010 can be scripted to do what ever one wants to do to the dats in that case.
S3D is a fine Tool! Many of us old modders contributed information during it's development.
But it has it's limits. Working with a great Hex editor excides what S3D can do.

[Jeff-Groves]

4096 words of random access memory. That wouldn't have been the 704's would it?

"I suggest that there are many projects which could use modder effort more effectively than cleaning up old inefficient models which do run adequately on current machines."

adequately

There's the word I hate.
The program runs adequately as is but is in-efficient and can load faster, reduce possible ID conflicts, and allow the complete interiors to work better.
(They have been done for several Mods)

[Jeff-Groves]

Originally Posted by JapLance

And I eagerly await for that program to convert SH3 models into SH4. I remember it was a painful, time consuming job to do.

Check your PM's.

Your review of the program would be great!

[ETR3(SS)]

Originally Posted by Jeff-Groves

adequately

There's the word I hate.

Better is the enemy of 'good enough.'

[Jeff-Groves]

Originally Posted by ETR3(SS)

Better is the enemy of 'good enough.'

Not sure which interpretation of this you intend?


We've been running SH3 the 'good enough' way since 2005.
In my 40 year trade in construction? I learned 'good enough' was just a way to justify sub par work.
I do not accept 'good enough' when training our employees or doing the work we are hired to do.

[ETR3(SS)]

Sorry for leaving that ambiguous; it was an agreement with your statement on adequately.

I've run into more than my fair share of good enough and it drives me nuts, particularly when I'm the one that has to make it better.

[Jeff-Groves]

Originally Posted by ETR3(SS)

I've run into more than my fair share of good enough and it drives me nuts, particularly when I'm the one that has to make it better.

Kind of thought so.
And I'm the one that gets sent 500+ miles from home to fix 'Good enough'.

[BigWalleye]

On the day when we have all the time in the world and all the manpower in the world, we will not have to prioritize anything, and everything that needs to be done will get done. Until then, I personally believe adding new capabilities to SH3 is a better use of scarce modder time than cleaning up existing mods that work adequately. But that's just my opinion. As I pointed out, I am not modding, and what project any modder chooses to work on is a strictly up to them. I just wanted to lobby for considering an alternative.

My interpretation of "Better is the enemy of good enough." is the one endorsed by Winston Churchill, who had considerable experience in effectively prioritizing limited resources. (AFAIK, it was Churchill who coined that aphorism, or at least popularized it.)

And no, it was not a 704. The 704 used two different instruction formats, but each one was 36 bits long. And the addressing was 36-bit oriented. A hint: It was a mini-computer and the bulk storage device was called DECtape.

[propbeanie]

All he's talking about doing is removing un-used "code", if you will (bloatware), so the data will not only fit on the tape drive, but also be in a nice orderly manner...

[areo16]

Originally Posted by Jeff-Groves

Re-using 3D meshes in a file.
I'll provide the example.
http://www.mediafire.com/file/4afldw...sell2.dat/file

Look at this dat with S3D. Then look at how the actual 3D meshes were done. Notice there are very few actual meshes used to complete the Sea Forts. The Game engine renders the needed parts as we placed them. No need for extra meshes!
At 2+ MB as is? Doing everything as separate meshes would place it at over 9 or 10 MB!
(Yes. They suffer the dreaded Dynamic shadow stuff! I know better now.)

Thanks for the offsets for the .dat file in hex very helpful.


So I understand the optimized approach you are talking about here, you are still copying the same model (obj) over and over again in S3D?


Isn't it possible to just reference the same model in S3D and just go about it that way, wouldn't that save space?


I guess, from a hex point of view, what edits would we want to make to these files to make them more optimized. Looks like looking for duplicate data chunks for models (type 1) is step 1, and then possibly deleting the duplicates and re-reference the deleted duplicate data chunks with the original model reference id?


@EDIT


Yes in S3D it is possible to reference the same model in multiple nodes and this works just fine. It also decreases file space.


Is it possible to reference a model ID that is in another .dat file?