View Full Version : Shall we talk about file sizes in mods?
Jeff-Groves
11-25-18, 04:28 PM
One thing We do know is the larger the file sizes get?
The longer the load times and other issues that show up.
There are a lot of ways around this that I have posted about in the past but I'll go over the basics again.
Even GWX 3 did not know about the easiest way to clone ships until SH4 came out. So nearly all mods have an over head of needless 3d objects.
Needless objects cramp the whole loading process and stress system resources.
That is a fact! You have all seen this with long load times and CTDs.
We do know that it is only necessary to load one 3d model to complete a full render of a more complex in Game unit.
I use the Maunsell Forts in GWX as an example. Open those with S3D and see how it was done. There are examples in the original release of SH3 so I did not invent the idea. The prospect was there from the start!
It is never necessary to add massive 3D objects to a file when placement nodes will do the work. Don't add ungodly verts to a file when one model can be placed with the placement node as many times as you wish with less overhead.
As mods become more complicated? Look for the best way to complete the work without adding unneeded stuff.
I could spend months culling and fixing files in most every Major mod.
Even GWX3 could be cut back with what I know now.
Jeff-Groves
11-25-18, 04:42 PM
Talking about placement nodes?
IF I wanted to create something I wanted no one to duplicate easily?
I could build a .dat file of nothing but placement nodes,
scatter the parts around in different .dat files,
and you'd spend a lot of time to build the 3d model out side of the Game.
You'd have to chase every link and export the 3d model THEN hope to adjust it properly!
Jeff-Groves
11-25-18, 04:55 PM
How to duplicate Units in SH3 the SH4 way:
Look at the BBNevada in SH4
ClassName=BBNevada
3DModelFileName=data/Sea/NBB_Colorado/NBB_Colorado
This type setting works in SH3 also!
Follow the files.
HanSolo78
11-25-18, 04:55 PM
There could be saved a lot by sharing 3d models and textures!! :Kaleun_Thumbs_Up::Kaleun_Applaud:
But... all this work and time that is needed is tremendous.
Especially WAC5.x could need this optimisation, because it is running on the edge with SH3.
Jeff-Groves
11-25-18, 04:58 PM
I have some time to assist you if you'd like.
It's for the good of the Community.
One great thing about duplicate ships done the SH4 way?
No ID conflicts!
Jeff-Groves
11-25-18, 05:12 PM
One thing I did with GWX was look at optimizing things.
The project came to the end before We could take it to the extreme.
Had We had 1 more year to optimize?
There would have been a bar set that would be hard to match!
Jeff-Groves
11-25-18, 07:12 PM
Dynamic Shadows.
The very first patch for SH3 removed this failed controller.
What was NOT removed is all the SHD_whatever 3D meshes from the dats nor the controller embedded in the dats.
Waste of space and processor time to read and ignore.
Water Reflections in most cases is the same.
No need for a separate 3D mesh.
Just assign the controller to the main mesh.
Very rare cases you may need the original way.
Some may say "If it's not being used what's the harm?"
How much processor time is wasted to read and refuse the ID's in those controllers and does it reject those ID's if assigned elsewhere?
Remember, Names in dat's don't really mean a thing. (Open most clones and see for yourself) It's that ID number that is the most important thing.
Jeff-Groves
11-25-18, 07:39 PM
Quick example of the wasted SHD_ objects.
The NAFT in stock SH3, Node 74, SHD_NATF_B
The 3D model is 848 verts never used in game and uses 2 ID's
There are 13 not use dynamicshadowcast controllers, 13 3D meshes not used.
That is one small Unit! add 100 more ships and do the math.
I can hit 4 MILLION verts very easily that are not used but are read on load.
Jeff-Groves
11-25-18, 07:53 PM
So removing such things is an advantage to any mod.
Hans pointed out it is a large amount of work IF done the old way by hand and editing every file one by one.
What if we run a program to scan the complete Sea folder and do all the work for us?
Would that be of interest?
propbeanie
11-25-18, 10:14 PM
Yes. :salute:
Jeff-Groves
11-25-18, 11:06 PM
A few minutes scripting for 010 I was able to scan the sea folder of major mods for dynamicshadows.
All Mods were extracted but not installed to a game folder so it's the actual count of that Mod's Sea units.
I'll list from low to high.
Note that the mods might be over writing stock units along with adding units or some units from MFM may already be included in Major mods.
This count ONLY looks at dats.
Stock SH3 - 48 units
NYGM - 32 units
MFM - 125 units
WAC - 162 units
GWX - 181 units (Knocked on the back door to extract)
LSH - 320 units (Provided a base SH3 folder with SH3 files and no other folders to extract)
I could scan for reflection stuff but the count would be the same I suspect.
Reflections are needed but many times the added 3D mesh may not be needed. I've 2 thoughts on that not proven.
So it's a massive amount of work to edit by hand!
Not to mention Units converted to SH4 and possibly slipped into SH5!
Jeff-Groves
11-25-18, 11:40 PM
Re-using 3D meshes in a file.
I'll provide the example.
http://www.mediafire.com/file/4afldw66dvczbc8/LAA_AAMaunsell2.dat/file
Look at this dat with S3D. Then look at how the actual 3D meshes were done. Notice there are very few actual meshes used to complete the Sea Forts. The Game engine renders the needed parts as we placed them. No need for extra meshes!
At 2+ MB as is? Doing everything as separate meshes would place it at over 9 or 10 MB!
(Yes. They suffer the dreaded Dynamic shadow stuff! I know better now.)
Jeff-Groves
11-26-18, 12:07 AM
One point I failed to mention about the dynamicshadows stuff.
In the early days one could clone a Unit but We did not have S3D to easily remove or add things.
It was mostly done with a hex editor and other very basic tools.
There's very few left here that can work the files with a hex editor.
In most cases you don't need one now. But one can still do things S3D and other Tools can not do!
:03:
It seems a pandora box you just opened :up:
Of course community would be very grateful if you could manage to reduce mega mods loading time and probably avoid some CTD linked to them.
I’ve tried almost all the super mods present here on subsim, and the longer/heavier to load on my « old » gaming machine are in this order:
- GWX Rework Rus (always CTD on launching career - at least 8mn load)
- WAC5 (often CTD on loading save - almost 8mn load)
- TWOS (never CTD - around 3mn load)
- LSH (very few CTD - around 2mn load)
- GWX3 (never CTD - around 2mn load)
- NYGM (never CTD - around 1mn load)
Also get some occasional stuttering on TC with WAC5 and TWOS, very few with LSH and GWX, almost never with NYGM, but that’s an other story...
Sounds have a huge impact on loading times too. Probably N°1 hit :yep:
And WAC5 has a nice sound folder...
Anvar1061
11-26-18, 03:26 AM
EINFACH GENIAL
http://www.subsim.com/radioroom/images/smilies/Kaleun_Thumbs_Up-1.gifhttp://www.subsim.com/radioroom/images/smilies/Kaleun_Applaud-1.gifhttp://www.subsim.com/radioroom/images/smilies/Kaleun_Applaud-1.gifhttp://www.subsim.com/radioroom/images/smilies/Kaleun_Applaud-1.gif
propbeanie
11-26-18, 08:43 AM
It seems a pandora box you just opened :up:
Of course community would be very grateful if you could manage to reduce mega mods loading time and probably avoid some CTD linked to them.
I’ve tried almost all the super mods present here on subsim, and the longer/heavier to load on my « old » gaming machine are in this order:
- GWX Rework Rus (always CTD on launching career - at least 8mn load)
- WAC5 (often CTD on loading save - almost 8mn load)
- TWOS (never CTD - around 3mn load)
- LSH (very few CTD - around 2mn load)
- GWX3 (never CTD - around 2mn load)
- NYGM (never CTD - around 1mn load)
Also get some occasional stuttering on TC with WAC5 and TWOS, very few with LSH and GWX, almost never with NYGM, but that’s an other story...
Sounds have a huge impact on loading times too. Probably N°1 hit :yep:
And WAC5 has a nice sound folder...
Ahhh but Fifi, this would help ~close~ the lid on Pandora's Box... :O: :salute:
Jeff you have raised one of the most important points ever here. So many years we have been following the inertia and ignoring how the snowball became ever bigger while rolling. There is no doubt the 3D part is a huge hit for our processors - having gone to much more modern PCs since 2005 has masked it but still on old game engines like SH3 becomes noticeable. Yes, shared resources in the 3D part are key to solve many problems but we have a formidable number of scattered units around, some in super mods, others in single unit mods/addons
Therefore the best way to do an optimization that works for everyone would probably be to list the units and proceed in a collaborative effort to slowly make our way through it. I guess that IABL merchant fleet mod is one of the most spread around mods, but I have no idea if he already did that optimization.
Blackswan1940
11-26-18, 09:47 AM
- TWOS (never CTD - around 3mn load)
Since when is WOS available for SH3?
I thought it is a mod for SH5!?!?
Since when is WOS available for SH3?
I thought it is a mod for SH5!?!?
Was talking about mega mods i’m Using on my computer :03:
Not only SH3.
And i know Jeff is also SH5 user :salute:
ETR3(SS)
11-26-18, 02:13 PM
I started doing this in SH4 with some of the mods I released once I got a better understanding of how the game worked. One of the added unintended effects was better mod compatibility because less files were used. Case in point is the UnitParts files for all the different Conning Towers in SH4. If you only used a single CT, you could combine all the files pertaining to the CT and hull into one. So not only is removing unused nodes beneficial, but removing redundant file structure is as well.
ETR3(SS)
11-26-18, 02:20 PM
And on the subject of useless entries...any ideas what the dmy_soclu entries are for? Soclu translates to pedestal in Romanian and the nodes seem to be positioned along the deck.
bigboywooly
11-26-18, 04:18 PM
Hmmm, so in the MFM there are a few ships that are basically 3 copies of the same unit.
M**A for America
M**B for Britain
M**X for neutrals
And you're saying you only need one model to be called for in the other cases as in SH5 ?
GWX 3 plus the Schleuse mod with AOTD_Maxs units plus MFM and Von Dos's ships really stretches my load times
Jeff-Groves
11-26-18, 04:41 PM
BBW,
Take a look at the NBB_Nevada in the SH4 sea folder.
(I can send it to you if you don't have SH4)
You might recall I tested this just after SH4 came out as We were putting the BuV_Dock1 and BuV_Dock2 into GWX.
Jeff-Groves
11-26-18, 04:51 PM
And on the subject of useless entries...any ideas what the dmy_soclu entries are for? Soclu translates to pedestal in Romanian and the nodes seem to be positioned along the deck.
I know on the Turms the crew is positioned on the DMY_Soclu_xxx
They may be left over from a never implemented use?
Remove them and see what happens is all I can suggest.
Or maybe assign something to one with the eqp file?
Jeff-Groves
11-26-18, 04:57 PM
Jeff you have raised one of the most important points ever here. So many years we have been following the inertia and ignoring how the snowball became ever bigger while rolling. There is no doubt the 3D part is a huge hit for our processors - having gone to much more modern PCs since 2005 has masked it but still on old game engines like SH3 becomes noticeable. Yes, shared resources in the 3D part are key to solve many problems but we have a formidable number of scattered units around, some in super mods, others in single unit mods/addons
Therefore the best way to do an optimization that works for everyone would probably be to list the units and proceed in a collaborative effort to slowly make our way through it. I guess that IABL merchant fleet mod is one of the most spread around mods, but I have no idea if he already did that optimization.
I believe a program written to scan every dat file and remove the useless stuff would be the way to go. Then anyone could download it, run it, and carry on without the hassle of reinstalling and so forth. No MFM has not been optimized. It has 125 Units with the stuff.
Jeff-Groves
11-26-18, 05:08 PM
For converting SH3 units to SH4? There is a program that will do all the hard work for you.
It converts the unit completely! only thing it does not do is create the AO mappings.
All dynamic shadow stuff is ripped from the file along with the 3D models,
Unifiedrender is installed, textures are converted to dds, val and other files fixed also.
So it NEARLY readies SH3 units for SH5 also!
Jeff-Groves
11-26-18, 05:56 PM
I just scanned the complete data folder of GWX.
I found 244 files with dynamicshadow.
:o
My scan lists the names and locations of every file found also.
I'm adding to the script to give me exact locations in each file.
From there I should be able to build up to removing things.
HanSolo78
11-26-18, 06:16 PM
Your programme would do a hell of work! :Kaleun_Applaud:
If you have a look into the WAC full interior you will notice that there is also a lot to optimize.
http://www.mediafire.com/file/jlrdsd6c6js4z1d/WAC5.0_Full_Interior.7z/file
I started with remeshing 3d meshes and reducing triangles, etc.
Also.. very time consuming... I only got 1 interior done (not published yet).
Jeff-Groves
11-26-18, 06:21 PM
WAC scan of data folder shows 236 files total. So yes. Could be cut way down.
:)
Anvar1061
11-26-18, 11:34 PM
Jeff, give instructions and together we can work on perspective in my opinion project WAC v5.02.
I think this is a damn good idea. Reason for many crashes too is not just id conflicts but hitting the 4gb limit for the 32 bit application which sh3 is (with 4gb patch).
I believe a program written to scan every dat file and remove the useless stuff would be the way to go. Then anyone could download it, run it, and carry on without the hassle of reinstalling and so forth. No MFM has not been optimized. It has 125 Units with the stuff.
We have S3D to open the .dat file and show its objects and relationships but I don't think we have a .dat decompiler that can output some sort of file format that is readable to such a script. :hmmm:
JapLance
11-27-18, 04:23 AM
Water Reflections in most cases is the same.
No need for a separate 3D mesh.
It's been a long time since I tweaked with SH3/SH4 files, but I have to disagree here.
The water reflections don't need the full poly model to be displayed. A low-poly model can be safely used (not actually noticeable in the game) and lessen the charge on the GPU.
And I eagerly await for that program to convert SH3 models into SH4. I remember it was a painful, time consuming job to do.
bigboywooly
11-27-18, 05:22 AM
BBW,
Take a look at the NBB_Nevada in the SH4 sea folder.
(I can send it to you if you don't have SH4)
You might recall I tested this just after SH4 came out as We were putting the BuV_Dock1 and BuV_Dock2 into GWX.
Hmmm, have SH4. Will take a look
BigWalleye
11-27-18, 10:10 AM
Some thoughts from an old programmer:
Once, a very, very long time ago, I worked on the operating system for a computer which had exactly 4096 words of random access memory. That's not a misprint! Not K. Not M. Four thousand ninety-six 18-bit words. (A beer for anyone who can correctly identify that machine!) Every instruction had to be carefully chosen. Every status bit had to be considered for maximum added functionality. For the size of the code, it required a huge investment in programmer effort.
Later, I worked on a real-time virtual OS which, for sound physical reasons, was based on magneitc drum storage. The latency time of the slowly-turning drum meant that data and executable storage had to be carefully optimized to be under the heads when needed. There was software to help, but any significant change to data structures meant that the entire drum had to be remapped. Again, massive amounts of effort to overcome hardware limitations.
Throughout the history of computing, huge amounts of human programmer effort have been used to compensate for the limitations of the existing hardware. I'm sure Ada Lovelace had the same problems.
When SH3 was developed, the 3D models taxed the generally available hardware to the point where load times were barely tolerable. As SH3 grew through mods, much more extensive modelling made use of improving machine capabilities, and load times stayed barely tolerable. So we are today.
I am sure it would be possible, through large amounts of modder effort, to optimize the 3D models with standard elements. This would likely make a significant improvement in load times, especially for older hardware. (Would it also add significant render times and stress older CPUs and GPUs?) But is this the most effective use of the time and talents of a small modding community? The Polish Engineer sez: "If it ain't broke, don't try to fix it."
There are still many possible improvements to SH3 waiting to be done. In reverse enginnering, hsie did wonderful things, but much more is possible. (Compare to what TDW has done for SH5.) While the skill set for 3D modelling is not the same, still I suspect there are many useful projects to be done. (How about a complete interior?) I suggest that there are many projects which could use modder effort more effectively than cleaning up old inefficient models which do run adequately on current machines.
Truth in advertising: I am not an active modder, except for personal consumption. FWIW, this is just the opinon of a long-time software developer. YMMV.
And for all those who labor to make this old horse better and better every year, THANK YOU. Much love, and I will be grateful to use whatever you care to contribute to the community. God bless!
Jeff-Groves
11-27-18, 03:03 PM
We have S3D to open the .dat file and show its objects and relationships but I don't think we have a .dat decompiler that can output some sort of file format that is readable to such a script. :hmmm:
Don't need to decompile the dats.
I can read them in Hex as many of the old modders did from the start.
Some of us still know the format of the dats. :03:
Here's a little sample:
SH III DAT FILE FORMAT
Skip the first 20 bytes they seem to be always the same.
Next 4 bytes are a long with the header string lenght.
Next is the author name and the "using Kashmir" stuff, total lenght as read above.
Next, file is made of "chunks". All chunks are like this:
ChunkType 4 bytes
Chunk Subtype/Version 4 bytes
chunk_size long (4 bytes)
Data (chunk_size bytes)
Chunk types and internal structure of each one:
type 0: EMPTY chunk
it's just 12 "00" bytes, so type, subtype and size are 0
type 1: 3D chunk
after the Chunk type, subtype/ver and size, format of the data part is like follows:
chunk_id: 8 bytes //this is used to reference this part inside the file. References usually found inside NODE (type 4) chunks
010 can be scripted to do what ever one wants to do to the dats in that case.
S3D is a fine Tool! Many of us old modders contributed information during it's development.
But it has it's limits. Working with a great Hex editor excides what S3D can do.
Jeff-Groves
11-27-18, 03:10 PM
4096 words of random access memory. That wouldn't have been the 704's would it?
"I suggest that there are many projects which could use modder effort more effectively than cleaning up old inefficient models which do run adequately on current machines."
adequately
There's the word I hate.:03:
The program runs adequately as is but is in-efficient and can load faster, reduce possible ID conflicts, and allow the complete interiors to work better.
(They have been done for several Mods)
Jeff-Groves
11-27-18, 04:26 PM
And I eagerly await for that program to convert SH3 models into SH4. I remember it was a painful, time consuming job to do.
Check your PM's.
:03:
Your review of the program would be great!
ETR3(SS)
11-27-18, 04:37 PM
adequately
There's the word I hate.:03:
Better is the enemy of 'good enough.'
Jeff-Groves
11-27-18, 04:52 PM
Better is the enemy of 'good enough.'
Not sure which interpretation of this you intend?
:hmmm:
We've been running SH3 the 'good enough' way since 2005.
In my 40 year trade in construction? I learned 'good enough' was just a way to justify sub par work.
I do not accept 'good enough' when training our employees or doing the work we are hired to do.
ETR3(SS)
11-27-18, 05:23 PM
Sorry for leaving that ambiguous; it was an agreement with your statement on adequately.
I've run into more than my fair share of good enough and it drives me nuts, particularly when I'm the one that has to make it better.
Jeff-Groves
11-27-18, 05:30 PM
I've run into more than my fair share of good enough and it drives me nuts, particularly when I'm the one that has to make it better.
:haha:
Kind of thought so.
And I'm the one that gets sent 500+ miles from home to fix 'Good enough'.
:o
BigWalleye
11-27-18, 07:41 PM
On the day when we have all the time in the world and all the manpower in the world, we will not have to prioritize anything, and everything that needs to be done will get done. Until then, I personally believe adding new capabilities to SH3 is a better use of scarce modder time than cleaning up existing mods that work adequately. But that's just my opinion. As I pointed out, I am not modding, and what project any modder chooses to work on is a strictly up to them. I just wanted to lobby for considering an alternative.
My interpretation of "Better is the enemy of good enough." is the one endorsed by Winston Churchill, who had considerable experience in effectively prioritizing limited resources. (AFAIK, it was Churchill who coined that aphorism, or at least popularized it.)
And no, it was not a 704. The 704 used two different instruction formats, but each one was 36 bits long. And the addressing was 36-bit oriented. A hint: It was a mini-computer and the bulk storage device was called DECtape.
propbeanie
11-27-18, 10:20 PM
All he's talking about doing is removing un-used "code", if you will (bloatware), so the data will not only fit on the tape drive, but also be in a nice orderly manner... :O:
Re-using 3D meshes in a file.
I'll provide the example.
http://www.mediafire.com/file/4afldw66dvczbc8/LAA_AAMaunsell2.dat/file
Look at this dat with S3D. Then look at how the actual 3D meshes were done. Notice there are very few actual meshes used to complete the Sea Forts. The Game engine renders the needed parts as we placed them. No need for extra meshes!
At 2+ MB as is? Doing everything as separate meshes would place it at over 9 or 10 MB!
(Yes. They suffer the dreaded Dynamic shadow stuff! I know better now.)
Thanks for the offsets for the .dat file in hex very helpful.
So I understand the optimized approach you are talking about here, you are still copying the same model (obj) over and over again in S3D?
Isn't it possible to just reference the same model in S3D and just go about it that way, wouldn't that save space?
I guess, from a hex point of view, what edits would we want to make to these files to make them more optimized. Looks like looking for duplicate data chunks for models (type 1) is step 1, and then possibly deleting the duplicates and re-reference the deleted duplicate data chunks with the original model reference id?
@EDIT
Yes in S3D it is possible to reference the same model in multiple nodes and this works just fine. It also decreases file space.
Is it possible to reference a model ID that is in another .dat file?
Dirk Gently
11-28-18, 06:23 AM
Revelutionary find! A tool like that would not only open up more possibilities in terms of heavier expansion mods but also to make very high quality lightweight mods for non-gaming laptops.
Extraordinary find and plans, guys.
Jeff-Groves
11-28-18, 10:44 AM
Thanks for the offsets for the .dat file in hex very helpful.
So I understand the optimized approach you are talking about here, you are still copying the same model (obj) over and over again in S3D?
Isn't it possible to just reference the same model in S3D and just go about it that way, wouldn't that save space?
I guess, from a hex point of view, what edits would we want to make to these files to make them more optimized. Looks like looking for duplicate data chunks for models (type 1) is step 1, and then possibly deleting the duplicates and re-reference the deleted duplicate data chunks with the original model reference id?
@EDIT
Yes in S3D it is possible to reference the same model in multiple nodes and this works just fine. It also decreases file space.
Is it possible to reference a model ID that is in another .dat file?
The Sea Forts were built in a program much like S3D. It's called Jammu.
ref never released it because he didn't feel it was good enough (Suffered from some annoying bugs now and then) and did not want to support it nor anyone that may use it. It actually pre-dates S3D by several years!
I still use it as it has the ability to mass export ALL 3D objects and UV mappings in one click! (Then it crashes) :D
You can use the Placement node as one way to use objects from a different dat. Look at how Harbors and Cities are done.
On the format of dats? Look at one in S3D with the List view. That show how the dat is actually coded from node 1 to the end of file.
The Tree view makes it easier for you to understand the relationship of each node
bigboywooly
11-28-18, 11:00 AM
BBW,
Take a look at the NBB_Nevada in the SH4 sea folder.
(I can send it to you if you don't have SH4)
You might recall I tested this just after SH4 came out as We were putting the BuV_Dock1 and BuV_Dock2 into GWX.
The Nevada comes with the Uboat missions addon :doh:
Well I tried the line additions as in the Nevada cfg and left the M06A and M06X Sea folders the same as in the Nevada.
All 3 units show in mission editor, all 3 show in recognition manual but only the M06B shows in game :hmmm:
Museum CTDs when close to those merchants calling on the other 3d model
Back to the drawing board
Jeff-Groves
11-28-18, 11:07 AM
Jammu looks kind of like this.
https://i108.photobucket.com/albums/n12/privateer_2006/Capture002.jpg
Jeff-Groves
11-28-18, 11:08 AM
The Nevada comes with the Uboat missions addon :doh:
Well I tried the line additions as in the Nevada cfg and left the M06A and M06X Sea folders the same as in the Nevada.
All 3 units show in mission editor, all 3 show in recognition manual but only the M06B shows in game :hmmm:
Museum CTDs when close to those merchants calling on the other 3d model
Back to the drawing board
I'll change some files in my install.
bigboywooly
11-28-18, 11:10 AM
Jammu looks kind of like this.
https://i108.photobucket.com/albums/n12/privateer_2006/Capture002.jpg
Ooh the harbour editor
It did exist then :har:
bigboywooly
11-28-18, 11:11 AM
I'll change some files in my install.
If you need any files give me a shout, not sure what you have or dont have
Jeff-Groves
11-28-18, 11:51 AM
Ooh the harbour editor
It did exist then :har:
Yes it did. In a way. You couldn't change anything. Just view it.
ref never finished it and I lost his source code long ago!
:wah:
I still have his source code for Jammu and SH3 to SH4 but not being a Delhi programmer could never get them built.
I may try again at some point.
Jeff-Groves
11-28-18, 11:59 AM
I'll have to play with the usemodel thing. I know it worked before as you pointed out at the time that one dock was submerged. So we scrapped the idea. To bad the old hang out is gone. There was an unreal amount of knowledge there!
bigboywooly
11-28-18, 12:10 PM
I'll have to play with the usemodel thing. I know it worked before as you pointed out at the time that one dock was submerged. So we scrapped the idea. To bad the old hang out is gone. There was an unreal amount of knowledge there!
Yes there was, masses of very useful knowledge
Love the signature pic btw
Jeff-Groves
11-28-18, 12:15 PM
On the day when we have all the time in the world and all the manpower in the world, we will not have to prioritize anything, and everything that needs to be done will get done. Until then, I personally believe adding new capabilities to SH3 is a better use of scarce modder time than cleaning up existing mods that work adequately. But that's just my opinion. As I pointed out, I am not modding, and what project any modder chooses to work on is a strictly up to them. I just wanted to lobby for considering an alternative.
My interpretation of "Better is the enemy of good enough." is the one endorsed by Winston Churchill, who had considerable experience in effectively prioritizing limited resources. (AFAIK, it was Churchill who coined that aphorism, or at least popularized it.)
And no, it was not a 704. The 704 used two different instruction formats, but each one was 36 bits long. And the addressing was 36-bit oriented. A hint: It was a mini-computer and the bulk storage device was called DECtape.
I agree with you on prioritizing. IF you have a Team that is working on things. There's a Very Good Team doing that in SH4 now. Along with the Great SH3 mod Teams still going strong.
The ending of GWX4 may have been one of the greatest mistakes in SH history!
Tools were written to MASS export models to the SH4 format, Tools were under way to convert the Campaign files to SH4 format, The GWX Team had Jammu way before S3D came into being, (Dan even threatened to steal my NoteBook when shown a copy at the 2008 SubSim meet),
other planned changes to the files and such are mind blowing!
The impending release of SH5 and what was promised resulted in the biggest loss to SH4 and all players/modders then many even know about.
I'm good at a lot of things when it comes to how the files all work together.
How to cut file sizes and so on. Not just in SH3 but SH4 and SH5.
It's better I work that area since it's what I am interested in doing and know the file structures. So my time is well spent for every other modder to do his thing.
In the case of some interiors being done? They were taken from SH5 (In some cases)and put into SH3. I wrote an exporter that allowed modders to grab the 3D files! Was that a waste of my time? NOPE!
(And I did get a cease and desist over that little tool!)
:03:
So WAC has the Type II, Type VII, and Type IX.
One of the PDP's?
Ooh the harbour editor
It did exist then :har:
Digging...
This is the only editors I found in Maiks collection:
./RACERBOY (THE DARK WRAITH)/TDW_SDL_Editor_v1_2_0.zip
./SH3 TUTORIALS/HexEditor-Anleitung.pdf
./SH3 TUTORIALS/SH3_MISSION_EDITOR_EN.pdf
./SH3 TUTORIALS/sh3_missioneditor_v15.pdf
Don't see Harbour Editor anywhere.
(I used lftp to grep all files in Maiks collection)
Jeff-Groves
11-28-18, 01:57 PM
Digging...
This is the only editors I found in Maiks collection:
./RACERBOY (THE DARK WRAITH)/TDW_SDL_Editor_v1_2_0.zip
./SH3 TUTORIALS/HexEditor-Anleitung.pdf
./SH3 TUTORIALS/SH3_MISSION_EDITOR_EN.pdf
./SH3 TUTORIALS/sh3_missioneditor_v15.pdf
Don't see Harbour Editor anywhere.
(I used lftp to grep all files in Maiks collection)
It was never released. Only a few people ever saw the Alpha version.
I had a copy along with the source code years ago. I've since lost even a working version.
You probably only see Jammu mentioned in some files as it was never released.
If memory serves me right? It would not import 3D models.
We'd do that in a hex editor then check with Jammu.
Jeff-Groves
11-28-18, 02:10 PM
We had another program that would take an obj file and create a hex out put that we would cut and paste into a dat.
Yes. A lot of work! But it was the only way 10 years ago.
I bought my first NoteBook to do work with as I travelled a lot back then also.
I'd go back to the Motel rooms and edit files when the other Guys went to a Bar or something.
I have travelled with a NoteBook every since. I never know when an idea will hit me.
We had an SDL editor LONG before RacerBoy/TDW came around.
bigboywooly
11-28-18, 02:30 PM
No I never saw the harbour editor either, even tho it was for me primarily :har:
Good old Ref, wonder where he vanished to. Such a shame
You and he made a great team,
Jeff-Groves
11-28-18, 02:37 PM
No I never saw the harbour editor either, even tho it was for me primarily :har:
Good old Ref, wonder where he vanished to. Such a shame
I tried for years to contact ref. I still know where his real life website is.
(GOOD LORD! I'm a stalker!)
I do not know why he just left. He was the MASTER of SH files!
I do know S3D was something he gave so much info to create!
ref, DivingDuck, Anvart (My crazy Russian Friend), and myself.
BigWalleye
11-28-18, 04:03 PM
Those SH3 interiors, are they still around? FM's is the only one I know of.
Cease & Desist for an editor? Crazy!
PDP-4
Jeff-Groves
11-29-18, 07:22 PM
The Cease and Desist was due to taking advantage of a dll.
Can't say I blame them for that outlook.
Not sure who did or released the interiors ripped from SH5.
You could ask HanSolo78.
I have the versions in WAC now that I'm looking at to see if I can help reduce the file sizes.
s7rikeback
11-30-18, 07:16 PM
So removing such things is an advantage to any mod.
Hans pointed out it is a large amount of work IF done the old way by hand and editing every file one by one.
What if we run a program to scan the complete Sea folder and do all the work for us?
Would that be of interest?
Hell yeah!!! :salute:
s7rikeback
11-30-18, 07:28 PM
How to duplicate Units in SH3 the SH4 way:
Look at the BBNevada in SH4
ClassName=BBNevada
3DModelFileName=data/Sea/NBB_Colorado/NBB_Colorado
This type setting works in SH3 also!
Follow the files.
Oh yes one of Jeff's first teachings...
Been doing this for a long time now, for personnal use and for a add-on mod for FOTRSU, which gives us 20+ "cloned" ships, all having the Maru name attached for the Japan roster, while keeping the original files within their own rosters and with totally different names. Yet the clone shares the dat / sim / val / dsd / cam / zon of the model it's cloned from.
bigboywooly
11-30-18, 08:04 PM
Oh yes one of Jeff's first teachings...
Been doing this for a long time now, for personnal use and for a add-on mod for FOTRSU, which gives us 20+ "cloned" ships, all having the Maru name attached for the Japan roster, while keeping the original files within their own rosters and with totally different names. Yet the clone shares the dat / sim / val / dsd / cam / zon of the model it's cloned from.
In SH4 ?
Jeff-Groves
11-30-18, 08:27 PM
In SH4 it works fine.
I know damned well I had it working in SH3 at one point!
I'm still looking at what I had working and trying to duplicate.
If I can't get it working I don't have a problem saying.....
"Oh Crap!! I was wrong!"
:haha:
s7rikeback
11-30-18, 08:38 PM
In SH4 ?
Yes sorry. :oops:
Pm with files coming your way.
bigboywooly
12-01-18, 03:44 AM
In SH4 it works fine.
I know damned well I had it working in SH3 at one point!
I'm still looking at what I had working and trying to duplicate.
If I can't get it working I don't have a problem saying.....
"Oh Crap!! I was wrong!"
:haha:
I vaguely remember you mentioning it back in the past.
Have looked thru all my emails from then and can't find anything, must have been in the lair
bigboywooly
12-01-18, 04:28 AM
Yes sorry. :oops:
Pm with files coming your way.
Thanx, will take a look
ArnoldR
12-01-18, 04:10 PM
Oh yes one of Jeff's first teachings...
Been doing this for a long time now, for personnal use and for a add-on mod for FOTRSU, which gives us 20+ "cloned" ships, all having the Maru name attached for the Japan roster, while keeping the original files within their own rosters and with totally different names. Yet the clone shares the dat / sim / val / dsd / cam / zon of the model it's cloned from.
So, how is it working????? :damn:
s7rikeback
12-01-18, 07:28 PM
So, how is it working????? :damn:
Very well... :yeah:
ArnoldR
12-02-18, 09:28 AM
Very well... :yeah:
I mean....what's the mechanism??? :roll:
bigboywooly
12-02-18, 02:43 PM
I mean....what's the mechanism??? :roll:
Do you have SH4 ? In the uboat expansion the Nevada was added.
In the Nevada cfg in the sea folder is a line of text which points the game to look for another 3d model.
Colorado if memory serves.
There is no Nevada dat, sns etc. Game uses the Colarado model for that but with a skin dedicated to the Nevada.
2 ships, one 3d model.
The mod above uses same set up but with merchants.
Nowhere near PC at mo so this is from memory
ArnoldR
12-02-18, 03:54 PM
Do you have SH4 ? In the uboat expansion the Nevada was added.
In the Nevada cfg in the sea folder is a line of text which points the game to look for another 3d model.
Colorado if memory serves.
There is no Nevada dat, sns etc. Game uses the Colarado model for that but with a skin dedicated to the Nevada.
2 ships, one 3d model.
The mod above uses same set up but with merchants.
Nowhere near PC at mo so this is from memory
Thank you for the reply! So,you mean that both ships (original and cloned) are exactly the same (AA guns, hitpoints.....), except for the texture?
I don't have SH4, could you send me the files, or show me a screenshot?
Jeff-Groves
12-02-18, 04:13 PM
So. I dug out some old DVDs and found my Delphi stuff and installed it.
Since ref gave me his source code for the SH3 to SH4 converter?
I'm going to ATTEMPT to adjust that to just search the SH3 files and whack the dynamic shadow stuff. The converter has that ability so I hope to remove the fluff and just scrub SH3 dats in one click.
Freaking Delphi programmers! C or C++ is SO much easier for me!
:/\\!!
propbeanie
12-02-18, 04:54 PM
Ah yesh... good ole Pascal... glad I forgot ya... :O:
Jeff-Groves
12-02-18, 05:32 PM
Ah yesh... good ole Pascal... glad I forgot ya... :O:
Been YEARS since I played with Pascal. If I can cut out the interface stuff and such? I'd be happy. Tell it to whack all files in a folder structure and we should be good.
ArnoldR
12-02-18, 07:33 PM
How to duplicate Units in SH3 the SH4 way:
Look at the BBNevada in SH4
ClassName=BBNevada
3DModelFileName=data/Sea/NBB_Colorado/NBB_Colorado
This type setting works in SH3 also!
Follow the files.
In SH4 it works fine.
I know damned well I had it working in SH3 at one point!
I'm still looking at what I had working and trying to duplicate.
If I can't get it working I don't have a problem saying.....
"Oh Crap!! I was wrong!"
:haha:
So, we still don't know if this is working for SH3...???? :doh:
ETR3(SS)
12-02-18, 08:26 PM
Thank you for the reply! So,you mean that both ships (original and cloned) are exactly the same (AA guns, hitpoints.....), except for the texture?
I don't have SH4, could you send me the files, or show me a screenshot?So, the file structure for SH4 is different than it was for SH3 despite being the same game engine. The Colorado is the base model for several of the US battleships utilizing the same 3d model, texture, and damage model. Each individual entry differs in it equipment and sensors to provide it with the appropriate main, secondary, and AA weapons and sensors for the particular class of battleship. For example the Colorado had 8 16" guns while the Nevada had 10 14" guns.
From the Colorado:
[Equipment 1]
NodeName=M01
LinkName=16in_Double_base_US
StartDate=19380101
EndDate=19451231
From the Nevada:
[Equipment 1]
NodeName=M01
LinkName=14in_Triple_base_US
StartDate=19380101
EndDate=19451231
Jeff-Groves
12-03-18, 02:19 PM
One thing I do know.
I did get the SH4 way working at one time in SH3!
There is a difference I need to check out. I was running the original release with the No-StarForce patch at the time.
Maybe the newer version with no StarForce is the reason?
Would not surprise me if Ubi jacked that ability or the patch allowed it.
On the program to clean files of the Damages stuff?
I asked for help from a member here at Subsim who works with Delphi.
I did cut some stuff in Studio 2006 and it seems it should all work but it is so different from what I know that help is needed.
I'm not giving up. Just asking for help.
bigboywooly
12-03-18, 02:39 PM
Nothing wrong with asking for help. You can't know everything
Jeff-Groves
12-03-18, 02:48 PM
What was it Jim use to say?
None of us are as smart as all of us?
Or something like that.
bigboywooly
12-03-18, 04:32 PM
Indeed he did and he was right
Jeff-Groves
12-03-18, 05:06 PM
Indeed he did and he was right
Like we need to help his head grow larger!!
:03:
:haha:
Jimbuna
12-04-18, 07:55 AM
What was it Jim use to say?
None of us are as smart as all of us?
Or something like that.
Just one word short " None of us are as clever as all of us? :03:
Jeff-Groves
12-04-18, 03:31 PM
Just one word short " None of us are as clever as all of us? :03:
Then I didn't do to bad for a Colonist!
:up:
Jimbuna
12-05-18, 09:04 AM
Then I didn't do to bad for a Colonist!
:up:
Rgr that matey :salute:
Interesting thread Jeff, I am following it :up:
Thank you for the reply! So,you mean that both ships (original and cloned) are exactly the same (AA guns, hitpoints.....), except for the texture?
I don't have SH4, could you send me the files, or show me a screenshot?
This is an example taken from SH5, but it works the same in SHIV and probably also in SHIII (never tried in the latter game but I trust what Jeff says :03:)
File: data\Sea\NAMC_Patroclus\NAMC_Patroclus.cfg:
[Unit]
ClassName=AMCPatroclus
3DModelFileName=data/Sea/NAMC_Rawalpindi/NAMC_Rawalpindi
UnitType=13
MaxSpeed=15
Length=165.89
Width=21.39
Mast=39.69
Draft=9.00
Displacement=11198
DisplacementVariation=10
RenownAwarded=220
CrewComplement=40
SurvivalRate=70
SurvivalPercentage=40
RecManualCategory=TroopTransport
BowShape=Plumb
Funnels=2
ShowInRecognitionManual=No;it is a clone. the player will identify it as another single ship
;*********** THE END ************
Note the line marked in orange: the Patroclus has not 3D model of her own, but her cfg file is pointing to Rawalpindi 3D model (dat file, or gr2 file in this case, since we are talking about SH5).
Other files found in the same folder (besides the above cfg file):
NAMC_Patroclus_sil.dds
NAMC_Patroclus.eqp
NAMC_Patroclus.sns
All other (sim, zon, val, dsd, etc.) files are shared with the Rawalpindi. The two ships have most features in common, but textures, armaments and sensor outfits can be customized.
The external look of two ships having the same 3D model can be customized as well, to some extent: all you need to do is removing parts you want to customize from the main model, edit them in two or more variants, and storing them in an external library file whose content can be linked to the units sharing the said model through equipemnt nodes/bones and .eqp files. There are not examples of this type of advanced technique in stock SH5, but there is at least one in SHIV.
In general, this method is very useful for dealing with sister ships, similar ship classes and aircraft variants. We refer to it as "proxy-cloning" as opposed to simple "cloning" where you duplicate an unit with all her binary files :yep:
ArnoldR
12-05-18, 04:59 PM
@gap: thankx for the detailed explain!!
A question: do the proxy-cloned unit have a dat file?? If no, how can you call the new texture, you must have a different name, or not????????? :hmmm: :/\\!!
Jeff-Groves
12-05-18, 05:17 PM
So far it does not work in SH3 as it does in SH4 and SH5.
I know I had it working somehow in the past.
Trying to duplicate it now.
@gap: thankx for the detailed explain!!
A question: do the proxy-cloned unit have a dat file?? If no, how can you call the new texture, you must have a different name, or not????????? :hmmm: :/\\!!
It doesn't. The dat file is the one of the parent unit. Provided that unit materials are set opportunely, textures can be assigned via roster cfg file.
Still taking SH5's Patroclus and Rawalpindi as an example, these are their entries in the British roster:
AMCPatroclus.cfg:
[UnitClass]
ClassName=AMCPatroclus
UnitType=13
AppearanceDate=19390912
DisappearanceDate=19411001
DisplayName=BR AMC Patroclus (proxy)
[Texture 1]
TextureName=
LightmapTextureName=data/Sea/NAMC_Penguin/Penguin_AO.tga
NormalmapTextureName=
StartDate=19390101
EndDate=19461231
Frequency=1
[Unit 1]
Name=HMS Patroclus
DOC=19390912
DOD=19401104
[Unit 2]
Name=HMS Hector
DOC=19391220
DOD=19420405
[Unit 3]
Name=HMS Antenor
DOC=19400101
DOD=19411001
;************ The End ************
AMCRawalpindi.cfg:
[UnitClass]
ClassName=AMCRawalpindi
UnitType=13
AppearanceDate=19391001
DisappearanceDate=19430501
DisplayName=BR AMC Rawalpindi
[Texture 1]
TextureName=*mil.dds
LightmapTextureName=data\Sea\NAMC_Rawalpindi\NAMC_ Rawalpindi_AO01.dds
NormalmapTextureName=
StartDate=19390101
EndDate=19461231
Frequency=1
[Unit 1]
Name=HMS Rawalpindi
DOC=19391001
DOD=19391123
[Unit 2]
Name=HMS Ranchi
DOC=19391023
DOD=19430316
[Unit 3]
Name=HMS Ranpura
DOC=19391130
DOD=19430501
[Unit 4]
Name=HMS Rajputana
DOC=19391201
DOD=19410413
;************ The End ************
That means that each proxy-copy can have its own texture different from the one used on its parent unit, even if both ship classes are in the roster of the same nation. Also note that, unlike the example above, proxy-cloned units can have a different unit type than their parents; this is another level of customization that I forgot mentioning in my previous post.
All of the above applies to SHIV and SH5. In SHIII though, things might bit more complicated than that... read Jeff's post below:
So far it does not work in SH3 as it does in SH4 and SH5.
I know I had it working somehow in the past.
Trying to duplicate it now.
Fingers crossed and let us know if you remember how it works...
So far it does not work in SH3 as it does in SH4 and SH5.
I know I had it working somehow in the past.
Trying to duplicate it now.
Any news? :hmm2:
How to duplicate Units in SH3 the SH4 way:
Look at the BBNevada in SH4
ClassName=BBNevada
3DModelFileName=data/Sea/NBB_Colorado/NBB_Colorado
This type setting works in SH3 also!
Follow the files.
Hi All,
I have tried this over the last few days in many different flavors and I was unable to get this to work in SH3.
Has anyone had any success with this? This would be a breakthrough.
I've tried setting the model of a unit I know has the same model, then removing the original .dat file so I know its not using it. I just can't get the ship to show up.
Perhaps only implemented in SH4 and SH5? I do see the 3DModelFileName entry in many cfg files in SH3, but they only reference their own model.
Thanks.
HanSolo78
01-13-19, 04:43 PM
@Jeff
Any further ideas or programmes that could help optimize Sh3 and reduce memory usage?
Hi All,
I have tried this over the last few days in many different flavors and I was unable to get this to work in SH3.
Has anyone had any success with this? This would be a breakthrough.
I've tried setting the model of a unit I know has the same model, then removing the original .dat file so I know its not using it. I just can't get the ship to show up.
Perhaps only implemented in SH4 and SH5? I do see the 3DModelFileName entry in many cfg files in SH3, but they only reference their own model.
Thanks.
@Jeff
Any further ideas or programmes that could help optimize Sh3 and reduce memory usage?
Yep, I am curious too :yep:
ArnoldR
01-15-19, 01:38 PM
So, we still don't know if this is working for SH3...???? :doh:
Bump*:/\\!!
Bump*:/\\!!
I'm quite confident it doesn't work for SH3. The other approach we can use was suggested by Richard, it was to find similar 3d models in the dat files by looking at the hex and then reference just one of them (reference 1 id) instead of rendering the same model over and over again.
Jeff-Groves
01-23-19, 11:51 AM
I'll admit I don't recall how I had something like that working at one time.
I haven't been messing with it for awhile now also.
But here's a thought I had while working on something else.
Since SH3 Commander can move files and such around by dates?
Why not remove Units that are not in Game at the start of the War and have Commander install them at the proper time?
Also remove Units when needed?
Anvar1061
01-23-19, 12:46 PM
Since SH3 Commander can move files and such around by dates?
Why not remove Units that are not in Game at the start of the War and have Commander install them at the proper time?
Also remove Units when needed?
this is subject to mandatory use of SH3Commander
Jeff-Groves
01-23-19, 01:30 PM
this is subject to mandatory use of SH3Commander
Yes. But as the load time and system demands would be reduced?
It would be worth it for some. Maybe many.
Right now I'm looking at doing something like Commander for SH5 and maybe allow it to work with SH4.
Right now I'm looking at doing something like Commander for SH5 and maybe allow it to work with SH4.
THAT would be awesome !! :Kaleun_Applaud:
Jeff-Groves
01-23-19, 01:56 PM
It would not be as full of things like SH3 Commander but IF, (And that's a big IF!) I may make it so one could write plugins to expand it.
It's just a concept at the moment. Not even WIP
Anvar1061
01-23-19, 02:29 PM
Right now I'm looking at doing something like Commander for SH5 and maybe allow it to work with SH4.
http://www.subsim.com/radioroom/images/smilies/Kaleun_Wink.gif
http://www.subsim.com/radioroom/showthread.php?t=224584
http://www.subsim.com/radioroom/images/smilies/Kaleun_Wink.gif
http://brat01.mybb.by/viewtopic.php?id=178&p=4#p91293
"Hello comrades! I apologize for being lost. I have no free time at all. And there are no prerequisites for it to appear, rather the opposite. KSD Commander I can not support anymore. In principle, he would have worked independently. But the developers of IL2 BZS chopped off access to their server, which they once provided. I do not know what the reason is, to understand, to correspond with them is again a lot of time and not the fact that they will go to the meeting. I leave the link to the source code KSDCommander.https://yadi.sk/d/5vNE_vkt3UQu6p If there is someone who wants and can find and deploy a server, then everything will work again. Having source codes and desire it is possible and to develop the project. Everything is written in C # and the server part of the program requires a Windows server."
Jeff-Groves
01-23-19, 02:37 PM
I would NEVER do anything that required a server or logging in.
That may be needed for that program but I will not be going that way as I would not be working with those files.
I don't program in C# but I will look at the source code.
Thanks!
We do know that it is only necessary to load one 3d model to complete a full render of a more complex in Game unit.
I use the Maunsell Forts in GWX as an example. Open those with S3D and see how it was done. There are examples in the original release of SH3 so I did not invent the idea. The prospect was there from the start!
I just wanted to clarify on this point.
Can we reference a model ID that is outside of the dat file?
Because S3D will not be able to link it, but SH3 will know where to find it?
Jeff-Groves
01-25-19, 03:52 PM
Yes. That's how location files work for one example.
I looked. This is true, however in location files they are chunk type 11 not chunk type 4. I wonder If we can convert the type 4s in the unit files to 11s. And I believe the referenced models would have to be in the Library directory. As I think all the Location model references, the models live in the Library.
Jeff-Groves
01-25-19, 05:56 PM
Best way to find out is to try it.
Well if the type 11s can reference outside dat, perhaps the type 4s can too. But maybe they only look in Library?
I'll test both conditions.
Looks like the type 11s can't have any other node's attach to them, including materials.
Takes investigating.
So the Type 11s are referencing Type 4s in the Library. The 3d Models (type 1 ) are not being referenced. This confuses things more...
Would it be useful to switch the type 4s in the unit dirs (air, land, sea, submarine) to type 11s and reference a common type 4, even though that referenced type 4 may have its own attributes (ie: type 10s) and textures (type 2s)?
bigboywooly
01-26-19, 06:21 AM
So the Type 11s are referencing Type 4s in the Library. The 3d Models (type 1 ) are not being referenced. This confuses things more...
Would it be useful to switch the type 4s in the unit dirs (air, land, sea, submarine) to type 11s and reference a common type 4, even though that referenced type 4 may have its own attributes (ie: type 10s) and textures (type 2s)?
Referring to the nodes in the dat not type= ingame
vBulletin® v3.8.11, Copyright ©2000-2025, vBulletin Solutions Inc.