Reducing campaign layers = faster loading?
 

Playing around with that new, lightweight parser that might be used for SHIV, I had an interesting idea:

What would happen if there was an external tool to reduce the campaign layers by removing all units that are outside the time period and geographical zone of the patrol? I'd guess you would end up with .mis files of only about 5-10 % the size of the complete ones, that I am sure of.

Now the question I ask myself are: Would this increase loading times significantly, or is the time used up for something else? And what would happen if the player loaded up a savegame .. would it be possible to manage the new campaign layers so that it would not cause problems on the users side??

This is basically a question to anybody with some experience with the campaign files. I just thought that theoretically it could be the starting point for a drastic reduction in loading times, the biggest problem I have with SHIII.

GE

When I'm testing mods I use a set of clear campaign layers, the game loads a mission in about 20", compared to the 3/4 minutes with the full layers.

Ref

Hi ref, 20 seconds vs 45 seconds or 20 sec vs 3-4 minutes??

And you mean you removed everything but the header and the game still loads?? I might try that to get an idea.

20 sec Vs. 3/4 minutes.

Just open the mission editor and save as rnd/lnd/scr without adding anything, even campaign (didn't test it on purpouse, just forgot to disable them) starts at coords 0,0 of the map.

Ref

You can strip the RND and SCR layers totally devoid of anything and it will still load.

Personllay ive always thought (and i could be wrong) that, graphics aside; load time is a mix of number of units the game is loading (IE the sea and air directories) , plus the size of the RND/SCR layers.

I did this for sh3 thats what MBFL mod does:up:

Yeah your right Duce
Try removing the campaign files AND the Sea and Roster files

I have a light set of files I enable through JSGME that only has The German roster folder and that only has subs in
The sea and Air folders are empty and so are the Scr and Rnd
I leave the LND as thats what I am usually testing with that set up

My comp is old and slow but that set up flies in

Makes port set up a breeze

Academy missions load very fast compared with Single and campaign.

i was thinking delete some unvisited world areas like pacific, india, south africa.. and focus in atlantic, caribbean and med

What is the MBFL mod? :hmm:

Now what I am suggestion is not as mod, but a program that you configure like:

Start: Month/Year
End: Month/Year
TOO: <area>

The program would the read all campaign files, discard all units outside the specs and write the reduced layer back into new files.

That would not be difficult, at least now with the kind of libraries I am using / developing.

But! I see lots of problems with loading of previous savegames. I have not done any testing of the implications and that's what I hope to find out through this thread:

a) Do the units in a campaign layer have to be in sequential order for SHIII to load?

b) If not, and if the index of a unit / group has changed in between save/load, will SHIII then crash because it cannot resolve the references?

c) will the reduction of the layers lead to a significant speedup of loading / that does justify such a program? (feedback so far indicates this might be the case)

Would anybody have an interest in such a program? I am not saying I will do it, but I can provide a c# library that does most of the work already (it makes parsing and writing the .mis files a breeze and is very fast).

GE

I must admit that I have written numerous tools for querying the mis files (have now got it where I can parse and perform calculations on 1,000,000 lines in mere seconds). I have always taken a "clean" approach and re-indexed to compensate for any removals that I have made.

Whether this is necessary, I have never tested. BUT, the ME does it so that's good enough for me. And many times I've removed one group or unit entry and caused no ill effect, although, I would guess if I had removed an entry that was referenced in the current mid-patrol save game, then well, it may be a different matter.

Thus I think if the programme simply removed the "out of scope" scripting everything *should* be OK. Save games seem to get more upset if units that are expected to be there at that time are not, so this means simply don't make changes mid-patrol.

Provided the programme makes no changes to the remaining entries, I can't see where there'd be problems. Just look at GWX Patch 1.02 - it made hundreds of mis file changes and most careers continued to work. The only careers that failed were those that had "pointers" to units or scripting that no longer existed at that time frame.

BBW or Rubini are prolly better placed to comment in this regard.

But my gut feeling is that it may cause more problems than it's worth (and this is assuming that removing entries creates no CTDs of its own) - to be blunt, some players have enough problems keeping up with JSGME and SH3Cmdr housekeeping. As the programme would need to maintain backups of the "pre-modified" mis files, there will be another set of backups (or at the least changes) for players to keep track of. All for what, one, two minute load gain? And what if a player dies and decides to load an earlier save? The reason behind the resulting CTDs may not be obvious to them at first (ie "D'oh, I forgot I was running x").

It's why I haven't released any of my SH3 tools to anyone other than a few people.

But just like JSGME and SH3Cmdr, if players still want such a tool taking this very likely risk into account, go for it - either way it's certainly an idea worth discussing. :up:

MBFL is the simplest JSGME mod in history all it does is put in
blank scr rnd lnd layers to speed loading and prevent random
units in missions while its active.
it was intended to help those who for mission building or modelling
purposes need to frequently jump in and out of sh3 to tweak stuff

It was my first mod ever and is a mod for mod makers as opposed to
players. interestingly It came to me independently as opposed to
seeing it somewhere

I cant help but feel that theres an Irony in that my first modding experience
resulted in a reduced mission loading waiting time.

I guess it was what I wanted the most at the beginning

I have the same gut feeling as JScones. Such a mod is possible but it would come with a lot of new problems of its own. Still wanted to discuss the idea, but as I see from comments like Mush Martin's the idea is not new at all.

JScones, just out of curiosity, I would like to know how you go about parsing the .mis files. What I am doing is offer a generic solution for reading ini files into memory and building a collection, which is simple and fast to access. The advantage is that this IniParser class can be used with any other SHIII file that is text based, because all follow the basic ini structure. Example:


IniParser parser=new IniParser("C:\\..\\SHIII\\data\\campaign\\...mis") ;

GenericIndexCollection<KeyValue> unit1 = parser.Sections["RndUnit638"]["Unit1"];

double latitude = (double)unit1["Lat"];


As the .mis files are ~10 MB, reading them into memory takes about 4-8 seconds, but after that access is in memory and extremely fast. I have also compared my solution against freely available alternatives like Nini.dll and - I still don't know why that is - it is up to 100 times faster. Possibly it is because from the beginning I went out with the aim of reading huge files and have optimized everything for that.

since this class might be useful for a lot of things in the SHIII world, if somebody is interested, he/she may contact me.

GE

Regarding the size of the RND/SCR layer combined with the contents of the Sea /air directories increasing load times, i just did a little experiment.

Size of Data/sea directory.
- stock has 59 objects in this directory (this includes cfg files).
- GWX has 143 objects in this directory (this also includes cfg files)

size of data/air directory
-stock has 15 objects (including cfg files)
- gwx has 38 objects (also including cfg files)

size of campaign_rnd.mis
- stock is 5.77 MB in size (or 6,056,257 bytes)
- GWX is 8.20 MB in size (or 8,609,569 bytes)

size of campaign_scr.mis
- stock is 2.58 MB (or 2,715,193 bytes)
- GWX is 6.97 MB (or 7,316,190 bytes - this is the HT light version)

----------------

Loading times, starting a new career in 1942, and loading a game

- stock was 53 seconds
- gwx was 2mins and 33 seconds.

Yeah tis all relative
If you want faster loading times use stock with a limited number of units
Both air and ships

The more you add the longer it takes

Simple

The more new units released then those times will increase

Imagine adding Iambecomes 50+ ships into that mix

:o

One thing ive always wondered is if it effects time compression.

I mean, being the carribean/us east coast whore that i am, im a gross abuser of TC to get accross the alantic. Most i can push is 2048, and not all the time. I wonder if i could get away with 4096 under stock.

It's not important, just an idle curiousity. Ive come to realize i get more fun out of SH3 monkeying around with the files then i do acutally playing it. Kinda of a sad statement really.

edit:
nevermind, answered my own question... does having less stuff reduce ingame lag with TC? Not really.