Reducing campaign layers = faster loading?

[GlobalExplorer]

Playing around with that new, lightweight parser that might be used for SHIV, I had an interesting idea:

What would happen if there was an external tool to reduce the campaign layers by removing all units that are outside the time period and geographical zone of the patrol? I'd guess you would end up with .mis files of only about 5-10 % the size of the complete ones, that I am sure of.

Now the question I ask myself are: Would this increase loading times significantly, or is the time used up for something else? And what would happen if the player loaded up a savegame .. would it be possible to manage the new campaign layers so that it would not cause problems on the users side??

This is basically a question to anybody with some experience with the campaign files. I just thought that theoretically it could be the starting point for a drastic reduction in loading times, the biggest problem I have with SHIII.

GE

[ref]

When I'm testing mods I use a set of clear campaign layers, the game loads a mission in about 20", compared to the 3/4 minutes with the full layers.

Ref

[GlobalExplorer]

Hi ref, 20 seconds vs 45 seconds or 20 sec vs 3-4 minutes??

And you mean you removed everything but the header and the game still loads?? I might try that to get an idea.

[ref]

20 sec Vs. 3/4 minutes.

Just open the mission editor and save as rnd/lnd/scr without adding anything, even campaign (didn't test it on purpouse, just forgot to disable them) starts at coords 0,0 of the map.

Ref

[Ducimus]

You can strip the RND and SCR layers totally devoid of anything and it will still load.

Personllay ive always thought (and i could be wrong) that, graphics aside; load time is a mix of number of units the game is loading (IE the sea and air directories) , plus the size of the RND/SCR layers.

[Mush Martin]

I did this for sh3 thats what MBFL mod does:up:

[bigboywooly]

Yeah your right Duce
Try removing the campaign files AND the Sea and Roster files

I have a light set of files I enable through JSGME that only has The German roster folder and that only has subs in
The sea and Air folders are empty and so are the Scr and Rnd
I leave the LND as thats what I am usually testing with that set up

My comp is old and slow but that set up flies in

Makes port set up a breeze

[Hartmann]

Academy missions load very fast compared with Single and campaign.

i was thinking delete some unvisited world areas like pacific, india, south africa.. and focus in atlantic, caribbean and med

[GlobalExplorer]

I did this for sh3 thats what MBFL mod does:up:

What is the MBFL mod? :hmm:

Now what I am suggestion is not as mod, but a program that you configure like:

Start: Month/Year
End: Month/Year
TOO: <area>

The program would the read all campaign files, discard all units outside the specs and write the reduced layer back into new files.

That would not be difficult, at least now with the kind of libraries I am using / developing.

But! I see lots of problems with loading of previous savegames. I have not done any testing of the implications and that's what I hope to find out through this thread:

a) Do the units in a campaign layer have to be in sequential order for SHIII to load?

b) If not, and if the index of a unit / group has changed in between save/load, will SHIII then crash because it cannot resolve the references?

c) will the reduction of the layers lead to a significant speedup of loading / that does justify such a program? (feedback so far indicates this might be the case)

Would anybody have an interest in such a program? I am not saying I will do it, but I can provide a c# library that does most of the work already (it makes parsing and writing the .mis files a breeze and is very fast).

GE

[JScones]

Now what I am suggestion is not as mod, but a program that you configure like:

Start: Month/Year
End: Month/Year
TOO: <area>

The program would the read all campaign files, discard all units outside the specs and write the reduced layer back into new files.

That would not be difficult, at least now with the kind of libraries I am using / developing.

But! I see lots of problems with loading of previous savegames. I have not done any testing of the implications and that's what I hope to find out through this thread:

a) Do the units in a campaign layer have to be in sequential order for SHIII to load?

b) If not, and if the index of a unit / group has changed in between save/load, will SHIII then crash because it cannot resolve the references?

c) will the reduction of the layers lead to a significant speedup of loading / that does justify such a program? (feedback so far indicates this might be the case)

Would anybody have an interest in such a program? I am not saying I will do it, but I can provide a c# library that does most of the work already (it makes parsing and writing the .mis files a breeze and is very fast).

GE
I must admit that I have written numerous tools for querying the mis files (have now got it where I can parse and perform calculations on 1,000,000 lines in mere seconds). I have always taken a "clean" approach and re-indexed to compensate for any removals that I have made.

Whether this is necessary, I have never tested. BUT, the ME does it so that's good enough for me. And many times I've removed one group or unit entry and caused no ill effect, although, I would guess if I had removed an entry that was referenced in the current mid-patrol save game, then well, it may be a different matter.

Thus I think if the programme simply removed the "out of scope" scripting everything *should* be OK. Save games seem to get more upset if units that are expected to be there at that time are not, so this means simply don't make changes mid-patrol.

Provided the programme makes no changes to the remaining entries, I can't see where there'd be problems. Just look at GWX Patch 1.02 - it made hundreds of mis file changes and most careers continued to work. The only careers that failed were those that had "pointers" to units or scripting that no longer existed at that time frame.

BBW or Rubini are prolly better placed to comment in this regard.

But my gut feeling is that it may cause more problems than it's worth (and this is assuming that removing entries creates no CTDs of its own) - to be blunt, some players have enough problems keeping up with JSGME and SH3Cmdr housekeeping. As the programme would need to maintain backups of the "pre-modified" mis files, there will be another set of backups (or at the least changes) for players to keep track of. All for what, one, two minute load gain? And what if a player dies and decides to load an earlier save? The reason behind the resulting CTDs may not be obvious to them at first (ie "D'oh, I forgot I was running x").

It's why I haven't released any of my SH3 tools to anyone other than a few people.

But just like JSGME and SH3Cmdr, if players still want such a tool taking this very likely risk into account, go for it - either way it's certainly an idea worth discussing. :up:

[Mush Martin]

I did this for sh3 thats what MBFL mod does:up:
What is the MBFL mod? :hmm:

Now what I am suggestion is not as mod, but a program that you configure like:

Start: Month/Year
End: Month/Year
TOO: <area>

The program would the read all campaign files, discard all units outside the specs and write the reduced layer back into new files.

That would not be difficult, at least now with the kind of libraries I am using / developing.

But! I see lots of problems with loading of previous savegames. I have not done any testing of the implications and that's what I hope to find out through this thread:

a) Do the units in a campaign layer have to be in sequential order for SHIII to load?

b) If not, and if the index of a unit / group has changed in between save/load, will SHIII then crash because it cannot resolve the references?

c) will the reduction of the layers lead to a significant speedup of loading / that does justify such a program? (feedback so far indicates this might be the case)

Would anybody have an interest in such a program? I am not saying I will do it, but I can provide a c# library that does most of the work already (it makes parsing and writing the .mis files a breeze and is very fast).

GE

MBFL is the simplest JSGME mod in history all it does is put in
blank scr rnd lnd layers to speed loading and prevent random
units in missions while its active.
it was intended to help those who for mission building or modelling
purposes need to frequently jump in and out of sh3 to tweak stuff

It was my first mod ever and is a mod for mod makers as opposed to
players. interestingly It came to me independently as opposed to
seeing it somewhere

I cant help but feel that theres an Irony in that my first modding experience
resulted in a reduced mission loading waiting time.

I guess it was what I wanted the most at the beginning

[GlobalExplorer]

I have the same gut feeling as JScones. Such a mod is possible but it would come with a lot of new problems of its own. Still wanted to discuss the idea, but as I see from comments like Mush Martin's the idea is not new at all.

JScones, just out of curiosity, I would like to know how you go about parsing the .mis files. What I am doing is offer a generic solution for reading ini files into memory and building a collection, which is simple and fast to access. The advantage is that this IniParser class can be used with any other SHIII file that is text based, because all follow the basic ini structure. Example:


IniParser parser=new IniParser("C:\\..\\SHIII\\data\\campaign\\...mis");

GenericIndexCollection<KeyValue> unit1 = parser.Sections["RndUnit638"]["Unit1"];

double latitude = (double)unit1["Lat"];


As the .mis files are ~10 MB, reading them into memory takes about 4-8 seconds, but after that access is in memory and extremely fast. I have also compared my solution against freely available alternatives like Nini.dll and - I still don't know why that is - it is up to 100 times faster. Possibly it is because from the beginning I went out with the aim of reading huge files and have optimized everything for that.

since this class might be useful for a lot of things in the SHIII world, if somebody is interested, he/she may contact me.

GE

[Ducimus]

Regarding the size of the RND/SCR layer combined with the contents of the Sea /air directories increasing load times, i just did a little experiment.

Size of Data/sea directory.
- stock has 59 objects in this directory (this includes cfg files).
- GWX has 143 objects in this directory (this also includes cfg files)

size of data/air directory
-stock has 15 objects (including cfg files)
- gwx has 38 objects (also including cfg files)

size of campaign_rnd.mis
- stock is 5.77 MB in size (or 6,056,257 bytes)
- GWX is 8.20 MB in size (or 8,609,569 bytes)

size of campaign_scr.mis
- stock is 2.58 MB (or 2,715,193 bytes)
- GWX is 6.97 MB (or 7,316,190 bytes - this is the HT light version)

----------------

Loading times, starting a new career in 1942, and loading a game

- stock was 53 seconds
- gwx was 2mins and 33 seconds.

[bigboywooly]

Yeah tis all relative
If you want faster loading times use stock with a limited number of units
Both air and ships

The more you add the longer it takes

Simple

The more new units released then those times will increase

Imagine adding Iambecomes 50+ ships into that mix

:o

[Ducimus]

One thing ive always wondered is if it effects time compression.

I mean, being the carribean/us east coast whore that i am, im a gross abuser of TC to get accross the alantic. Most i can push is 2048, and not all the time. I wonder if i could get away with 4096 under stock.

It's not important, just an idle curiousity. Ive come to realize i get more fun out of SH3 monkeying around with the files then i do acutally playing it. Kinda of a sad statement really.

edit:
nevermind, answered my own question... does having less stuff reduce ingame lag with TC? Not really.

[bigboywooly]

Ive come to realize i get more fun out of SH3 monkeying around with the files then i do acutally playing it. Kinda of a sad statement really.



Hear that
If it wasnt for testing I wouldnt see the inside of the game at all

Still , soon be time to stop and play

[Ducimus]

Just an off the wall comment. If a player was to concenrate their playing in:

- the med/black sea, one could remove 98% of the traffic outside the med in the RND and SCR layers and run this as a JSGME plugin. Theoritically reducing load times.

- the indian ocean, same thing. One could remove.. (in this case), 95% of the traffic outside of the indian ocean in the RND/SCR layer and run it as a JSGME plugin. theortically reducing load times.


This wouldnt work for the north/south atlantic, carribean or us eastern seaboard as everything is interconnected there.

[Rubini]

Edited: Ducimus, I don´t read your post above (we post at same time). My idea here is exactly the same. Again...:D
Cheers Mate!
=======================================

The problem in reducing for example the traffic in Med when using Atlantic campaign is that some convoys could start in the Med. It´s not so frequent but it´s a possibility
So what is possible is to "delete" really outside areas like far east when in Altantic career. But not the Med. (as an example)

But if someone makes a detailed work over the campaigns the result could be very good in reducing loading times.

The saved games will only work with the exact same campaign files. It´s a locked question for Sh3.

GWX really have a lot of new units and a huge Campaign. The loading times is a bit so much really...

And another idea could be to divide GWX SCR and RND in two or three parts and use it as needed. But this approach probably will work only for modders or others expert, not for the public in general.
Yes, it´s a good idea. I will try this for my personal use.:D

Rubini.

[Mush Martin]

of course its possible but youd need to fabricate the layers and assumably manually put them in. I dont know of a way to make a program that
would read your campaign geographical location and insert the appropriate
layering.:hmm:

for me MBFL works great for what it is and I dont use it for campaign as it wouldnt work
so I just accept the loading time I did think of localized campaign layers when I thought
up MBFL but I couldnt be bothered.

[JScones]

Guys, just a process check here.

- stock was 53 seconds
- gwx was 2mins and 33 seconds.

suggests that between bare stock and full GWX is a difference in loading of 1:40 (all things being equal etc).

Is getting overly fancy really worth a 1:40 saving? Seriously? And that's between stock SH3 and GWX - remember the resulting GWX files would still prolly be more populated than stock files due to the number of Operations added to each theatre.

Also remember that any extra file versions result in backup and maintenance requirements for users, and version integrity problems for developers. Need I remind about map name labels, for example?

1:40 saving v "Why does my game CTD?"... :hmm:

One can get too fancy. :yep:

[Ducimus]

Is getting overly fancy really worth a 1:40 saving? Seriously?

Ive asked myself that same question a number of times. For me i think its just a pet peeve. Like alot of things that relate to.. i dunno, anything in the IT world (how my system runs, how such programs run etc) im a fan of lean, and highly efficient - more bang for teh buck with less resources consumed. I think its that same fetish that extends into Sh3 in my case.

At any rate i exerimented with cutting the campaign layers down this morning. I went through and painstakingly by hand went through and eleminated all traffic in both the SCR layer and the RND layer that did not cross into the med. Then loaded a game in the med. It acutally took a few seconds longer. Maybe i had some free floating waypoints or something, or some other unresolved calls in the files, but im pretty sure i got everything. Not a good test really since being close to land acutally increases load times. But the point is - didn't work.

I fear the only way to really cut down load times, is simply to reduce the amount of /sea and air units the game tries to load - which is also, ALOT of damn work to weed them out. But here i degrees, in another month, i wont care one way or the other once i get my grubby mitts on SH4.

[GlobalExplorer]

I have written a small routine that reads in Campaign_RND.mis, removes all entries outside the time frame and writes back a new file.

Full file was GWX1.02 about 8MB - the reduced file is 1.44 MB. Further reduction is of course possible by checking if groups are in surrounding theaters of operation for the patrol, then you should end up with a much smaller file.

Personally I don't have the time to write the full program, but my hope is that somebody would want to take over from that. Implementing the out of scope routine is one thing that is easy, solving all the problems with different campaign files, adding a gui, testing etc, another.

If somebody want to try loading the reduced file, it can be downloaded from here (http://www.christian-wendt.org/Campaign_RND.exe). Though I assume it will CTD as I have not renumbered the groups, but that would not be difficult.

Attached is the example - that is all that is required to process the Campaign_RND layer.

But it looks like this idea is going nowhere. I am not going to do anything more in this direction myself ..



http://www.christian-wendt.org/example_rnd.gif

[bigboywooly]

Interesting GE

Though not sure how SHIII will handle changeover points
FE if you made the campaign for 2 months and then spent 5 weeks on patrol
When you return to base you will have 4 weeks in port - longer if upgrading or repairs
That would be outside your original 2 month campaign

Now if you were to run another 2 month Rnd off I am not sure if the game looks back at previous files when loading

After updating to GWX 1.02 a lot of careers didnt work after the campaign file changes

Worth looking at though
Personally loading times dont bother me

[Ducimus]

Just an FYI, my previous test with a reduced layer where i said it made no difference.

I found out later i misnamed the campaigns directory in the test mod i was using so the test files were never applied :roll:

Shame i deleted it in disgust. lol.