Why the load time?

[Commander Gizmo]

After my recent attempt to get SH3 running just right with GWX 2.1, I definatly began to hate watching that progress bar run across the screen at app startup. That begs the question, just what is it loading there? I know it is loading the menu GUIs, but obviously there is more going on. A few questions arise:

• Why is there no speed loading mod widely used?
• Does the menu backgroun movie (a bik, I think) load, or just stream from file?
• Can we effect what is loaded at app launch and what is loaded at game start?
• Why is the artwork for the loading screens much larger than the screen area it fills?
• Why are so many blank textures not incredibly small, or just reused from the same file?
• Does the game allow modders to choose what sections of what files are used where on the screen?
• Can the modder choose tiling or stretching of a texture piece to fill an area of the screen?
• Has anyone attempted to optimize the loading time?

Since the GWX team has moved on and this game appears to be reaching some sort of modding maturity plateau, this seems like a good time to visit this topic and perhaps clean up a few things. Looking at the texture and config files, it looks like a real mess with terrible efficiency. No offense to any of the great modding teams that have worked so hard to make me love this game so much! Most of the mods stick pretty close to the stock file naming system.

[CCIP]

The GWX team hasn't moved on! See this:

http://www.subsim.com/radioroom/showthread.php?t=143272

As for the loading times, the real reason for them is the much more sophisticated campaign that most modern mods use (with much-improved variation and realism in shipping) - which, due to the way the engine was designed, really pushes it. The greatly increased amount of units and other objects is also a culprit. When you start the game, it loads ALL traffic and units that are present in the game's entire world, regardless of whether you'll see them or not. In current mods like GWX, there is exponentially more content to load, which is why it takes so long.

The only way to cut down the loading times is simplifying the campaign - which is why separate Black Sea file is included for example, so that it only loads campaign traffic for the Black Sea.

SHIV had picked up on that idea and re-designed the engine to load traffic in a "smart" way that avoids loading the entire world at the same time and instead plugs in chunks as you proceed from area to area, and that is why in SHIV the loading times are just a small fraction of what you see in SHIII.

There's no way to change it in SHIII if you want to keep the complex campaign and unit variety. It's not a big price to pay for mods like GWX, to be honest!

[TarJak]

The actual reason behind the long load times is explained in the GWX manual. The SHIII game engine loads everything (and I mean everything) in one go at the beginning when you start the game. SHIV does it differently and unless the devs revisit SHIII the load times are what they are. I normally go make a cuppa, read a good book, play with the kids, mow the lawn and then the game is ready to play.

To specifically respond to your questions:

• Why is there no speed loading mod widely used? One does not exist nor can it be made.
• Does the menu backgroun movie (a bik, I think) load, or just stream from file?Everything loads at start up.
• Can we effect what is loaded at app launch and what is loaded at game start?No the load sequence is hard coded by the devs.
• Why is the artwork for the loading screens much larger than the screen area it fills?I'm not sure what you mean. Mine fits perfectly.
• Why are so many blank textures not incredibly small, or just reused from the same file?Again I'm not sure what you mean. Can you give an example?
• Does the game allow modders to choose what sections of what files are used where on the screen?Yes and no. It depends on whether it is an object created by a modder or not. On some objects it would be possible. On others it would not.
• Can the modder choose tiling or stretching of a texture piece to fill an area of the screen? Again it depends on the object in question.
• Has anyone attempted to optimize the loading time? Yes GWX spent many hours attempting to make this as optimised as possible, hwoever given the way the game works it is what it is.

There is one way to reduce the load times when playing a Black Sea or Indian Ocean campaign and that is to load the campaign files for these theatres instead of loading all theatres.

[Commander Gizmo]

First of all: GWX IS BACK??!!!

AWESOME!!!

Ok, now a few responses:

• Why is there no speed loading mod widely used? One does not exist nor can it be made. That sounds rather definate. Do modders have control over which file is referenced for each graphic, or are some/most hard coded?
• Does the menu backgroun movie (a bik, I think) load, or just stream from file?Everything loads at start up. Cutting a 40MB video could shorten load time somewhat, yes?
• Can we effect what is loaded at app launch and what is loaded at game start?No the load sequence is hard coded by the devs. Then what loads at game start (like starting a mission)? That takes even longer.
• Why is the artwork for the loading screens much larger than the screen area it fills?I'm not sure what you mean. Mine fits perfectly. The loading images in GWX are 1024x1024 pixels even though they are shown in an area of 1024x768. Cutting back on those pixels would make the files smaller (since TGA's are not compressed). This makes me wonder why, and were else this occurs.
• Why are so many blank textures not incredibly small, or just reused from the same file?Again I'm not sure what you mean. Can you give an example? For example, the file Back00.tga is 512x512 pixels, though I can't find anything different between any of the pixels, one from another. Again, it seems a waste of data size. If this occurs often, it could add up to a sizable amount of load time.
• Does the game allow modders to choose what sections of what files are used where on the screen?Yes and no. It depends on whether it is an object created by a modder or not. On some objects it would be possible. On others it would not. The purpose of this question is to acertain whether multiple spirtes/textures, could be derived from a single file. For instance, back000.tga could be used for all space that conforms to its color/transparency by 'cropping' a section of the texture and using it here or there instead of multiple images. Another example might be how map.tga and singlemission.tga are both the same file (from what I can tell), just loaded twice.
• Can the modder choose tiling or stretching of a texture piece to fill an area of the screen? Again it depends on the object in question. This is really just part of the above question.
• Has anyone attempted to optimize the loading time? Yes GWX spent many hours attempting to make this as optimised as possible, hwoever given the way the game works it is what it is. I greatly appreciate their efforts. I'll bet when they first loaded it up it took much, much longer. I certainly don't envy their long hours of testing just staring at that darn progress bar plodding along.

While I realize that keeping the file names and sizes the same certainly makes a mod far more likley to work with other mods, I see some possible potential for a great deal more optimization if such a restriction were not needed. Perhaps the game engine is simply hard coded with too many file names and such to make it possible. Being the anal efficiency natzi that I am, I feel like possibly giving it a go.

What if GWX 3.0 Gold had an optimized version with black and white menu graphics (or even none at all save the text) and all the files cleaned up? I suppose I might be able to do such a thing for myself if I wanted since I don't see any reason the GWX team should spend long hours remaking what they've already made just to satisfy me. Although, I would probably need to be given a few details on what is hardcoded.

[danlisa]

Originally Posted by Commander Gizmo

After my recent attempt to get SH3 running just right with GWX 2.1, I definatly began to hate watching that progress bar run across the screen at app startup. That begs the question, just what is it loading there? I know it is loading the menu GUIs, but obviously there is more going on. A few questions arise:

Hi.

Obviously you are referring to both loading sections, first after clicking the SH3 shortcut and then loading a campaign or save file.

The first load bar: Not much can be done to speed this loading. Really the only thing that will affect it is PC Specs. At this point the game loads it's code into your PC's memory, that's about it. IIRC, at this stage SH3 uses around 400mb of system memory. Regardless of menu layout, movie format/size or texture size the only way to increase performance here is to have enough system RAM and a killer processor. This is limited by the architecture of SH3's coding.

Second loading bar (campaign/save file): Believe it or not, GWX has optimized this as best as it can while keeping historical accuracy and content. Lets, go back in time to when Stock SH3 was first released.....

SH3 had a minimal campaign structure, small amount of content and actually handled game objects in a very different way. This handling of objects is the contentious point in your request. I'll try to explain.

In stock SH3, you had the object DAT file which basically contained the model (and other bits) and then the textures for that object were contained in and called from the Low/High Res textures folder. Now here's the killer, these textures were called & loaded each time you encountered (or within rendering range) each object while playing the game, this caused HUGE freezes & slowdowns when you least wanted them. GWX remastered this routine. By placing every objects texture inside it's DAT file, it meant that the game pre-loads this information before you even see the inside of your boat, thus improving ingame performance. Texture files were also reduced in resolution to help this. So you see, there was a tradeoff, smooth gameplay/accurate historical representation/additional content etc vs. Stock SH3.

A quick comparison:

Stock 1.4b:
Air Units = 13
Sea Units = 49
U-Boats+Subs = 8
SCR Campaign = 19249 lines
RND Campaign = 402082 lines
LND Campaign = 11968 lines
Total = 433299

GWX 2.0: (not 2.1)
Air Units = 40
Sea Units = 182
U-Boats+Subs = 24
SCR Campaign = 486859 lines
RND Campaign = 629050 lines
LND Campaign = 53925 lines
Total = 1169834

Thats a 169.98% increase in campaign scripting alone. Don't forget, when you click START, all the items above are loaded each & every time into your PC's memory.

All of that said, I have no doubt that if the original game routines were still in place, GWX would not work. To bring that level of content the core of the game had to be changed.

On to your original questions (mine in red):

• Why is there no speed loading mod widely used? There is no way to implement this without loosing something.
• Does the menu backgroun movie (a bik, I think) load, or just stream from file? Stream.
• Can we effect what is loaded at app launch and what is loaded at game start? Nope, this is limited by game design and coding.
• Why is the artwork for the loading screens much larger than the screen area it fills? Again, the loading artwork is 1024 x 1024 because it needs the Black areas above & below the main image for positioning.
• Why are so many blank textures not incredibly small, or just reused from the same file? There are no BLANK textures in GWX or SH3. They have Alpha channels which displays them as blank when viewing them as thumbnails in explorer.
• Does the game allow modders to choose what sections of what files are used where on the screen? To a point but this will not help to improve performance. The easiest example of this is the Menu_1024_768.ini. This file will extract parts of a single texture (by given coordinates) and then place it on screen, again given coordinates.
• Can the modder choose tiling or stretching of a texture piece to fill an area of the screen? Again, to a point. Tiling = No but you can stretch a texture by use of the menu.ini.
• Has anyone attempted to optimize the loading time? Yes, GWX has. Seriously, if you want to experience the dramatic differences, create a empty set of campaign files and test the differences. Bottom line is, you can optimise as much as you like but what do you removes to get the added performance?

[Commander Gizmo]

Thanks for the responses. I don't mean to annoy anyone with my questions. Just to clarify, I am an experienced coder who has written some casual games of his own in multiple lanuages, and used Photoshop for graphics testing/looking. I did check the alpha channels.

I thought there was some confusion about what loading screen I was referring to, though I did try to make it as clear as possible. Thank you for answering that part of the question. As a coder, I find it hard to believe that the game's code requires 400MB of memory. I would be more inclined to believe that a great deal of that is base data for much of the objects in the game, as well as some artwork for getting around the menu system outside of an actual mission. Or perhaps that is what you meant. Either way, if this data is hard coded to load at application launch (why devs, why?), I can see why that would be difficult to further optimize.

Since I am mostly interested in optimizing the initial app launch load screen, perhaps there is indeed nothing I can do to further my interest here. I do however have a few further clarifications to request if you don't mind answering them:

If there is data in back000, I'd like to know what and how to see it for my own modding purposes.

I still don't quite understand why the loading screens are 1024x1024. Is it a hard coded thing the devs through in to put blank/black space at the bottom of the image that would not appear on screen? What purpose does it serve? I am probably just missing something painfully obvious.

[danlisa]

Originally Posted by Commander Gizmo

As a coder, I find it hard to believe that the game's code requires 400MB of memory. I would be more inclined to believe that a great deal of that is base data for much of the objects in the game, as well as some artwork for getting around the menu system outside of an actual mission. Or perhaps that is what you meant. Either way, if this data is hard coded to load at application launch (why devs, why?), I can see why that would be difficult to further optimize.

You can test this. Run SH3 in windowed mode and have Task Manager open at the same time. Record the load at the menus and then record the load at campaign/save start. It doubles at least. Yes, we asked the Devs the same question.:rotfl:

Originally Posted by Commander Gizmo

If there is data in back000, I'd like to know what and how to see it for my own modding purposes.

I still don't quite understand why the loading screens are 1024x1024. Is it a hard coded thing the devs through in to put blank/black space at the bottom of the image that would not appear on screen? What purpose does it serve? I am probably just missing something painfully obvious.

IIRC back000.tga is the main menu graphics screen. If so, this file is 512 x 512 and it gets stretched to fit 1024 x 768 by the menu_1924_768.ini. You can find the settings for this by opening the menu.ini and searching for the tgs file name.

Regards the loading screen being 1024 x 1024, yes this is an oddity and there many other screens which use the same format, I think back000.tgs is the only one that doesn't. I see no reason why you could not apply the same properties that the main menu uses. However, the menu.ini only likes square ratios, so 1024 x 768 would not work and the ini would scale it up automatically. Maybe that's the bit you're missing. Infact, in some intances SH3 will completely ignore non compliant ratios and default to the original. Trust me, I tried. I wanted to stop the upscaling of the menu background but ran out of time. I think it looks ugly and pixelated. Although, my copy doesn't.

EDIT - If the back000 file is the background movie, to view this etc you'll need BINK - http://www.radgametools.com/

[CCIP]

Well, as for your comments regarding cropping the image files and video - those account for a very negligible amount of data that any modern PC that can run SHIII will really be unaffected. 99.9% of the data SHIII loads during the loading phase is things such as 'building' the 3D gameworld, generating traffic, and pre-loading objects - things that are associated with the CPU and RAM. Anything relating to textures and other real-time graphics is related to the GPU and have negligible effect on the CPU/RAM loads if you have a modern video card. If textures were too large, they would sooner start affecting your frame-rate in game than the loading times as such.

Again, the real answer to maximizing loading efficiency lies in what the game's developers did for SHIV. If you have SHIV, take a look in the campaign folder - you'll see that the campaign is split up into dozens of smaller files which are loaded selectively based on date (and also I believe location) each time the game starts up. In SHIII, on the other hand, you have only 3 files accounting for the entire campaign which always have to be loaded, with one of the files (Campaign_RND.mis) accounting for the vast majority of the traffic you see in the game. It's an incredibly complex file of nearly 5mb of text entries containing information on each randomly-generated waypoint and unit in the game. All of these units and waypoints have associated probabilities of appearance and random radii of appearance. So think of it as the program parsing 5,000 pages of text line by line during loading, and for each line generating a set of ships or convoys, determining their formation and route, resolving conflicts, spacing them properly based on the variables set by the scenario etc. etc. It's a really complex operation, and there is no way around it in SHIII, unless you simply cut off parts of the campaign.

[Hanomag]

Originally Posted by TarJak

The actual reason behind the long load times is explained in the GWX manual.

Theres a manual?? Why don't people tell me these things???!!

[Philipp_Thomsen]

You can make GWX campaign loading times cut back to something like 20 seconds.

Take four raptor hds and plug them together in raid 0.

Already tested it, it's incredible.

[danlisa]

He's just upgraded to SSD's. This thread is now pointless.:rotfl:

[kylania]

I just recently upgraded to a new hard drive, Seagate Barracuda 7200.10 (Perpendicular Recording), and my GWX 2.1 + SH3 Commander w/Rollback loading a save game from 1940 near a convoy =

From SH3 Commander Start Game Button to Main Menu usable = 48 seconds
From Yes, Load Save Game to 3D interior viewable = 2 minutes 19 seconds

This was a huge difference from my old hard drive which often took 4-5 minutes to load a game. Rest of the PC is a P4 2.13GHz, 2GB of RAM.

[Commander Gizmo]

Thanks for the replies. I now have a much better understanding of the SH3 engine and data structure. I figured it had been tried, just thought I'd see more discussion on it in a search.

A bit off topic, but I went with OCZ for my SSD's since they're memory almost always out performs the specs they list with it. The drives do quite well. At this point, a complete system restart of Vista x64 Ultimate with all may dev tools and software installed is 62 seconds from clicking restart system fully started and ready to go. Photoshop loads in 4 seconds. Yes, you read that right. 4 seconds. Awesome, just awesome.

For those that care, my system is a bit old, but maxed. I am running a Opteron 185 on an MSI K8N SLI Platinum board with 4GB of OCZ DDR400 overlocked to 1T timing and something like 2-3-2-5. This is my old dev box and so it is actually geared more towards software dev and beta testing than gaming, but does a good job none the less. It's also whipser quiet at full load with optional booster fans if I feel like hearing a jet take off. I mounted custom switches and LED dials on the front for a custom wired fan setup, so it's one of those heavily customized boxes that stick around for a while.

Havn't exactly timed SH3, but I figure from SH3 Commander icon to giving orders aboard my boat is less about 50 seconds or so. Still, that is a long time if I am testing a mod constantly.

[Graf Paper]

Still, [50 seconds ] that is a long time if I am testing a mod constantly.

So just use dummy campaign files or create your own, smaller campaign_RND.mis and campaign_SCR.mis files with just a handful of ships and planes for testing.

Load times will be dramtically faster when you have little or no traffic to generate and track.

There was a mod released that does just that. I want to say it was called "SH3 Campaign Mod for Testers".

[Scape]

Originally Posted by CCIP

Again, the real answer to maximizing loading efficiency lies in what the game's developers did for SHIV... It's a really complex operation, and there is no way around it in SHIII, unless you simply cut off parts of the campaign.

Random thought:

How about a GWX mod for SHIV then? Use the engine for SH IV and transpose the hard data or is that just too much work?