[WIP] The SH5 EcoMod

[gap]

Originally Posted by Jeff-Groves

Send the files your working with to my e-mail and I'll work with them.

Okay, thanks. Check your email

[Jeff-Groves]

Had to go back to Cleveland for a few days but I have the files.
Now I just got word I have to go to California for a few days.

[gap]

Originally Posted by Jeff-Groves

Had to go back to Cleveland for a few days but I have the files.
Now I just got word I have to go to California for a few days.

California is not exactly around the corner
how is your jet lag

In the last few days I have been experimenting with Ref's Terrain Extractor-Importer, and I can confirm that it works for SH5 as well as for SHIV, though I think there is a little glitch with the inbuilt ZHF to RAW conversion utility.

Low-detail height maps extracted from TerrainData.BFD are greyscale images with a size of 601x601 pixels and covering an area of 1 deg x 1 deg. That means that they have a resolution of six seconds: four times lesser that the high-detail height masks generated by Terrain Editor.

Being able to edit the low-detail game terrain tiles is very important, especially for the sea bottom and for areas that we don't visit that often and/or we only see from far away; using 1-sec elevation data for the whole SH5 world would be a waste of time and resources.

The GEBCO website has bathymetric data that can be freely downloaded:

https://www.gebco.net/data_and_produ...thymetry_data/

Its maps have a resolution of 30 arc-seconds; this is five times lesser than we need, but while better resolution public domain maps would be welcomed (suggestions anyone?), the lack of resolution is not a big problem either, as underwater everything looks blurred and smoothed out anyway. Data offered by GEBCO is in many formats. The ones I have found more useful are GeoTiff (which can be opened and converted in a regular raster format using softwares as QGIS, thank you for pointing me into this program Seaowl!), and asc, which is a plain ASCII format with heigh values sorted in an array.

What I did, was creating an Excel spreadsheet which converts from metric height values contained in the asc files, to 8-bit grey shades calculated according to SH5's height table; the calculated grey values are then sorted by the same spreadsheet in the order required by the portable bitmap format (eternal gratitude to OldCoder for making me aware of it).

The next steps are importing the output maps in photoshop, re-scaling them to the desired size (some post-processing is needed for hiding the pixelization caused by magnification), replacing unacceptably blurred landmasses (where they apply), with the greyscale version of SRTM 30m data, (similarly edited to match the in-game height table) and, finally, replacing pixels along the borders of each map so that they match the pixels along the adjoining borders of the next maps (for avoiding abrupt height changes: as usual there is some overlapping between nearby tiles).

When I finish with those tasks I export to RAW/ZHF format, and the maps are ready to be used in game. The result I get is visually convincing: at least for the tiles I have edited so far, the edited seafloor has a lot more depth changes than the stock one, with trenches, sandbanks and shoals which obviously resemble the real bathymetric profile of the same areas. There is something I have found out though: in spite of the general similarity, and of all the care I have placed into matching they grey values of those maps with real heights, there will always be some important discrepancies, caused by the fact that there is not a simple relation between grey values and height/depth at any given point: due to the blending of an hardcoded noise mask, even if a set an area totally flat, there will always be a slope of some sort, and the height of that area might vary considerably relative to the height expected according to table I linked above.

If/when we need that level of accuracy, we will have to edit the 1.5 arsecond raw maps using Jeff's importer/exporter and/or terrain editor



The Channel Islands seen in Mission Editor. Left: as they look in stock game; right after the edits I described above

[Jeff-Groves]

Originally Posted by gap

In the last few days I have been experimenting with Ref's Terrain Extractor-Importer, and I can confirm that it works for SH5 as well as for SHIV, though I think there is a little glitch with the inbuilt ZHF to RAW conversion utility.

I don't think there is a glitch.
The compressed tiles are 601x601 and have been since SH3.
That's one of the main reasons the Kiel canal is like it is in SH3/4.
Limited detail terrain work can be done in SH3/4 because of the restriction.

SH5 we can go to greater details thanks to the Detailed RAW files that use actual height values over a smaller area rather then a crude 0-255 scale over a much larger area.

Given the size exported by Ref's program and based on info we have on the details RAW files? The Ref Exported files are 6 time larger then a true SH5 Detail RAW. (Not a fault of his program. That's how it is stored in the files)

Given We know the difference? I'm thinking We could cut the exported files down to the needed detail RAW size with one problem!
It will still be just a 0-255 limited output.

On California? I have a few friends very near there that I'm hoping to spend a weekend with to shoot some nice Air Guns.

[gap]

Originally Posted by Jeff-Groves

I don't think there is a glitch.

Maybe not exactly a glitch. I din't check if the uncompressed RAW files work as well as ZHF files and RAW files saved by photoshop, probably yes, but if you want to understand what I mean try doing the following: extract one tile from TerrainData.BFD and open the resulting RAW file in photoshop. The RAW importing menu will select for you the best import options and once opened everything will look in order.
Now try converting the same RAW file to the ZHF format, and then uncompress it again, or if you prefer uncompress one of the stock ZHF files. On opening the resulting RAW file, you are no longer suggested the optimal dimensions. If you try selecting anything bigger than 601x601 obviously you will be told that the image you are trying to open is bigger than the file and you are not allowed to go on with the import; if you select 601x601 you are told that the image is actually smaller than the file, and once opened, five or six columns of pixels in the map are shifted from left to right or vice versa, I don't remember exactly now.

Originally Posted by Jeff-Groves

The compressed tiles are 601x601 and have been since SH3.
That's one of the main reasons the Kiel canal is like it is in SH3/4.
Limited detail terrain work can be done in SH3/4 because of the restriction.

Well, SHIII terrain limitations are actually bigger than in SHIV/5: height maps are five times smaller in SHIII than they are in SHIV, i.e. 126 x 126 pixels, with an overlapping between neighbouring tiles of six pixels rather than one pixel. That means one height point every 30 seconds / kilometer as opposed to the resolution of one point every 6 seconds / 200 m of SHIV and the maximum resolution of one point every 1.5 seconss / 50 m attainable in SH5. Nonetheless, due to the extremely poor height resolution of SHIII, the devs used a trick: they introduced 1-bit masks similar in size to the SHIV height maps (602x602 pixels, with an overlapping of two pixels) that are used for determining the shape of islands, coast lines or canals like the Kiel canal

Originally Posted by Jeff-Groves

SH5 we can go to greater details thanks to the Detailed RAW files that use actual height values over a smaller area rather then a crude 0-255 scale over a much larger area.

Given the size exported by Ref's program and based on info we have on the details RAW files? The Ref Exported files are 6 time larger then a true SH5 Detail RAW. (Not a fault of his program. That's how it is stored in the files)

Given We know the difference? I'm thinking We could cut the exported files down to the needed detail RAW size with one problem!
It will still be just a 0-255 limited output.

If they will occupy a smaller space on file and the game will read them more readily, why not? Sure, they would still have an 8-bit output, but I suppose that that's a limit of the game code and, as we said, for very accurate terrain changes (where actually needed) we can always use the 1.5 sec RAW maps

Originally Posted by Jeff-Groves

On California? I have a few friends very near there that I'm hoping to spend a weekend with to shoot some nice Air Guns.

Enjoy

[Jeff-Groves]

If you need to work with a file saved to ZHF with Ref's program?
Extract it to RAW again and then delete the first 4 bytes with 010.

Not sure why that is added but I'll look closer at those 4 bytes.

I can do a script for 010 to remove them if you'd like.

[propbeanie]

Using the GWX Terrain Extractor, SH4 tiles are 601x601 when opened in PS for the first time, and the tiles do overlap by 6 pixels also. If you do NOT get the transitions between tiles edited exactly correct, you often end up with pillars of rock six pixels wide and hundreds of ?? pixels high at the corner junctions. You also have to do all edits in one session. You cannot go back in and re-edit any mistakes you make, else you might end up with 6 pixel wide walls or trenches at the borders, depending upon your edits. Therefore, it's best to print out the original tile on paper, and noodle with that and a pencil first, then use the software. There is an issue with either the PS edits, the Save back to RAW, or the conversion from RAW to the game's ZHF, and you end up with the blank dimensions in the Open dialog if trying to edit a 2nd time. If you just do one edit on the RAW file, then it will display in the game correctly after a single edit session with PS, else you get that shift of six pixels, and other areas near the 6 pixle "border" area that have changed their values. Anymore, if editing terrain for SH4, I export to RAW each time I do an edit, instead of trying to re-edit. Very frustrating...

Edit: oooohhh, as I was typing Jeff posts about those first four bytes... I will be doing some experiments with that in a few days! I could try that 010 script for you Jeff!... thank you! :salute

[gap]

Originally Posted by Jeff-Groves

If you need to work with a file saved to ZHF with Ref's program?
Extract it to RAW again and then delete the first 4 bytes with 010.

Not sure why that is added but I'll look closer at those 4 bytes.

I can do a script for 010 to remove them if you'd like.

Okay, now it makes more sense though, like you, I am curious to know the function of the extra bytes added by Ref.

Deleting them manually is something I should be able to do myself but, while not impelling, a script automating the process would be nice indeed

[Jeff-Groves]

Those first 4 bytes are in the ZHF files before the first extraction with Ref's tool. On the first extraction they are not there. If you save to ZHF then extract again? They are there.
I'd think a slight mistake on Ref's part of course. Now the question is does that mistake show in Game?
I suspect it's a magic header that is ignored or tells the engine to unzip the file.
Since we never use the unzipped version? We can just remove it for editing reasons.

I'll do some more research on the re-compression Ref's tool does and see if that is added to the files.
If it is? We many need to do a whole new tool to fix that issue.
ZHF is just a part of a compression library so should be easy enough to do.

What I will do is use a ZHF in stock SH4. Extract it then save again to ZHF and compare.

[propbeanie]

I wonder if the first four bytes are a "This file has been tampered with!" for the game?

[Jeff-Groves]

Originally Posted by propbeanie

I wonder if the first four bytes are a "This file has been tampered with!" for the game?

No. That is something Ref never did. Aside from the Jammu he did to replace Kashmir in headers of files? There was only one other branding he did at my request. That is in an area of dat files that have no impact on the Game that anyone has ever proven.
It is also coded so only the program that places the information can read it.
That was done in response to others claiming ownership of some type for files.
We quickly caught them using GWX files as the base. Thus ended that.
Sneaky bastards aren't we?

[gap]

@ propbeanie

Thank you for the suggestions mate. Yes, definitely there is an overlapping that needs to be observed between one tile and the next ones. Reason for that is that each pixel on those maps doesn't represent a quadrant in a grid, but a verted in a continuos mesh, and at least one row/column of vertices must be in common for two neighbouring terrain tiles for them to be treated as a continuous surface. If their heigh/grey values are not identical, an abrupt height change or a step is created, that from the game perspective are like a trench ir a wall.

That said, I think that SH5 tiles have an overlapping of just one pixel. Why I think that?

1 - by comparing stock tiles with real world terrain height maps, if a 6-pixel overlapping was to be applied, land masses on those tiles should look a bit smaller than they do, whereas they actually show the same land proportions as seen in the big (3601x3601 pixels) srtm maps

2 - as I said before, 601 pixels - 1 pixel of overlapping makes a resolution of exactly one height point every six arcseconds (= 200 m in SH5). A bigger overlapping would involve a decimal resolution which seems not likely to me.

3 - Before starting with my editing I had exporter two adjoining maps and overlapped them using the "divide" blending mode. The division of two identical pixels is equal to pure white (any number divided by itself makes 1 which, in image processing, is equal to white). The only way I could get all white pixels was by setting an overlap of only one pixel

I will do more test, but I think that my working method is correct for SH5. Yes, editing those maps requires a general plan. The error is around the corner, but as far as you keep track of all the changes made and you save every change in separate layers (that's easy in photoshop), keeping backup copies of the edited files, You don't necessarily need to edit everything at once. The four extra bits added by Ref's exporter are a bit annoying, but if you have saved copies of your working files in psd format, that's not a big problem either

[Jeff-Groves]

I'd say compression format is not the same when compressed with Ref's Tool. SH will read it however so that is not a bad thing.
Compression is based on zlib. Just check the main SH folder and you see that.

[gap]

Originally Posted by Jeff-Groves

Those first 4 bytes are in the ZHF files before the first extraction with Ref's tool. On the first extraction they are not there. If you save to ZHF then extract again? They are there.

That's exactly what happens to me too

Originally Posted by Jeff-Groves

I'd think a slight mistake on Ref's part of course. Now the question is does that mistake show in Game?

I suspect no. I think that the possible mistake becomes evident when you want open the RAW file in photoshop, but I am not 100% sure; I need to check that.

Originally Posted by Jeff-Groves

I suspect it's a magic header that is ignored or tells the engine to unzip the file.
Since we never use the unzipped version? We can just remove it for editing reasons.

Do you mean something added to the ZHF file whose remnants are found in the ouptput RAW file in the form of those extra bits? That seems unlikely to me, and I don't see the point of adding any header to those files, since the only reason for converting from ZHF to RAW is because RAW files can be edited in photoshop (but the supposed header would make the editing harder)

Originally Posted by Jeff-Groves

I'll do some more research on the re-compression Ref's tool does and see if that is added to the files.
If it is? We many need to do a whole new tool to fix that issue.
ZHF is just a part of a compression library so should be easy enough to do.

What I will do is use a ZHF in stock SH4. Extract it then save again to ZHF and compare.

Okay, good idea, though as I said I keep psd copies of my working files with all the editing layers, and in SH5 there are only two stock ZHF files that we might need to edit.

[propbeanie]

No, I didn't mean ref put that in there Jeff, but that maybe it looks like an unused portion of the file, and ref didn't touch it, but the game sees and uses it, though if it sees the ZHF in the folder, it already knows that that tile is altered... - maybe PS is putting it in there to note the edit?

gap, I'd forgotten I had saved in psd format and did that... but once you convert RAW to ZHF, you cannot then convert the edited ZHF to RAW for further edits, and expect good results... sorry for my addled brain...