[WIP] The SH5 EcoMod

[gap]

Originally Posted by propbeanie

No, I didn't mean ref put that in there Jeff, but that maybe it looks like an unused portion of the file, and ref didn't touch it, but the game sees and uses it, though if it sees the ZHF in the folder, it already knows that that tile is altered... - maybe PS is putting it in there to note the edit?

I can be wrong, but I think that the most likely answer is a little oversight by Ref (see my considerations below), though I am a bit surprised that no one made Ref aware of it, when he released his tool...

Originally Posted by propbeanie

gap, I'd forgotten I had saved in psd format and did that... but once you convert RAW to ZHF, you cannot then convert the edited ZHF to RAW for further edits, and expect good results... sorry for my addled brain...

No need to say sorry, mate. You are absolutely right though, going by what Jeff has said, the fix could be easy

[Jeff-Groves]

What I suspect is those 4 bytes are identification bytes.
Does Ref's tool put them into the file when recompressed to ZHF?
I suspect so.
Does SH Game engine ignore them then?
Maybe but We need to prove that before we say 100% Yes.
If Ref's tool does in fact throw in the extra data to the ZHF files (Which tests show me it does) Should we remove that issue or just work around it by removing the extra 4 bytes on re-export from a modded file.
Since SH4 depends more on those type files? The SH4 modders should let us know about issues and we can go from there.

[Jeff-Groves]

I've downloaded all the source code for zlib so I can check that 4 bytes.

[gap]

Okay, just tested:



Do you see the vertical line to the very left of the map? I didn't actually check in game, but if ME "sees" it, there is an high chance that the game engine will see it too.

That should end all our doubts: that's a little mistake by Ref.

Yes, probably SHIV modders could tell us more on that, though I think that propbeanie's experience is relative to the latter game and he knows as much as us on the topic.

On a side note: why the GWX name for the Terrain Extractor-Importer, since it is a tool eminently done for SHIV?

[Jeff-Groves]

GWX 4 for SH4 was well under way even as SH5 was being worked on.
There was even a Type 7A done for SH4 before SH5 ever came out.
It's a long story about the GWX Team. Some of it is not happy stuff.
Ref wrote many tools and all were for the GWX Team. That's how he wanted it.
If he released them at the time? They were named as he saw fit.

[Jeff-Groves]

Now. If you re-export a ZHF saved with ref's tool. Do nothing to it then re-ZHF it THEN export again?
It is all kinds of messed up.

My conclusion is the original export to RAW is fine.
The compression to ZHF is flawed thus any further decompression and recompression only increases the flaws leading to a useless file.
I have not tested deleting those bad 4 bytes with multi compression-decompression.

[gap]

Originally Posted by Jeff-Groves

GWX 4 for SH4 was well under way even as SH5 was being worked on.
There was even a Type 7A done for SH4 before SH5 ever came out.
It's a long story about the GWX Team. Some of it is not happy stuff.
Ref wrote many tools and all were for the GWX Team. That's how he wanted it.
If he released them at the time? They were named as he saw fit.

I understand. I am afraid that starting from the next version of the mod, the acknowledgement list will be actually bigger than the list of features lol
Definitely Ref deserved a place in it

Originally Posted by Jeff-Groves

Now. If you re-export a ZHF saved with ref's tool. Do nothing to it then re-ZHF it THEN export again?
It is all kinds of messed up.

My conclusion is the original export to RAW is fine.
The compression to ZHF is flawed thus any further decompression and recompression only increases the flaws leading to a useless file.

No, I have used ZHF-compressed files; they are fine.
RAW files exported from TerrainData.BFD are fine too.
The error is introduced with the ZHF to RAW conversion.

[Jeff-Groves]

Originally Posted by gap

The error is introduced with the ZHF to RAW conversion.

Did you remove the first 4 bytes?

[gap]

Originally Posted by Jeff-Groves

Did you remove the first 4 bytes?

No, I didn't, let me check if it works. I will report back in a short while

[gap]

Okay, I think it actually works.

I used XVI32 for that, because I am more accustomed to its basic interface.
I removed the bytes marked in green in the picture below, and after that I could open the file in photoshop without further problems

 

That was easy, thanks

[Jeff-Groves]

Ref's decompression is putting those bytes in. It's a magic number that should have been excluded from the raw file output.
Makes me wonder a little about the SH3 Terrain files in GWX.

Now to test if Ref's tool messes with terrain in Game? We'd need to find a land area that it can show up on.
Or build a Deep Sea Explorer Sub.

[gap]

Originally Posted by Jeff-Groves

Ref's decompression is putting those bytes in. It's a magic number that should have been excluded from the raw file output.

Yes, that sounds plausible a flag or a marker of some sort that was needed by its code and he forgot to remove at the end of its cycle.

It is 4 am here. Time to rest.

Thank you for your inputs that are always illuminating, and let's continue this discussion tomorrow.

gap's off for now

[Jeff-Groves]

Sleep Good Mate!


Your questions and observations feed me and give me directions to check that I don't think of on my own. Every small bit of the Game we learn helps us go further into other areas never imagined just a year ago!

You started with some terrain modding ideas and have led us to a whole new way to do terrain modding!

Not just for SH5. But has impact on SH4 as We have discovered, and possibly for SH3!

That is not an everyday, all kinds of people do it everyday, thing that happens!

[propbeanie]

Art Anker encountered what he called "Lot's Super Wives" in an altered Carotio_SH4_TrukCoralHarbour mod, at what I called the "corners" of the Truk Atoll tile. I can do some digging tomorrow for the actual tiles in the SH4 Terrain / Data folder. I also encountered similar when I was attempting to alter Pearl Harbor to make room for a more accurate Submarine base in the Southeast corner of the East Loch, near the Navy Yard. I wrote one row into the 6 pixel overlap, and ended up with a wall from as deep as you could go, to as high as you could go, a solid reddish / brown wall... There are more than one way to get errors when editing the files... Attempting to re-edit an already edited ZHF file, or editing within the 6 pixel overlap in SH4.

You are correct gap that what you see in the ME does show in the game - at least in SH4 it does... I was trying to Search for Art's old post, but failed to find it, and I was trying to find the "Morse Code" post that someone reported having seen "dots" & "dashes" in FotRSU. It was an odd looking bunch of "dots" and "dashes" in the ocean of deep and shallow "water", probably caused by an edit that I believe I had done with GWX & PS, but I can't remember if I did a re-edit or got into the 6-pixel overlap... Most of the edits I did in FotRSU were kind of done like the Army Corps of Engineers coming in and dredging channels and parking spots for the ships in a lot of the harbors, that while historically deep enough for BB and CV, but in which the game just can't even handle a CL, due to the way SH4 draws a ship's hull. You need about an extra 4 to 6 meters underneath, else you get all that shtinking crashing and grinding noise of the ship hitting "bottom"...

[gap]

Originally Posted by Jeff-Groves

Your questions and observations feed me and give me directions to check that I don't think of on my own. Every small bit of the Game we learn helps us go further into other areas never imagined just a year ago!

That's true. I am glad that, besides keeping us awake until late, my silly questions have unexpectedly positive side effects too

Originally Posted by Jeff-Groves

You started with some terrain modding ideas and have led us to a whole new way to do terrain modding!

Not just for SH5. But has impact on SH4 as We have discovered, and possibly for SH3!

That is not an everyday, all kinds of people do it everyday, thing that happens!

One lesson I learned since I started messing with SH5 files, back in 2011, is that the distinction between SH5, SHIV and SHIII modding is somewhat artificial, and detrimental to the development of the SH modding community as a whole.

Personally I tend to see the three games as different specimens of the same thing, and when I have a modding idea I often think about its feasibility within the three of them. Indeed, some important changes have been brought in SHIV over SHIII, and in SH5 over SHIV and SHIII; each game has its own character, its pro's and its con's, and not all that is possible in one of them is doable in the others too, but all the SH games also have many striking similarities that we can exploit. That's obvious for someone who started from SH5, because this game was the last in the series and it owes many of its features to its predecessors, so it is natural for him to look back at them, but we should also admit that there are things about SHIII that you learn better when you dig into its younger brothers.

Unfortunately, at least in the past, there are has been an hostility between different modding communities and groups that has not eased the sharing of knowledge and ideas, but I am convinced that in our small way we should do our best to bypass this narrow-minded mentality and encourage the knowledge-sharing, and I think this is exactly the spirit of our conversation on this mod

Originally Posted by propbeanie

Art Anker encountered what he called "Lot's Super Wives" in an altered Carotio_SH4_TrukCoralHarbour mod, at what I called the "corners" of the Truk Atoll tile. I can do some digging tomorrow for the actual tiles in the SH4 Terrain / Data folder. I also encountered similar when I was attempting to alter Pearl Harbor to make room for a more accurate Submarine base in the Southeast corner of the East Loch, near the Navy Yard. I wrote one row into the 6 pixel overlap, and ended up with a wall from as deep as you could go, to as high as you could go, a solid reddish / brown wall... There are more than one way to get errors when editing the files... Attempting to re-edit an already edited ZHF file, or editing within the 6 pixel overlap in SH4.

Yes, though to a much lower scale, what you are talking about is also visible in stock SH5 where 'high-detail' terrain tiles interconnect with each other and with 'regular' tiles. In theory there are ways to make tile borders seamless (by 'overlapping' nearby tiles, or by smoothing their seams in TE), but probably devs didn't have time to do it, or the judged that no one would have noticed the slight height mismatches around the borders of all the main ports, canals and choke points, or wherever they needed an higher terrain detail level.

Besides not observing the 6-pixel overlapping, what could have caused the problems you have described above is probably the little glitch that we have discussed yesterday (concerning the ZHF to RAW conversion) and, maybe, the lossy nature of the ZHF compression.

In any case, this morning I checked my edit of the SH5 terrain around the Channel Islands, and everything looks okay on its borders. What I had done to keep the 1-pixel overlapping I had found in stock files, and to ensure a smooth transition between my edited terrain and the stock terrain that surrounded it, was pasting the stock height map for the same area as a new layer on top of of the height map I had previously edited, and applying to it the following layer mask:

 

Originally Posted by propbeanie

You are correct gap that what you see in the ME does show in the game - at least in SH4 it does... I was trying to Search for Art's old post, but failed to find it, and I was trying to find the "Morse Code" post that someone reported having seen "dots" & "dashes" in FotRSU. It was an odd looking bunch of "dots" and "dashes" in the ocean of deep and shallow "water", probably caused by an edit that I believe I had done with GWX & PS, but I can't remember if I did a re-edit or got into the 6-pixel overlap...

Okay, let me know if you can find the old posts describing those problems...

Originally Posted by propbeanie

Most of the edits I did in FotRSU were kind of done like the Army Corps of Engineers coming in and dredging channels and parking spots for the ships in a lot of the harbors, that while historically deep enough for BB and CV, but in which the game just can't even handle a CL, due to the way SH4 draws a ship's hull. You need about an extra 4 to 6 meters underneath, else you get all that shtinking crashing and grinding noise of the ship hitting "bottom"...

Yep, that's a known problem;in practice collisionable units collide with the sea bottom several meters before they actually hit it. Add to that the other issue of AI units getting stuck in low waters before they actually collide with anything, and the picture is complete.

I don't remember whether the cause for the first problem are collision spheres protruding from the keel or what else but, IIRC, back in the days TDW had created a patch that solved the issue for SH5; it should be included in the last version of the patcher.