The SH5 EcoMod

[gap]

https://i.imgur.com/ujY0pUh.png

The purpose of this mod is to bring a few improvements to the SH5 landscapes by replacing some global terrain files/parameters, and to add more variety to them by creating several new combinations of unique vegetation and terrain colors.

Only two new terrain types have been created so far but my aim is adding many other types, so to reflect the changes in world vegetation at the biome (for the least visited areas) or at the ecoregion level (for the busiest areas on the map). With this idea in mind, the information on terrestrial habitats and ecoregions (https://www.worldwildlife.org/biome-categories/terrestrial-ecoregions) compiled and made availble on the web by the WWF will be crossed with the climate classification (https://en.wikipedia.org/wiki/K%C3%B6ppen_climate_classification) and with selected climographs for each area, so to make sure that each terrain type will feature a vegetation realistic for its location, and that its look will evolve realistically over the year, following a realistic seasonality. If I will manage attainig this last goal, certain seasons will no longer occur in the wrong time of year, no snowy landscapes will appear where/when not expected in real world, no emerald-green grass will be visible during the driest seasons of seasonally arid and semi-arid areas, etc.

Change log


v0.2.1


Testing missions

- All four Brunsbüttel testing missions tweaked so that they are now compatible with OHII/TWoS terrain modifications.


v0.2


Main mod

- Fixed a bug where weird lens flares were visible close to the shores, when a dark texture was used on them.

- Stock default_mask_field.dds (a generic detail mask used to increase the terrain detail in flat areas) was replaced with a better and higher resolution seamless mask, featuring sharper roads and paths which connect with each other instead of leading nowhere, and grass/dirt patches representing agricultural fields.

- Fixed a bug where a segment of terrain located near the mouth of the Elbe river, just in front of Brunsbüttel, had an abrupt change in height caused by the mismatching of two adjoining terrain height masks.

- Improved stock detail masks for the aforementioned area.

- A new terrain type for the WWF Ecoregion PA0402, Atlantic mixed forests (https://editors.eol.org/eoearth/wiki/Atlantic_mixed_forests) was added in game, together with a new set of trees of the appropriate type and size for that region, different sand, grass and terrain detail texture combinations for each season, and a rock texture taken from a picture of the 'Rotes Kliff', on the German island of Sylt; this texture will represent in game the glacial sediments abundant along the German Bight.

Please note that in the current release this new terrain type will cover the whole extension of the PA0402 ecoregion, from W Denmark to SW France, but due to its big extension, to the big variety of landscapes that in real world can be found within it, and to its importance for the game (many player bases being located there), in the next releases of the mod it will be split in many smaller terrain types, each with it own characteristics.
Season fine-tunings submod

- Minor changes applied to ClimateZones.tga. For most of its extension, the new terrain type will lay on one climate zone with four seasons of equal length (spring: Mar to May; summer: Jun to Aug; autumn: Sep to Nov; winter, Dec to Feb) corresponding with Köppen's Cfa (mild oceanic (https://en.wikipedia.org/wiki/Oceanic_climate#Marine_west_coast_(Cfb))) climate.


Tree texture resolution submods

- Both submods updated with textures from the new terrain type


Detail mask resolution submods

- Similar to the tree texture resolution submods, these recently added submods are available in two versions:



Full resolution detail masks: 1024 x 1024 square pixels.


Low resolution detail masks: 256 x 256 square pixels.


As with the three textures, detail masks included in the main mod have an intermediate resolution of 512 x 512 pixels, comparable with their resolution in stock game and good for most situation, but not for appreciating the finest details I have drawn on the full resolution masks.


Testing missions

- Added four new testing missions (one for each season) located in Brunsbüttel, at the western end of the Kiel Canal, so to make easier the evaluation of the newly added terrain type.


v0.1


Main mod

- Stock default_mask_alpine.dds (a generic detail mask used to increase the terrain detail in mountainous areas) was replaced with a better and higher resolution seamless mask, featuring sharper roads and paths which connect with each other instead of leading nowhere, and more natural-looking grass patches.

- Terrain types defined in stock TerrainTypes.raw and Terrain.cfg, but actually unused in game, were removed from files and the remaining terrain types were reorganized so to make space for the newly planned types.

- A new terrain type for the WWF Ecoregion PA0403, Azores temperate mixed forests (https://editors.eol.org/eoearth/wiki/Azores_temperate_mixed_forests) was added in game together with a new set of trees of the appropriate type and size for that region (see further below for a list of the trees featured and their expected approximate heights in game), different sand, grass and terrain detail texture combinations for each season, and a magmatic rock texture for mimicking the cliffs of volcanic islands.


Increased tree density submods


This is going to need some explaination. SH5 tree distribution is dictated by the file TerrainObjectsDistributio.raw, a bitmap image in shades of gray, whose pixels can get values from 0 (black) to 255 (white). This invisible mask is overlayed and tiled over the SH5 terrain for deciding whether in any particular point of the map there should be a tree or not; it is my understanding that black pixels mean no chance, white pixels a 100% chance, and grey ones an intermediate probability of a tree being rendered in any given spot.It replaces stock TerrainObjectsDistributio.raw with a better mask featuring more natural-looking tree clusters and increased average tree density, compensating for the general decrese in tree sizes of this mod over stock trees. The submod is available in two alternative versions:



2x increased tree density: average grey value of the mask (this should be a rough estimate of how many trees will be rendered in game, see spoiler above for more on that): 14.63, standard deviation: 10.07 (this is a measure of how scattered are the trees). Stock values were 7.33 and 10.37 respectively.


3x increased tree density: average gray value of the mask: 21.98; standard deviation: 19.96.


NOTE: either of these optional mods might slow down your game. Remove them from your mod list if you experience any unexpected lag while sailing near land.


Season fine-tunings submod

ClimateZones.tga and ClimateZones.cfg (see post #2 in the Dynamic Environment thread (http://www.subsim.com/radioroom/showpost.php?p=2016081&postcount=2) for a detailed explaination on the meaning of those files) edited so that the newly created terrain type will lay on three different climate zones:

- Csa (hot Mediterranean climate (https://en.wikipedia.org/wiki/Mediterranean_climate#Hot-summer_mediterranean_climate)): long summer with partly dry grass lasting from mid May to mid September);

- Csb (warm Mediterranean climate (https://en.wikipedia.org/wiki/Mediterranean_climate#Warm-summer_Mediterranean_climate)): "standard-duration" summer lasting from June through to August;

- Cfa/Cfb (humid subtropical (https://en.wikipedia.org/wiki/Humid_subtropical_climate)/mild oceanic climates (https://en.wikipedia.org/wiki/Oceanic_climate#Marine_west_coast_(Cfb))): no summer, long autumn (July through to October), winter (November through to February) and spring (March through to June).

NOTE: this submod is based on Dynamic Environment files, and it should be enabled only by DynEnv or TWoS users. Using it with vanilla game or with any other enviromental mod than the ones specified above WILL lead to the harshest of the CTD's!


Tree texture resolution submods

The resolution I have chosen for individual tree textures of the main mod is 512 x 512. This is probably the best compromise between eye-candy and performance. Should you experience significant lags while sailing near land, or should you want to enjoy the trees in their original resolution, you can enable one of the following optional mods on top of the main mod:



Full resolution tree textures: 1024 x 1024 square pixels individual tree textures;


Low resolution tree textures: 256 x 256 square pixels individual tree textures (as in stock game).



Testing missions

Azores are not around the corner. Should you decide to help me with some alpha testing, I suggest you to enable this optional mod, and run one of the four missions included (one for each season; only mission date changes, all other parameters are identical from a mission to the other).
Included in each mission, but temporarily disabled, is a 30m-high ruler with 1m, 5m and 10m marks which I used for measuring/adjusting tree heights. Should you want to check their heights yourself, before you enable the mod you can open either of the misge files included in it using notepad, and replace the disabled rows (i.e. the ones starting with a semicolon) with the following rows:

[Unit 2]
Name=_Ruler#1
Class=XXruler
Type=408
Origin=Environmental
LayerOperation=0
Side=0
Commander=0
CargoExt=-1
CargoInt=-1
Doctrine=0
CfgDate=19440401
MainEquipment=1
SecondaryEquipment=1
DeleteOnLastWaypoint=true
DockedShip=false
IsEscort=false
GameEntryDate=19380101
GameEntryTime=0
GameExitDate=19451231
GameExitTime=0
EvolveFromEntryDate=false
Long=-3051760.000000
Lat=4539310.000000
Height=11.765031
IsFromSingleMis=false
Heading=0.000000
Speed=0.000000
CrewRating=3
DelayMin=0
ReportPosMin=-1
ReportPosProbability=100
HighPrioContact=false
RandStartRadius=0.000000
TacticalUnit=false
AvailStartDate=19380101
AvailEndDate=19451231
TempHeading=0.000000
TempNextWP=0
CombatSubmarine2D=false
CombatRadius2D=30.000000
CommandedSpeed2D=0.000000
Num2DCombatKills=0
Intercept2DCooldown=6.000000
NextWP=0


Acknowledgments


- Mihai Stoian, a.k.a. stoianm (http://www.subsim.com/radioroom/member.php?u=271282), for pointing me to terrain files for the first time, during the assembling of 'Dynamic Environment'.

- Mihai Gosa, a.k.a. Pintea (http://www.subsim.com/radioroom/member.php?u=253261), who patiently taught me most of what I know about those files, who helped me with the tweaked shaders included in this mod, and from whom I have still a lot to learn.

- Ubisoft Romania, textures.com (https://www.textures.com/) and outworldz.com (https://www.outworldz.com/cgi/free-seamless-textures.plx), for some of the seamless textures used in this mod, though in most cases I reworked/recombined them.

- The many photographers and 3D artists from whose pictures and renders I extracted the tree textures and some of the seamless terrain/rock textures featured in this mod.

- Charles Hollemeersch, creator of Njob (http://charles.hollemeersch.net/njob/): a simple utility that I use for quickly creating height, normal and ambient occlusion maps from diffuse textures. I must say that so far it has done a superb job of all the textures I tried it on, with only minor manual adjustments required.

- Last but not least: subsim.com for hosting this mod and the relative discussion, and all the subsim friends who helped me with advise and beta testing. Your support is priceles guys! :subsim:

Install instructions and compatibility issues


The mod and all the optional mods are JSGME-ready, they can be enabled in no special order and they should be compatible with vanilla game and with any other mod except for the 'Season fine-tunings' submod which should only be used together with TWoS or DynEnv (and enabled on top of them), and either of the 'Tree texture resolution' and 'Detail mask resolution' submods, which should be enabled on top of the main mod.

No campaign restart required (http://www.subsim.com/radioroom/showpost.php?p=2583075&postcount=53).

---

Well, I think that's all for now. If you have any remarks, bug reports or simple feature suggestions, or if you want to put your modding skills at work with this mod, I am very open to any kind of contribution and this is the right time showing your feed-back. But first have a look yourself; here is the download link :salute:


https://i.imgur.com/0PY2mJw.png (http://www.subsim.com/radioroom/downloads.php?do=file&id=5408)

[gap]

https://i.imgur.com/EB01jBR.png

Overview of the terrain types featured in this mod

https://i.imgur.com/yNXmnYc.png


Legend:

Stock terrain types


Lime green: Rain forests
Dark green: Temperate forests and taigas
Ochre: Savannas
Light yellow: Temperate grasslands
Bluff: Montane grasslands and shrublands
Pale green: Tundra
Pink: deserts and xeric shrublands
Grey: polar deserts
White: polar ice caps


New (numbered) terrain types


PA0402, ATLANTIC MIXED FORESTS (https://www.worldwildlife.org/ecoregions/pa0402):


Western Denmark, Northwestern Germany, Netherlands, Belgium, Northern and Western France


PA0403, AZORES TEMPERATE MIXED FORESTS (https://www.worldwildlife.org/ecoregions/pa0403):


Azores archipelago




List of trees/shrubs species, rock types and sand types featured in this mod, sorted by terrain type



PA0402, ATLANTIC MIXED FORESTS (https://www.worldwildlife.org/ecoregions/pa0402)

Trees:

European beech (Fagus sylvatica) - 22.6 m
European beech (Fagus sylvatica) - 25 m
Common oak (Quercus robur) - 18 m
Silver birch (Betula pendula) - 16 m
Scots pine(Pinus sylvestris) - 17.2 m
Scots pine(Pinus sylvestris) - 21.2 m
Norway spruce (Picea abies) - 17.5 m
Norway spruce (Picea abies) - 18.5 m


Rocks:

glacial till


Sand:

silica sand


Terrain features:

Snowcover during the coldest months (December, January and February).



PA0403, AZORES TEMPERATE MIXED FORESTS (https://www.worldwildlife.org/ecoregions/pa0403)

Trees:

Azores heather (Erica azorica) - 5.5 m
Azores heather (Erica azorica) - 6.5 m
Faya tree (Myrica faya) - 8 m
Faya tree (Myrica faya) - 9 m
Azores white wood (Picconia azorica) - 7 m
Azores white wood (Picconia azorica) - 8 m
Azores laurel (Laurus azorica) - 10 m
Azores juniper (Juniperus brevifolia) - 6 m


Rocks:

magmatic rock


Sand:

black sand


Terrain features:

Moderately dry grass on some islands during the summer.





Notice on SH5 trees

The trees we see in game are in reality 2D square "billboards" with a semi-transparent texture mapped on top of them; they rotate with the camera, so that they are always facing the observer, and from a distance we get the illusion that they are three dimensional objects, while they are not.

Only eight different trees per terrain type are allowed, each corrisponding to a square of fixed dimensions on a texture with 8:1 proportions. The mapping of each tree object on top of that texture is hardcoded, thus preventing us from increasing the number of trees for each terrain type and/or from importing simplified 3D tree models in place of two dimensional ones.

Tree sizes are randomized in game. The heights stated in the spoiler above, are relative to the highest specimens of each tree type. No matter how small is the scale of the 2D objects stored in vegetatio.gr2 file, if the whole texture-space assigned to each tree type is used, the highest specimens of that tree will have an in-game height of about 30 m. That explains why stock trees look so disproportionately big. For the above reason, the only way to make trees smaller (without editing the game code), is to make their texture smaller than the texture-space they are drawn on (a trick applied for the first time by our fellow subsim member echolot). This is a waste of memory indeed, but I could't think of any other workaround.

Conversely, we can make trees taller than 30 m by increasing the size of the 2D billboard they are mapped on, i.e. there not seem to be any fixed maximum size, and while there is a limit beyond which huge trees will look distorted, we will seldomly need to trespass that limit.

[gap]

The Kiel Canal and the Elbe river seen from Brunsbüttel in springtime:
https://i.imgur.com/bLC9D6o.png
Smooth glacial hills covered in a lush vegetation borther the mouth of the Elbe river:
https://i.imgur.com/5UeNvPl.png
Fall foliage in the mixed forests of the German Bight:
https://i.imgur.com/O3gcD8w.png
Snowy landscape in NW Germany:
https://i.imgur.com/yXIYiDe.png
Agricultural fields on the plains of Northern Europe:
https://i.imgur.com/CIQ4D1w.png
Windy paths cross each others on the mountainous land of the Azores:
https://i.imgur.com/YAYzTgT.png
Springtime on São Miguel, Azores:
https://i.imgur.com/gMZLUtn.png

[gap]

RESERVED 3

https://i.imgur.com/oROcJMC.png

[Fifi]

WOW!!
You are back with your amazing mod :Kaleun_Applaud:

TWOS compatible? Ahh campaign restart required...

:Kaleun_Cheers:

[gap]

WOW!!
You are back with your amazing mod :Kaleun_Applaud:

Thank you mate. Already tested it in your game? :)


TWOS compatible?

I think so, unless Vecko has switched to any environmental mod other than Dynamic Environment or made substantial changes to any of its core files (ClimateZones.tga and ClimateZones.cfg).


Ahh campaign restart required...

I am not really sure about that. There is a chance that the game reads information on climate zone and terrain tipe "on the spot", only for the areas surrounding player's sub, in which case campaign restart might not be required. After so many tests carried out for tracking down possible causes of CTD's, Vecko might know more than me on this topic. In the doubt, I would stay on the safe side. :hmm2:

[Fifi]

I will try in next bunker save... can’t restart right now, just had my best result ever :haha: and have already restarted from last Vecko’s update... :timeout:

If i experience any trouble or unexpected CTD, i will deactivate the mod, and wait to be killed!

[gap]

I will try in next bunker save... can’t restart right now, just had my best result ever :haha:

Heheh, I noticed it from the SH5 Screenshot & Video thread :arrgh!:


and have already restarted from last Vecko’s update... :timeout:

If i experience any trouble or unexpected CTD, i will deactivate the mod, and wait to be killed!

...or you can enable the mod, run one or more of the testing missions included with it, exit the game, report back here in case you notice anything wrong with it or have any other remark, and then disable again the mod before you continue your brilliant career :D

Anyway, at its current state this mod is of little use in career, as it only affects Azores terrain and vegetation...

[Fifi]

:Kaleun_Thumbs_Up:

[Fifi]

Jesus that looks nice! Scenery is photo realistic! :o
I never sailed in this area, so i don’t know how this island looks like with stock TWOS...

Anyway, i’ve Installed your Eco mod on top of all in JSGME, in this order: 1 Main mod + 2b + 3 + 4a + testing missions
I believe that’s the maximum in computer impact matter, and have to say i still have a steady 60 FPS even in external camera.
On side note, there are no other boats around...maybe you could add some action around there to see.

I guess it only apply on this zone area, and i’m Impatient to see it on the whole theatre! :up:

[THEBERBSTER]

Hi gap
Very nice work and so unexpected.

Makes you wonder about the possibilities for improving the vegetation around Memel and the Kiel Canal.
Peter

[vdr1981]

Hi gap
Very nice work and so unexpected.

Makes you wonder about the possibilities for improving the vegetation around Memel and the Kiel Canal.
Peter


Indeed :yep:...Very nice, thanks Gap! :salute:

[gap]

Jesus that looks nice! Scenery is photo realistic! :o

Hi gap
Very nice work and so unexpected.


Indeed :yep:...Very nice, thanks Gap! :salute:

Thank you guys, I am glad that you appreciate my efforts :smug: :shucks:


I never sailed in this area, so i don’t know how this island looks like with stock TWOS...

You can enable only the testing missions without the actual mod if you want, and check yourself what those islands looked like before my tweaks. They were covered with a luxuriant jungla-like vegetation composed of gigantic palm trees and banana plants lol :D

I have never been in the Azores, but from what I have read and from the pictures I have seen, the shrubs and small trees that I have chosen for this mod are typical of the Azores flora at medium and low altitudes. I have put a great care in scaling those plants correctly; it is unbeliavable how much difference makes, in terms of lanscape realism, getting vegetation in the right proportions. Yet, I would be curious to hear the impressions of someone coming from there. Is any Azorean subsim member reading these notes? :O:


Anyway, i’ve Installed your Eco mod on top of all in JSGME, in this order: 1 Main mod + 2b + 3 + 4a + testing missions
I believe that’s the maximum in computer impact matter, and have to say i still have a steady 60 FPS even in external camera.
On side note, there are no other boats around...maybe you could add some action around there to see.

Okay, that would be the maximum in terms of performance impact, but take in mind that without many 3D objects in rendering radius and without campaign data loaded in memory, we cannot consider the current testing missions a reliable benchmark to evaluate the performance hit by this mod.

In future we could add some cities/ports/airports on the Azores and marine traffic around them, for making them more lively. For the record, there was an important airfield in the island of Terceira which, starting from '43, was used by the Allies for closing the mid-Atlantic gap, and by '44 a second military airfiled was built by the Americans in the Santa Maria Island. Probably those additions would be better addressed in OHII/TWoS than with an environmental mod as this one but, for sure, your idea of making the testing missions more appealing by adding some units and a bit of action to them, is a good one; I will put it in practice with the next releases :up:


I guess it only apply on this zone area, and i’m Impatient to see it on the whole theatre! :up:

The next terrain type that I will work on, the Atlantic mixed forests, will encompass a big portion of map, from E Denmark to SW France (though I will probably split it in smaller parts for adding more variety to the game) and it will cover many of the U-boat bases featured in campaign. After adding that ecoregion, I think the EcoMod will be more interesting for a wider audience, and by setting the next test missions near one of the big ports located there, we will get a better idea of its impact on computer performance.


Makes you wonder about the possibilities for improving the vegetation around Memel and the Kiel Canal.
Peter


Sure: the Brunsbüttel end of the Kiel Canal will be covered with one of the next releases (see my reply to Fifi above). Kiel and Memel, on turn, are located in two other ecoregions (respectively: Baltic mixed forests and Central European mixed forests), but they are high in my todo list too. Let me know if you have any suggestions or special requests about them :salute:

[vdr1981]

Will this mod finally get rid of snow in Mediterranean?:hmm2:

[gap]

Will this mod finally get rid of snow in Mediterranean?:hmm2:

Definitely yes! Snow there has always pissed me off lol ...and grass will turn brown in summer, and if you plan reworking/extending the Mediterranean campaign, adding the Pola U-boat base to the pool of Mediterranean bases, I promise I will extend the effects of this mod also to Croatia and Yugoslavia :03:

[vdr1981]

Are these strange looking color anomalies intentional ?

https://i.postimg.cc/zG1Ngnk4/SH5-Img-2018-12-02-22-07-57.jpg

[Jeff-Groves]

That's a dds file.
I saw that today looking at files in the terrain folders.
detaile.dds maybe?

Where were you in Game?

[vdr1981]

That's a dds file.
I saw that today looking at files in the terrain folders.

Misplaced random flower fields maybe? :hmm2:

EDIT:


Where were you in Game?
Test mission, Azores...Summer IIRC.

[Jeff-Groves]

No. A miss called image file is my first thought.
:hmmm:
Then again it could be a file gap and I are interested in.
:hmmm:

[gap]

Are these strange looking color anomalies intentional ?

No, this is only happening where incident sunlight illuminates portions of shore under certain angles, after changing the sand texture. It looks like the result of messy normal map, but shores have no normal maps applied, only rocky cliffs have one.

This glitch has always bothered me, since the time I worked on some of the DynEnv's textures. There are two methods that I can think to iron it out, or at least reduce its extent: reworking the sand textures (for some obscure reason, some colors in them seem to cause the problem), which I will certainly do, or tweaking the light glare FX parameters :hmmm:

[vdr1981]

No, this is only happening where incident sunlight illuminates portions of shore under certain angles, after changing the sand texture. It looks like the result of messy normal map, but shores have no normal maps applied, only rocky cliffs have one.

This glitch has always bothered me, since the time I worked on some of the DynEnv's textures. There are two methods that I can think to iron it out, or at least reduce its extent: reworking the sand textures (for some obscure reason, some colors in them seem to cause the problem), which I will certainly do, or tweaking the light glare FX parameters :hmmm:

Are you saying that the issue was present even with the original DynEnv 2.9 mod, because I can't remember that ever saw it?:hmmm:

[gap]

That's a dds file.
I saw that today looking at files in the terrain folders.
detaile.dds maybe?

Where were you in Game?

Not detaile.dds. That's a simpe decal used for increasing the detail level and for masking out the tiling of the main texture, but it might be detaile_nm.dds though no terrain ini/cfg file calls it (applied automatically maybe?). The name of the file tells me that it is used as a normal map of some sort, though it doesn't look like one :hmmm:

[gap]

Are you saying that the issue was present even with the original DynEnv 2.9 mod, because I can't remember that ever saw it?:hmmm:

No, while testing different sand textures for one of the optional (and least popular) DynEnv mods, I and stoianm realized that some of them caused some crazy rainbow light reflections, similar to the ones seen in your screenshot, but I think to remember that in that case the glitchy textures were much lighter than the one used in this mod.

[Jeff-Groves]

Try a normals dds in grey.
Like the original one in detaile_nm.dds

That file is more a height map then a normal map.

[vdr1981]

No, while testing different sand textures for one of the optional (and least popular) DynEnv mods, I and stoianm realized that some of them caused some crazy rainbow light reflections, similar to the ones seen in your screenshot, but I think to remember that in that case the glitchy textures were much lighter than the one used in this mod.

I think I know what you mean. Similar glitches can sometimes bee seen while looking at burning ships or similar ?:hmmm:

[gap]

Try a normals dds in grey.
Like the original one in detaile_nm.dds

I have never altered that file. What I am going to do is replacing it with a plain grey texture and, if that doesn't work, with a proper normal map based on the original texture.


That file is more a height map then a normal map.

True, even though the _nm suffix seems to suggest that it is a normal map. I wonder if that is the culprit and why the glitch is only apparent on the shores, and only in conjunction with certain diffuse textures :hmmm:

I think I know what you mean. Similar glitches can sometimes bee seen while looking at burning ships or similar ?:hmmm:

Exactly the same! looks like overdone light reflection... :yep:

[gap]

Okay, I have replaced the RGB and the alpha channels of detaile_nm.dds with plain grey color, and the problem persists. At least we can exclude tha file from the list of suspects.

Something that I have noticed is that the visual glich only appears along step shores, where the sand texture gets blended with the texture used for slopes, the one combining diffuse and normal maps. That might confirm my impression that the blue spots are actually abnormal light reflections. Now that I think about it, there is something else that I have noticed in the past days: the darker the sand texture, the more obvious is the bug; at some point, while making experiments with an especially dark texture, the glitch was so severe that it looked like lens flares...

My guess is that, when rocky slopes and shores mix together, not only cliff's diffuse texture gets multiplied by the color values of the diffuse shore texture (what is desiderable), but also the normal texture gets similarly multiplied, thus causing abnormal lighting effects. That coul explain why the bug is not noticeable with light sand tectures, as in image computing a colour multiplied by white is equal to the same color (white pixels are equal to 1's). If that was true, the only solution would be giving up dark sand textures or finding the mistake in the game shaders :hmmm:

[gap]

Try a normals dds in grey.
Like the original one in detaile_nm.dds

That file is more a height map then a normal map.

Okay, after several tests I am reasonably sure that the one terrain texture having an effect on the "glaring shore glitch", is Tex02 (in terrain .ini files), the one commonly set up to a sandy beach texture. The darker this texture, the more obvious the problem.

Despite the fact that the glare is only visible in step segments of the coast, where the sand texture gets blended with the rock texture, neither Tex01 nor Tex05 (respectively: the texture applied to cliffs and its normal map) have any significant effect on the problem.

Now looking into terrain shaders:


TerrainSurfacePS.fx

#include <data/Shaders/Shadowmap/ShadowMapUtils.inc>

float3 SunLightColor : register(c0);
float4 AmbLightColor : register(c1);
float3 FogColor : register(c2);
float3 CamPosition : register(c3);
float4 Params : register(c4);
float MaxDepth : register(c5);
float3 UnderWaterColor : register(c6);
float g_drawShadows : register(c7);
float4 sixColorAmbient[6] : register(c8);

#ifdef EDITOR_MODE
float2 g_edTexOffset : register(c15);
float2 g_edTexRot : register(c16);
#endif

#ifdef OUTPUTDEPTH
float2 biasParams:register(c30);
#endif

struct PS_IN
{
#ifdef OUTPUTDEPTH
float depth :TEXCOORD0;
#else
float4 Tex0 : TEXCOORD0;
float3 WSPos : TEXCOORD1;// world space pos
#ifdef CLOUDS
float2 Tex2 : TEXCOORD2;
#endif
float3 SunDir : TEXCOORD3;
#ifdef SPECULAR
float3 EyeVertex : TEXCOORD4;
#endif
float3 VPos : TEXCOORD5;
float3 VNormal : TEXCOORD6;
float3 ProjPos : TEXCOORD7;
float4 Color0 : COLOR0;
float4 Color1 : COLOR1;
#endif
};

sampler HardSlope : register(s0);
sampler Sand : register(s1);
sampler Grass : register(s2);
sampler Detail : register(s3);
sampler Normal : register(s4);
sampler Clouds : register(s5);
sampler MaskMaterials : register(s6);
sampler Mask : register(s12);//register(s7);// s7 is for shadowmap

#ifdef EDITOR_MODE
sampler EditorMask : register(s8);
#endif

#define MASK_TEXTURE_TILE_SIZE float2(512, 512)// size of a tile inside the masks texture (which is 2048x512)
#define MASK_TEXTURE_TILE_MIPS 8 // maximum mip levels of a sub-texture

#define MASK_NUM_TILES 4 // number of texture masks inside the texture pack
#define MASK_INV_NUM_TILES (1.0 / MASK_NUM_TILES)

float CalculateMipmapLevel(float2 uv, float2 textureSize)
{
float2 dx = ddx(uv * textureSize.x);
float2 dy = ddy(uv * textureSize.y);
float d = max(dot(dx, dx), dot(dy, dy));
return 0.5 * log2(d);
}

float3 GetMaskedTextures(in float3 Color, in sampler MaskTexture, const in float2 Tex0, const in float2 Tex1)
{
//float maskRock = saturate(In.Color0.r + In.Color0.g);
float4 MaskC = tex2D(MaskTexture, Tex1);// * maskRock;

float2 TexMask = Tex0 * 2;
float lod = CalculateMipmapLevel(TexMask, MASK_TEXTURE_TILE_SIZE);
lod = clamp(lod, 0.0, MASK_TEXTURE_TILE_MIPS);
float texSize = pow(2.0, MASK_TEXTURE_TILE_MIPS - lod);
float2 size = float2(texSize * MASK_NUM_TILES, texSize);
float2 TexMat = frac(TexMask);
TexMat.x = TexMat.x * ((size.x * MASK_INV_NUM_TILES - 1.0) / size.x) + 0.5 / size.x;
TexMat.y = TexMat.y * ((size.y - 1.0) / size.y) + 0.5 / size.y;

// mat 0
float4 TexMatOff = float4(TexMat, 0, lod);
Color = lerp(Color, tex2Dlod(MaskMaterials, TexMatOff), MaskC.r);

// mat 2
TexMatOff = float4(TexMat.x + MASK_INV_NUM_TILES * 2, TexMat.y, 0, lod);
Color = lerp(Color, tex2Dlod(MaskMaterials, TexMatOff), MaskC.b);

// mat 3
TexMatOff = float4(TexMat.x + MASK_INV_NUM_TILES * 3, TexMat.y, 0, lod);
Color = lerp(Color, tex2Dlod(MaskMaterials, TexMatOff), MaskC.a);

// mat 1 is shadow, we don't need to sample again, just use mask
Color *= (1 - MaskC.g);

/*
// the material texture has 4 textures inside it
// we need to convert the texture coords to those 4 (meaning tileable textures from 0 to 0.5 and from 0.5 to 1 on x/y)
float2 matTc = float2(Tex0.x, Tex0.y * 4);// tile
float quadTc = matTc.x * 4;
float floorQTc = 0.25 * floor(quadTc);
float ceilQTc = 0.25 * ceil(quadTc);
float2 tc;
tc.y = matTc.y;

// mat 0
tc.x = matTc.x - floorQTc;
Color = lerp(Color, tex2D(MaskMaterials, tc), MaskC.r);

// mat 2
tc.x = matTc.x - ceilQTc - 0.25;
Color = lerp(Color, tex2D(MaskMaterials, tc), MaskC.b);

// mat 3
tc.x = matTc.x - ceilQTc;
Color = lerp(Color, tex2D(MaskMaterials, tc), MaskC.a);

// mat 1 : shadow, must always be last
//tc.x = matTc.x - floorQTc + 0.25;
Color *= (1 - MaskC.g);
*/


// mask details as well, but let some detail come out so "some" normal mapping is applied
//maskRockGrass *= saturate(1.4 - min(MaskC.r + MaskC.g + MaskC.b + MaskC.a, 1));

return Color;
}

float4 main( PS_IN In ) : COLOR
{
#ifdef OUTPUTDEPTH
return In.depth;

/*float d = In.depth;
float z_slope = abs(ddx(d)) + abs(ddy(d));
float K0=1E-5;
float K1=1E-1;
float zBias0=10*K0;
float zBias1=10*K1;
zBias0=biasParams.x;
zBias1=biasParams.y;
float od = d+zBias0+z_slope*zBias1;
return od;
*/
#else
#ifdef REFLECTION
clip(In.VPos.y);
#else
clip(In.VPos.y + MaxDepth);
#endif

float3 Color;
float3 HardSlopeC = tex2D(HardSlope, In.Tex0.xy);
float3 SandC = tex2D(Sand, In.Tex0.xy);
float3 GrassC = tex2D(Grass, In.Tex0.xy);

#ifdef CLOUDS
float3 CloudsC = tex2D(Clouds, In.Tex2);
#endif

// smoothen the texture transition
// this can be done in the vertex shader with acceptable quality loss, if there is vertex bandwidth to spare
float fDelta1 = saturate(lerp(In.VPos.y - 0.5, In.Color0.g, 0.1));
float fDelta2 = saturate(lerp(In.Color0.r, In.Color0.g, 0.53));
In.Color0.r = 1 - fDelta1;
In.Color0.g = min(1, fDelta2 * 2) - In.Color0.r;

// blend textures
Color = lerp(HardSlopeC, SandC, In.Color0.r);
Color = lerp(Color, GrassC, In.Color0.g);

float3 DetailC;
#ifdef MICRO_DETAIL // - needs an overlay ready detail texture, doesn't work with the current one
// add micro detail for all terrain
//DetailC = tex2D(Detail, In.Tex0.xy * 8);
//Color = 1.7 * DetailC * Color; // calculate overlay factor
#endif

//float maskRockGrass = (1.0 - In.Color0.r);

// add macro detail for the entire terrain
//DetailC = 1.1 - (tex2D(Detail, In.Tex0.xy * 0.05) - 0.5); // for b & w
//Color *= DetailC;
DetailC = tex2D(Detail, In.Tex0.xy * 0.08);
Color *= DetailC;

#ifdef EXTRA_TEXTURE_MAPS
Color = GetMaskedTextures(Color, Mask, In.Tex0.xy, In.Tex0.zw);
#endif

#ifdef EDITOR_MODE

// rotate and translate
float2 texEd;
float2 te = In.Tex0.zw - float2(0.5, 0.5);
texEd.x = te.x * g_edTexRot.x + te.y * g_edTexRot.y;
texEd.y = te.y * g_edTexRot.x - te.x * g_edTexRot.y;
texEd += float2(0.5, 0.5);

float2 trans;
trans.x = g_edTexOffset.x * g_edTexRot.x + g_edTexOffset.y * g_edTexRot.y;
trans.y = g_edTexOffset.y * g_edTexRot.x - g_edTexOffset.x * g_edTexRot.y;
texEd += trans;
Color = GetMaskedTextures(Color, EditorMask, In.Tex0.xy, texEd);
//return float4(Color, 1);
#endif

// use clamped sunlight for clouds and specular
float3 clampedSunColor = min(1, SunLightColor);

#ifdef CLOUDS
Color *= 1 - CloudsC * AmbLightColor.w * float4(clampedSunColor, 1);
#endif

// normal map for micro detail
float3 NormalC = tex2D(Normal, In.Tex0.xy).xyz * 2 - 1;
float maskRock = 1 - saturate(In.Color0.r + In.Color0.g);
float3 normal = float3(NormalC.rg * maskRock, NormalC.b);// only apply nm to rock
//float maskGrass = In.Color0.g;
//normal.rg *= saturate(1 - maskGrass + 0.5);

float dist = distance(CamPosition, In.VPos);

// fade tree shadows in the distance
float treeDist = saturate((dist - Params.z) * Params.w);

// add normal map for macro detail
NormalC = tex2D(Normal, In.Tex0.xy / 5.0).xyz * 2 - 1;

normal.xy += (NormalC.xy * maskRock) * 1.5 ; // multiply by how much we want the macro normal map to stand out
//normal.z = sqrt( 1 - normal.x * normal.x - normal.y * normal.y );
normal = normalize(normal);

// add diffuse
float3 lightDir = normalize(In.SunDir);
float3 lightDiff = SunLightColor * saturate((dot(normal, lightDir) + 0.5) / 1.5);

// add shadows, not for reflection
#ifndef REFLECTION
if ( g_drawShadows > 0.0 )
lightDiff *= GetJitteredCascadeShadowMapValue_16(In.ProjPos, float4(In.WSPos, 1)).r;
#endif

#ifdef SIX_COLOR_AMBIENT
// 6-color ambient, use vertex normal "slightly" combined with the per pixel normal, since it gives a more uniform look
float3 vnormal = In.VNormal;
vnormal.xz += normal.xz;
vnormal = normalize(vnormal);

float3 squaredNormal = vnormal * vnormal;
int3 isNegative;
isNegative.x = step(0, -vnormal.x);// must use a function that returns integer values
isNegative.y = step(0, -vnormal.y);
isNegative.z = step(0, -vnormal.z);

float3 ambient =
squaredNormal.x * sixColorAmbient[isNegative.x] +
squaredNormal.y * sixColorAmbient[isNegative.y+2] +
squaredNormal.z * sixColorAmbient[isNegative.z+4];

ambient *= saturate((dot(normalize(In.VNormal.rbg), -lightDir) + 1.5) / 2.5);
Color *= lightDiff + ambient;
#else
Color *= lightDiff + AmbLightColor;
#endif

Color *= lerp(In.Color1.xyz, 1, treeDist);// fade out tree shadows where there are no trees

#ifdef SPECULAR
float specPerturb = max(Color.r, max(Color.g, Color.b));
float3 eyeDir = normalize(In.EyeVertex);
float3 halfAngle = normalize(lightDir + eyeDir);
float specPower = maskRock * 32 + In.Color0.g * 4;
float specMask = maskRock * 0.6 + In.Color0.g * 0.3;
float specFactor = pow(saturate(dot(normal, halfAngle)), specPower) * specMask * specPerturb;
Color += clampedSunColor * specFactor;
#endif

// add fog
dist = saturate((dist - Params.x) * Params.y);
Color = lerp(Color, FogColor, dist);

#ifndef REFLECTION
// add underwater color
//Color = lerp(Color, UnderWaterColor, saturate(-In.VPos.y/MaxDepth));
#endif

// if ( In.VPos.y < 0 )
// {
// float ratio = -In.VPos.y / 3;
// float3 blendColor = {0.3, 0.7, 0.7};
// Color = lerp(Color, Color * blendColor, ratio);
// }

#ifdef REFLECTION
Color = lerp(Color, FogColor, 0.2);
return float4(Color, 1);
//return float4(Color,(1-saturate(In.VPos.y/40))*0.65);
//return float4(lerp(float3(0,0,0),Color,(1-saturate(In.VPos.y/40))*0.65), 1);
#endif
return float4(Color, 1);
#endif
}




TerrainSurfaceVS.fx

float4x4 matVP : register(c0);
float3 SunLightDir : register(c4);
float3 CamPosition : register(c5);
float4 Clouds : register(c6);
float4 Params : register(c7);
float4 NormalParams: register(c8);
float3 matW : register(c9);// for shadows, xyz = translation
//#undef SPECULAR

// NormalParams.x is the sTangent X and Z.
// NormalParams.y is the normal Y

// Params.x = fogZmin
// Params.y = 1.0 / ( fogZmax - fogZmin )
// Params.z = texture tile
// Params.w = detail texture tile

// Clouds.x, Clouds.y = clouds texture offset
// Clouds.z = clouds coverage

struct VS_IN
{
float4 Pos : POSITION;
float3 Normal : NORMAL;// Y holds the sTangent Y, build the normal with Normal.x, NormalParams.y, Normal.z
float4 Color0 : COLOR0;
float4 Color1 : COLOR1;
};

struct VS_OUT
{
float4 ProjPos : POSITION;
#ifdef OUTPUTDEPTH
float depth :TEXCOORD0;
#else
float4 Tex0 : TEXCOORD0;
float3 WSPos : TEXCOORD1;
#ifdef CLOUDS
float2 Tex2 : TEXCOORD2;
#endif
float3 SunDir : TEXCOORD3;
#ifdef SPECULAR
float3 EyeVertex : TEXCOORD4;
#endif
float3 VPos : TEXCOORD5;
float3 VNormal : TEXCOORD6;
float3 OProjPos : TEXCOORD7;// needed for shadow jittering
float4 Color0 : COLOR0;
float4 Color1 : COLOR1;
#endif
};

VS_OUT main( VS_IN In )
{
VS_OUT Out;

//float dist = distance(CamPosition, In.Pos);
//In.Pos.y *= min(2000, 2000 - max(0, dist - 1000)) / 2000;

#ifdef OUTPUTDEPTH
In.Pos.xyz += matW;
Out.ProjPos = mul(In.Pos, matVP);
Out.depth=Out.ProjPos.z;
return Out;
#else

Out.ProjPos = mul(In.Pos, matVP);

float2 TexCoord;
TexCoord.x = ( In.Pos.x + 250 ) / 500;
TexCoord.y = ( 250 - In.Pos.z ) / 500;

Out.Tex0.xy = TexCoord * Params.z;
Out.Tex0.zw = float2(TexCoord.x, 1 - TexCoord.y);// tex1 outputs the tex coords in the 0..1 range for each tile

#ifdef CLOUDS
Out.Tex2 = (TexCoord + Clouds.xy) * Clouds.z;
#endif

// decompress normal
In.Normal = (In.Normal - 0.5) * 2;
In.Normal.xz *= 255;// divided in code

// calc bitangent & normalize tangent vectors
float3 normal = normalize(float3(In.Normal.x, NormalParams.y, In.Normal.z));
float3 sTangent = normalize(float3(NormalParams.x, In.Normal.y, NormalParams.x));
float3 bitangent = normalize(cross(sTangent, normal));

// put light dir into tangent space
Out.SunDir.x = dot(sTangent, SunLightDir);
Out.SunDir.y = dot(bitangent, SunLightDir);
Out.SunDir.z = dot(normal, SunLightDir);

#ifdef SPECULAR
// put eye direction into tangent space
float3 eyeDir = normalize(CamPosition - In.Pos.xyz);
Out.EyeVertex.x = dot(sTangent, eyeDir);
Out.EyeVertex.y = dot(bitangent, eyeDir);
Out.EyeVertex.z = dot(normal, eyeDir);
#endif

Out.Color0 = In.Color0;
Out.Color1 = In.Color1;
Out.VPos = In.Pos.xyz;
Out.VNormal = float3(In.Normal.x, NormalParams.y, In.Normal.z);
Out.WSPos = In.Pos.xyz + matW;
Out.OProjPos = Out.ProjPos.xyz;

return Out;
#endif
};




TerrainUnderwaterPS.fx

#include <data/Shaders/Shadowmap/ShadowMapUtils.inc>

float3 SunLightColor : register(c0);
float4 AmbLightColor : register(c1);
float3 FogColor : register(c2);
float3 CausticsParam : register(c5);

// Caustics.x = caustics color factor
// Caustics.y = 1.0 / animation frames
// Caustics.z = blend factor between frames( 0 to 1 )

struct PS_IN
{
float4 Tex0 : TEXCOORD0;
float4 Tex2 : TEXCOORD1;
float3 SunDir : TEXCOORD2;
float3 TexParams : TEXCOORD3;
float4 Depth : TEXCOORD4;

#ifdef SHADOWS
float3 WSPos : TEXCOORD5;
#endif
};

sampler HardSlope : register(s0);
sampler Sand : register(s1);
sampler Normal : register(s4);
sampler Caustics : register(s12);// s7 is for shadows

float4 main( PS_IN In, float2 vPos : VPOS ) : COLOR
{
clip(-In.Depth.w);

float3 Color;

float3 HardSlopeC = tex2D(HardSlope, In.Tex0.xy);
float3 SandC = tex2D(Sand, In.Tex0.xy);
float3 NormalC = tex2D(Normal, In.Tex0.zw).xyz;
//NormalC.rg *= 0.5;
NormalC = normalize(NormalC);

Color = lerp(HardSlopeC, SandC, In.TexParams.y);

#ifdef SHADOWS
Color *= GetJitteredCascadeShadowMapValue_16(In.Depth.xyz, float4(In.WSPos.xyz, 1)).r;
#endif

// blend between 2 frames of animation, using the mipmap calculated in the vertex shader
// the lod level is in TexCoord.w, gets smaller with depth
float3 CausticsC1 = tex2Dlod(Caustics, In.Tex2);
In.Tex2.x += CausticsParam.y;
float3 CausticsC2 = tex2Dlod(Caustics, In.Tex2);

float3 lightDir = normalize(In.SunDir);
float lightCoef = saturate(dot(NormalC, lightDir));

//float3 colorFactor = min(1, SunLightColor * In.TexParams.x * CausticsParam.x);
Color += lerp(CausticsC1, CausticsC2, CausticsParam.z) * CausticsParam.x;// * colorFactor;
Color *= SunLightColor * lightCoef + AmbLightColor;

Color = lerp(Color, FogColor, In.TexParams.z);

return float4(Color, 1);
}




TerrainUnderwaterVS.fx

float4x4 matWVP : register(c0);
float3 SunLightDir : register(c4);
float3 CamPosition : register(c5);
float4 Caustics : register(c6);
float4 Params : register(c7);
float4 NormalParams: register(c8);
float3 matW : register(c9);

// NormalParams.x is the sTangent X and Z.
// NormalParams.y is the normal Y

// Caustics.x = texture offset
// Caustics.y = tile factor on X
// Caustics.z = tile factor on Y

struct VS_IN
{
float4 Pos : POSITION;
float3 Normal : NORMAL;// Y holds the sTangent Y, build the normal with Normal.x, NormalParams.y, Normal.z
float4 Color0 : COLOR0;
float4 Color1 : COLOR1;
};

struct VS_OUT
{
float4 ProjPos : POSITION;
float4 Tex0 : TEXCOORD0;
float4 Tex2 : TEXCOORD1;
float3 SunDir : TEXCOORD2;
float3 TexParams : TEXCOORD3;// put fog, color0.w and color1.w in a single texture param
float4 Depth : TEXCOORD4;// proj pos in xyz, height in w

#ifdef SHADOWS
float3 WSPos : TEXCOORD5;
#endif
};

VS_OUT main( VS_IN In )
{
VS_OUT Out;
Out.ProjPos = mul(In.Pos, matWVP);

float2 TexCoord;
TexCoord.x = ( In.Pos.x + 250 ) / 500;
TexCoord.y = ( 250 - In.Pos.z ) / 500;

Out.Tex0.xy = TexCoord * Params.z;
Out.Tex0.zw = TexCoord * (Params.w * 5);

Out.Depth.xyz = Out.ProjPos.xyz;

#ifdef SHADOWS
Out.WSPos.xyz = In.Pos.xyz + matW;
#endif

// offset the caustics texture
Out.Tex2.x = TexCoord.x * Caustics.y + Caustics.x;
Out.Tex2.y = TexCoord.y * Caustics.z;
Out.Tex2.z = 0;

// in Tex2.w put the mipmap level to lookup, it's smaller the bigger the depth
float depthVal = -min(0, In.Pos.y + 1.5); // from 0 to 15 meters clip to mipmap 0
depthVal = max(0, min(1, depthVal / 20));// depthVal has to be in the [0,1] range
Out.Tex2.w = round(9 * depthVal); // choose mimap level
Out.TexParams.x = min(5, Out.Tex2.w + 2); // put color modifier in texparams.w, make the caustics brighter with depth

// calc bitangent & normalize tangent vectors
float3 normal = normalize(float3(In.Normal.x, NormalParams.y, In.Normal.z));
float3 sTangent = normalize(float3(NormalParams.x, In.Normal.y, NormalParams.x));
float3 bitangent = normalize(cross(sTangent, normal));

// put light dir into tangent space
Out.SunDir.x = dot(sTangent, SunLightDir);
Out.SunDir.y = dot(bitangent, SunLightDir);
Out.SunDir.z = dot(normal, SunLightDir);

// fog distance
float dist = distance(CamPosition, In.Pos.xyz);
Out.TexParams.z = saturate((dist - Params.x) * Params.y);

Out.TexParams.y = In.Color0.r;
Out.Depth.w = In.Pos.y;
return Out;
};




Before I start banging my head against that big pile of code, is there anyone who is familiar with SH shaders and who knows where to look? :doh:

[gap]

I have a few questions for the subsim friends coming from NW Germany or W Denmark, or for whoever is familiar with any location(s) near Germany's and Denmark's North Sea coasts. Would you be so nice to answer, at your earliest convenience, as many as possible of them?


How common is wintertime snowfall in your area?


On the average, during which months one can expect a significant snowcover, staying within a radius of few kilometers from the sea?


Is there any season/month during which grass becomes dry, at least in part?


In which month deciduous tree leaves start turning yellow/orange/red?


In which month, after the winter break, deciduous trees start producing new leaves?


if you were to provide a simplified description of local vegetation, which tree species would you mention and in which number, knowing that the total must be eight? My current tree selection is:



2 x European beech (1 big size + 1 medium size)
1 x Common oak (medium size)
1 x Silver birch (medium size)
2 x Scots pine (mediu size)
2 x Norway spruce (medium size)


(small size = 10 m in heigth or lesser; medium size = more than 10 m but lesser than 20; big size = 20 m or more)

How realistic would that be?

[Muckenberg]

Hello Gap when I tried your mod so I was amazed. Really perfect work. I only worry that if I can finish the mod and integrate into TWoS, I will just sail around the coast and watch that splendor.
:Kaleun_Thumbs_Up::Kaleun_Thumbs_Up::Kaleun_Applau d::Kaleun_Salute:

[gap]

Hello Gap when I tried your mod so I was amazed. Really perfect work. I only worry that if I can finish the mod and integrate into TWoS, I will just sail around the coast and watch that splendor.
:Kaleun_Thumbs_Up::Kaleun_Thumbs_Up::Kaleun_Applau d::Kaleun_Salute:

Hi Muckenberg,
thank you for your encouragement; I am glad that you like what I have done so far!

The next update for NW Germany and W Denmark (that would include the U-boat base of Wilhelmshaven), should be ready in a matter of days. I have many of its files/textures ready. Among the features and the eye-candys of the next release, it is worth mentioning a decent rendition of the red sandstone cliffs of Helgoland that I am already quite satisfied with, though some small tweaks are still needed. I am now mostly focusing on fine-tuning/blendig the new stuff. I hope that in the meanwhile someone will answer my questions, so to enable me creating a realistic season/color sequence for the new terrain.

Answering your other remark, yes the mod is compatible with TWoS without need of special tweaks, and for someone wanting to use the EcoMod together with TWoS, only texture resolution should be pondered carefully for making sure that the game will run smootly also in the busy TWoS campaign (but you can ceasily switch from one texture resolution to another without any problem). Now it is a bit too early to talk about that, but if in future Vecko will decide merging the two mods, I won't have anything in contrary :up:

[tonschk]

Hi, thank you very much Gap :yeah:

[gap]

Hi, thank you very much Gap :yeah:

Hi mate, thank you too for your feedback :salute:

[gap]

With the help of Pintea, the weird lens flares reported by vdr1981 at post #16 (http://www.subsim.com/radioroom/showpost.php?p=2580034&postcount=16) are now gone.

https://i.imgur.com/rMKZxAV.png

https://i.imgur.com/2zq6lOM.png

The fix, together with several terrain improvements and a new terrain type will be part of the upcoming update. Stay tuned guys :sunny:

[Fifi]

F A N T A S T I C !!! :yeah:

[THEBERBSTER]

This is quite brilliant work gap. :Kaleun_Applaud:
Peter

[skin-nl]

Looking good gap :up:

[gap]

Thank you guys, all the merit for the fix goes to Pintea, aka Mihai Gosa; I only helped him with testing and remarks.

Mihai is one of the devs, and he is the one who dealt with shaders and environmental settings, so hopefully with his insight several other improvements are to be expected. He has always been very kind and helpful. Once he used to visit the forum regularly, and in a couple of occasions we had PM exchanges, but he doesn't work any longe for Ubi Romania and, despite his love for the game and for the subsim community that has not changed over time, now he rarely logs in here.

A big thanks goes to Jeff Groves too, who recently put me again in touch with him :salute:

[Muckenberg]

VAAAAAAAAUUUUUU Great job:Kaleun_Applaud::Kaleun_Applaud::Kaleun_Applaud ::Kaleun_Applaud::Kaleun_Applaud::Kaleun_Applaud:

[gap]

https://i.imgur.com/rZrW1Qp.png

Hi guys, version 0.2 of the EcoMod is ready for testing. Please visit post #1 (http://www.subsim.com/radioroom/showthread.php?p=2579821#post2579821) in this thread for the list of new features and download link, post #2 for an overview of the terrain types included in the mod and a list of their mean features, and post #3 for a few screenshots.

Enjoy and let me know what you think. Any type of criticism or suggestion is welcome here :up:

[Muckenberg]

Hello
Unfortunately, I have a submarine at a single mission in the Kiel Canal outside the water. The submarine is immediately destroyed and I can not look at that splendor. :Kaleun_Salute:

[gap]

Hello
Unfortunately, I have a submarine at a single mission in the Kiel Canal outside the water. The submarine is immediately destroyed and I can not look at that splendor. :Kaleun_Salute:

Really? With TWoS installed? :hmmm:

Okay, found what is the problem; it has been a long time since I last ran the game with OHII/TWoS installed. I normally run the game with a basic mod loadout, for ease of testing. Apparently among the many features that TWoS inherited from OHII, there is a more realistic Kiel Canal layout, with gates at the entrance, and the Brunsbuttel port moved a bit to the east. That's why in the current testing mission, the U-boat lays on land for TWoS users.

Tomorro I will post an update that will fix the problem. Thank you for reporting it Muckenberg!

:Kaleun_Salute:

[gap]

I have uploaded un updated version of EcoMod which fixes the Brunsbuttel testing missions for OHII/TWoS users. Other features are unchanged. As usual, find the download link at post #1.

Sorry for the inconvenience and a big thanks to Muckenberg for reporting it :salute:

[Seaowl]

WOW. What for a nice Christmas present. Thank you so much, Gap.
Started a single mission and chugged to Heligoland. All is working nicely so far. Landscape is beautiful.
And because this is my first post on Subsim: Ahoy to all Captains and merry christmas to everyone of course...
:Kaleun_Salute:

[gap]

WOW. What for a nice Christmas present. Thank you so much, Gap.
Started a single mission and chugged to Heligoland. All is working nicely so far. Landscape is beautiful.
And because this is my first post on Subsim: Ahoy to all Captains and merry christmas to everyone of course...
:Kaleun_Salute:

Heligoland will be the next area to be customized, with no trees and a different rock texture used.

Welcome to susbims and merry Xmas to you too Seaowl. It is nice having gentlemen like you aboard :salute:

[Muckenberg]

Hello
Downloaded, installed and I have to say that now everything is all right. :Kaleun_Salute::Kaleun_Applaud:
If I understand correctly, are you going to fix all the shores in the game? So is it premature to install your mod on an ongoing campaign and the best will wait for the final version? When are you going to the French coast?

[gap]

Hello
Downloaded, installed and I have to say that now everything is all right. :Kaleun_Salute::Kaleun_Applaud:

:up:

If I understand correctly, are you going to fix all the shores in the game?

If for 'fixing', you mean assigning to coastal areas proper vegetation and plausible landscape colors, then the answer is yes, I plan to modify more or less all the coasts, with higher priority to Europe and North Africa and the Atlantic Coast of North and Central America, but it will be a long process, and not all the areas will benefit from the same level of detail.


So is it premature to install your mod on an ongoing campaign and the best will wait for the final version?

Well, I hope someone to test my mod in campaign and tell me if it requires a campaign restart or being enabled while in bunker. Hopefully not, but I am not sure about this point. Maybe Vecko can say it better...

When are you going to the French coast?

My recent update already covers the North and West coasts of France, but I plan to diversify them with one of the next releases :yep:

[Muckenberg]

I installed your mod at sea during the Happy Hours, so I'll add it to Germany and I'll let you know. :Kaleun_Salute:

[gap]

I installed your mod at sea during the Happy Hours, so I'll add it to Germany and I'll let you know. :Kaleun_Salute:

That would be nice by you. :up:

With the last alpha version of the mod installed, you should expect the new terrain and vegetation textures to apply to most of Northwestern continental Europe, from W Denmark to SW France.

Don't forget to make a backup copy of the last savegame that you saved before enabling the mod, just in case the new savegames become broken because of it (they shouldn't, but you never know) :03:

[THEBERBSTER]

A Warm Welcome To The Subsim Community > Seaowl and a Merry Christmas
Subsim <> Make A Donation <> See The Benefits <> Support The Community (http://www.subsim.com/radioroom/showpost.php?p=2027002&postcount=1)
SH3 – 4 - 5 Tutorials > Downloads > Other Useful Information > See Links in My Signature Below

[Seaowl]

Thanks for the warm welcomes.
Even if not finished Heligoland looks a lot better with this mod enabled, than as grey basaltcluster before.
Eager to see the innovations of version 2.2.9, I will soon start a new TWOS-campaign and certainly using this mod.
I will report, if I run into some troubles.

[Aktungbby]

Seaowl!:Kaleun_Salute:

[Muckenberg]

Hi
Unfortunately, the BDU changed my home port to Lorient so I could not go to Germany to end the patrol.
In Lorient on departure, the mod was loaded correctly and there was no error.:Kaleun_Salute::Kaleun_Salute::Kaleun_Thumbs _Up:

[gap]

Hi
Unfortunately, the BDU changed my home port to Lorient so I could not go to Germany to end the patrol.
In Lorient on departure, the mod was loaded correctly and there was no error.:Kaleun_Salute::Kaleun_Salute::Kaleun_Thumbs _Up:

Perfect! That means that terrain/texture information is read on the fly and no campaign restart is required. Thank you for helping me discovering that :up:

[Jeff-Groves]

Really glad you and Mihai hooked up!
:salute:

[gap]

Really glad you and Mihai hooked up!
:salute:

What do you know about LOD distances in the GR2 format? :D

[Jeff-Groves]

As far as I know off the top of my head there is nothing in the GR2 about LOD. No extra 3D models or such.
Now SH4 has a LOD setting but I don't recall without looking where it's at.
Let me check a few things and get back to you on this.

[gap]

What do you know about LOD distances in the GR2 format? :D

I will make myself clearer. This is a first rework of Helgoland:

https://i.imgur.com/kpzDCgq.png

Maybe, on the far right, you will notice a sea stack, the famous Lange Anna, that I have modelled and imported in game.

The problem is that it keeps popping in and out, depending on camera distance and angle. The same goes for shadows, water reflections and other lighting effects. The same applies to other terrain objects (but not all of them!), though normally we won't notice them disappearing as we move away from them due to their proportionally smaller dimensions and to their fading in the background.
IIRC, SHIII and IV terrain objects have far rendering controllers which set the minimum dimension of objects printed on screen before the game stops rendering them. In SH5, sea and air units have LOD models, but nowhere I could find a controller similar to the ones used in SHIII. Units and terrain objects are probably treated in a different way. For one, the latter have no LOD models, when the player is too far from them, they simply disappear, but as I said they all don't disappear at the same time. An object-specific pararmeter must tell the game when it is time to stop rendering any given object and since most of them have only the StaticObject and the HarborObjectCtrl controller applied to them (one related with collision, and the other with night/winter textures), this something must be stored directly in the GR2 models :hmmm:

[Jeff-Groves]

Is that imported as a dat file with the Unifiedrender controller?
There is a LOD factor under that.

[gap]

Is that imported as a dat file with the Unifiedrender controller?
There is a LOD factor under that.

No, it is a GR2 file. I imported it into Iceberg01.GR2 first, and then into church.GR2. Little seems to have changed though.

Here is a closer view of the red stack:

https://i.imgur.com/pKTM2ij.png

Considering how low poly it is, I think it looks pretty nice, especially when the lighting effects "pop in" :)

[Jeff-Groves]

what texture are you using and how many mip maps?

[gap]

what texture are you using and how many mip maps?

diffuse map: DX5 dds format, 1024 x 1024 pixels, 11 mip levels, with self-illumination map in the alpha channel.
ambient occlusion and normal maps: DX1 dds format, 1024 x 1024 pixels, 11 mip levels each.

PS: congratulations for your recently earned and well deserved patch :)

[Jeff-Groves]

Scene LOD for the iceburg is 6.027+ to 12.055+
Scene LOD for most terrain GR2's are higher. Maybe that is the problem.

I'll look into the GR2 and see if we can up the LOD range.

[gap]

Scene LOD for the iceburg is 6.027+ to 12.055+
Scene LOD for most terrain GR2's are higher. Maybe that is the problem.

I'll look into the GR2 and see if we can up the LOD range.

Well, among bone's extended data editable through TDW's editor, there is a LODError whose settings (for iceberg models) are more or less in the range that you have just mentioned. I don't know if that's what you meant, but I already played with those settings, though without getting clear results. Maybe I should persist in that direction, and try copying LODError values from the land objects which seem to have the longest LOD distances in game :hmm2:

[Seaowl]

Kaleun no more.
Now it´s time to leave the Kriegsmarine and join the career of a cruise ship captain.
Beautiful work!
:Kaleun_Applaud:

[gap]

Kaleun no more.
Now it´s time to leave the Kriegsmarine and join the career of a cruise ship captain.

:03:


Beautiful work!
:Kaleun_Applaud:

Thank you Seaowl, if I manage fixing the LOD-related problems, it will be part of the next release :up:

[Jeff-Groves]

Your inbox is full. :03:

[gap]

Your inbox is full. :03:

Yep, I know. As usual. I made some space now :D

[OldCoder]

Amazing work going on here. Thank you so much.

[gap]

Amazing work going on here. Thank you so much.

Thank you OldCoder. Christmas/New Year celebrations and some unexpected problems with terrain object's LOD distances, have slowed down my work on it, but expect an update within the next few days :salute:

[Muckenberg]

Hi
I look forward to the next update. The coast looks absolutely fantastic. And I have to admit I'm really looking forward to the Norwegian fjords. :Kaleun_Salute: :Kaleun_Applaud:

[gap]

Hi
I look forward to the next update. The coast looks absolutely fantastic. And I have to admit I'm really looking forward to the Norwegian fjords. :Kaleun_Salute: :Kaleun_Applaud:

Thank you Muckenberg!

It will take a while before I start editing Norwegian fjords; I had planned to finish with Eurepean temperate terrain types first, but if you have direct experience of Norwegian vegetation and landscapes (see post #29 (http://www.subsim.com/radioroom/showpost.php?p=2580812&postcount=29) in this thread for the kind of information that I looking for), I might actually do an exception :03: :salute:

[Muckenberg]

Hi Gap
Unfortunately, I'm from the Czech Republic, so I do not know the details of the Norwegian climate. Holt will have to wait for the fjords, but that does not mean I will not look forward. You do a great job and push SH5 a little higher.

[iambecomelife]

Just wanted to break in and say, excellent work!
The fields are so lush and yummy looking. It's the lack of stuff like this that turned me off SH5. Love how the sea stack has geological layers - reminds me of the hills in Pennsylvania.

[gap]

Hi Gap
Unfortunately, I'm from the Czech Republic, so I do not know the details of the Norwegian climate. Holt will have to wait for the fjords, but that does not mean I will not look forward. You do a great job and push SH5 a little higher.

Just wanted to break in and say, excellent work!
The fields are so lush and yummy looking. It's the lack of stuff like this that turned me off SH5. Love how the sea stack has geological layers - reminds me of the hills in Pennsylvania.

Thank you guys, I had no time to work on this mod in the last weeks, but I am planning an update within the next few days :salute:

[gap]

Quick update on terrain object's LOD issue

As expected, the MinRenderDim controller does nothing on GR2 object, but there is more: wanting to discover what made LOD ranges too short for my sea stack model, I created a copy of one of the stock objects with the longest ranges, and I imported my model into it.

I supposed LOD ranges to be governed by some little know or hidden parameter in mesh's extended data, and I hoped that after switching to a GR2 file with the appropriate ExtendedData settings my model would have been visible from further away, but unfortunately I was wrong and, at close distance, the model was popping in and out as I moved closer of further away from it.

The next step was copying the same (stock) GR2 file used in my previous test, and using it in place of the sea stack without further changes. Result: I could see the model (a group of buildings) from a long distance with no sudden "pop up effect".

At this point I started thinking that LOD distances could be ID/bone name related and that by renaming the bones I could have made the game not to recognize my object as a proper terrain object. To check that I imported the sea stack mesh in the same file as before without changing bone names, but again LOD ranges were too short. That excluded bone names from my list of suspects and it left me with just one reasonable hypothesis LOD ranges are mesh-related. To confirm that, I re-imported the original (stock) mesh in the file of my tests and, in accordance with my last statement, the object was visible again from a long distance.

What can tie meshes and LOD distances? There must be an obvious link, but I am missing it :hmmm:

[Jeff-Groves]

Can you post pics of what your talking about?

[gap]

Can you post pics of what your talking about?

Please check your e-amail: I sent you a copy of my working files and a description of them. I hope that by reproducing the issue on your machine and by looking directly into the files, you will make yourself a better idea on the issue than you could get from simple screenshots :)

[Jeff-Groves]

OK. I see what you are talking about now.
Try changing the collision mesh with the one the city GR2 has.
I'd do it but I'm not home and don't have all the needed programs with me.
:oops:

If that solves the issue create a new collision mesh with a broad base but deep enough to not hit a sub or ship in the area.
I'm looking out the window of my temporary apartment in Cleveland, Ohio right now.
https://i108.photobucket.com/albums/n12/privateer_2006/river.jpg

All that view needs is a VIIA in the river. LOL

[gap]

OK. I see what you are talking about now.
Try changing the collision mesh with the one the city GR2 has.
I'd do it but I'm not home and don't have all the needed programs with me.
:oops:

If that solves the issue create a new collision mesh with a broad base but deep enough to not hit a sub or ship in the area.


I think I've got your point, I will do that and I will let you know if it works :up:


I'm looking out the window of my temporary apartment in Cleveland, Ohio right now.
https://i108.photobucket.com/albums/n12/privateer_2006/river.jpg

All that view needs is a VIIA in the river. LOL

So nice! I wish we had a SH Great Lakes campaign lol :D

[gap]

Try changing the collision mesh with the one the city GR2 has.

Jeff, you are the man!
After so much messing, the too short LOD range issue is gone for good, and the red pillar rock is now visible from far, far away :yeah:

https://i.imgur.com/uMSEKRH.png

What matters for terrain object's rendering distance isn't actually the size of the collision mesh, but the radius of main model's bounding sphere. You put me in the right direction, thank you very much!

This little discovery means that we can add to the game any kind of landmark with all the goodies of the GR2 format. You know what will the next model that I am going to import. Time to PM Jim :sunny:

[Seaowl]

Amazing!
Thanks to both of you for all these efforts.
Beeing on the edge for the next release now...
:Kaleun_Applaud:

[Muckenberg]

:Kaleun_Thumbs_Up: :Kaleun_Applaud: :Kaleun_Thumbs_Up: :Kaleun_Applaud:

[Jeff-Groves]

Fantastic work My Friend!
:salute:

[gap]

Fantastic work My Friend!
:salute:

Now that the rendering range issue is solved, something else bothers me :hmmm:

How it is:
https://i.imgur.com/2xHkpqm.png

How it should be:
https://i.imgur.com/vtjHBuN.png

Bottom drawing courtesy of Kendras, re-scaled and georeferenced by me.

Each quadrant in the above charts corresponds to one of the SH5 height maps and it measures 2.5 x 2.5 minutes (i.e. 5 x 5 km2). The map resolution is 25 meters. SH5 terrain can be edited using the inbuilt Terrain Editor. The output is one or more of the height maps that I described above.
Unfortunately their editing in photoshop is not easy. You can open them, but all you is an apparently meaningless bunch of grey pixels. The height information contained in them seems encoded or mixed with noise of some sort :hmmm:

[Jeff-Groves]

You can see better details if you zoom out to 12.5% actual size.

Not that it will help to hand edit them but take a look at them like this.
You may figure something out.
:03:

[gap]

You can see better details if you zoom out to 12.5% actual size.

Not that it will help to hand edit them but take a look at them like this.
You may figure something out.
:03:

Yes, you are right: if you look at them you can see a vague resemblance with the terrain profile, but very faint. I asked Mihai on the possibility to convert TE-generated raw files to something that we can edit even outside TE, and I have just received an answer:

Here you go, though this won’t help you much since photoshop can’t do custom things like you need.





elevations[256] =

{

// Underwater band

-1500.f, -448.f, -416.f, -384.f, -352.f, -320.f, -288.f, -256.f,

-234.f, -220.f, -211.f, -202.f, -193.f, -184.f, -175.f, -166.f,

-157.f, -148.f, -137.f, -128.f, -120.f, -111.f, -104.f, -97.f,

-90.f, -85.f, -80.f, -76.f, -72.f, -68.f, -64.f, -60.f,

-56.f, -52.f, -48.f, -44.f, -40.f, -37.f, -34.f, -32.f,

-30.f, -28.f, -27.f, -26.f, -25.f, -24.f, -20.f, -10.f,



// Skipping from -15 directly to above zero makes the shore look jagged,

// but avoids problems with ships running into low ground around harbors and shore areas

// Ideally, all harbor areas and battle zones should be manually edited to lower the ground in ships' paths.



// 0-64 meter band

10.f, 20.f, 25.f, 20.f, 25.f, 30.f, 35.f, 40.f,

44.f, 45.f, 46.f, 47.f, 48.f, 49.f, 50.f, 51.f,

52.f, 53.f, 54.f, 55.f, 56.f, 57.f, 58.f, 59.f,

60.f, 61.f, 62.f, 63.f, 64.f, 65.f, 66.f, 67.f,



// 64-192 meter band

68.f, 70.f, 72.f, 76.f, 80.f, 84.f, 88.f, 92.f,

96.f, 100.f, 104.f, 108.f, 112.f, 116.f, 120.f, 124.f,

128.f, 132.f, 136.f, 140.f, 144.f, 148.f, 152.f, 156.f,

160.f, 164.f, 168.f, 172.f, 176.f, 180.f, 184.f, 188.f,



// 192-448 meter band

192.f, 200.f, 208.f, 216.f, 224.f, 232.f, 240.f, 248.f,

256.f, 264.f, 272.f, 280.f, 288.f, 296.f, 304.f, 312.f,

320.f, 328.f, 336.f, 344.f, 352.f, 360.f, 368.f, 376.f,

384.f, 392.f, 400.f, 408.f, 416.f, 424.f, 432.f, 440.f,



// 448-960 meter band

448.f, 464.f, 480.f, 496.f, 512.f, 528.f, 544.f, 560.f,

576.f, 592.f, 608.f, 624.f, 640.f, 656.f, 672.f, 688.f,

704.f, 720.f, 736.f, 752.f, 768.f, 784.f, 800.f, 816.f,

832.f, 848.f, 864.f, 880.f, 896.f, 912.f, 928.f, 944.f,



// 960-1984 meter band

960.f, 992.f, 1024.f, 1056.f, 1088.f, 1120.f, 1152.f, 1184.f,

1216.f, 1248.f, 1280.f, 1312.f, 1344.f, 1376.f, 1408.f, 1440.f,

1472.f, 1504.f, 1536.f, 1568.f, 1600.f, 1632.f, 1664.f, 1696.f,

1728.f, 1760.f, 1792.f, 1824.f, 1856.f, 1888.f, 1920.f, 1952.f,



// 1984-8000 meter band

1984.f, 2048.f, 2112.f, 2176.f, 2240.f, 2304.f, 2368.f, 2432.f,

2496.f, 2560.f, 2624.f, 2688.f, 2752.f, 2816.f, 2880.f, 2944.f,

3008.f, 3072.f, 3136.f, 3200.f, 3264.f, 3392.f, 3584.f, 3712.f,

3840.f, 3968.f, 4032.f, 4160.f, 4288.f, 4416.f, 4544.f, 4762.f,

4800.f, 5184.f, 5412.f, 5668.f, 5846.f, 6080.f, 6208.f, 6436.f,

6664.f, 6892.f, 7020.f, 7276.f, 7404.f, 7632.f, 7888.f, 8000.f

}

Whad do you get from it? :hmmm:

[Jeff-Groves]

Appears to be what is hard coded somewhere is my guess at the moment.

[gap]

Appears to be what is hard coded somewhere is my guess at the moment.

I wish that we could read that matrix in ascending order from the top left to the bottom right, assigning to each height value a grey value as follows:

-1500=0 ; -448=1 ; ... ; 7888=254 ; 8000=255

Am I asking too much? :O:

EDIT: I have just received confirmation; my guessing was correct :up:

Now I need to create two custom palettes: one for sorting grey values of those raw files in ascending height order, so that we can edit them in photoshop, and a second palette for sorting values back in the SH5 (dis-)order :doh:

[Jeff-Groves]

I'm playing with one of the RAW files now.
So the scale is 256 colors as the code snippet shows.
Each pixel is 8 bits I believe.
In Hex that all checks out.

[gap]

I'm playing with one of the RAW files now.
In hex they are not a perfect match to what I think is the floats chart listed but real close.

So the scale is 256 colors as the code snippet shows.
Each pixel is 8 bytes I believe.

Before you waste your time on complex calculations read my edit below please: :03:


EDIT: I have just received confirmation; my guessing was correct :up:

Now I need to create two custom palettes: one for sorting grey values of those raw files in ascending height order, so that we can edit them in photoshop, and a second palette for sorting values back in the SH5 (dis-)order :doh:

[gap]

Something doesn't add up :hmmm:

At first I thought that the height values in the code provided by Mihai was in casual order, but with only a few exceptions that's not true. Most values are sorted in ascending order. If the matrix was read from top left to bottom right, the grey values on the raw heightmaps should be much more intelligible than they actually are...

[gap]

These are the five heightmaps composing Helgoland and Dune, assembled together:

https://i.imgur.com/JkZiQOt.png

As a test, I selected all the white pixels (#ffffff). If there was an univocal correspondance between grey values on the raw maps and elevations in game, I would have expected pixels with the same intensity to coincide with areas having at least similar heights. Well, this is the result of my selection, overlayed to the chart of the two islands, as we see it on the navigation map:

https://i.imgur.com/VbmZ5Ia.png

Obviously some kind of noise gets applied to the height information of those heightmaps, and/or they don't represent absolute elevation values, but variations of elevation over terrain elevation data stored in TerrainData.BFD :-?

[Jeff-Groves]

What matters for terrain object's rendering distance isn't actually the size of the collision mesh, but the radius of main model's bounding sphere.

Something I posted from back in 2011.

"The bounding sphere is not relevant for SH5 that I could prove.
Maybe in a LOD situation?"

:har:

[gap]

Something I posted from back in 2011.

"The bounding sphere is not relevant for SH5 that I could prove.
Maybe in a LOD situation?"

:har:

You see in the future :D

[Jeff-Groves]

Reading back through that thread? I forgot how much I knew or suspected about the GR2 files at the time.
:o

[gap]

Reading back through that thread? I forgot how much I knew or suspected about the GR2 files at the time.
:o

Can you post a link to it? I like reading old threads, I must be affected by some sort of necrophilia :O:

[Jeff-Groves]

The GR2 viewer/editor thread.
:)
http://www.subsim.com/radioroom/showthread.php?t=188290

Post 284 is where I got that quote I posted above.

[gap]

The GR2 viewer/editor thread.
:)
http://www.subsim.com/radioroom/showthread.php?t=188290

Post 284 is where I got that quote I posted above.

That long thread is one of those that I need to re-read from the first to the last post. When it was started I wasn't probably sure even about the difference between a mesh and a texture, but with the little experience I have gained in the meanwhile I am sure I would learn a lot of new things from it :up:

[skipjack578]

What a great "little" project, really great mod. Not only that we gonna have new beautiful terrain but also an education. :salute:
May i ask how did you get to this idea in the first place to make EcoMod?

[Jeff-Groves]

You'll have to wait to see what he's working on next.
Lot's going on in the background!
:yeah:

[gap]

What a great "little" project, really great mod. Not only that we gonna have new beautiful terrain but also an education. :salute:
May i ask how did you get to this idea in the first place to make EcoMod?

Hi skipjack578 and thank you for showing your interest in this mod, forgive me for my late reply, but as Jeff said, besides private life routine there is something else - related to this mod - that kept us busy lately. More on it within the next few days.

To reply to your question: I like exploring new places, both real and virtual, and I have always been disappointed by the lack of variety/realism of the SH5 world. For this reason, in 2011 I and stoianm started working on Dynamic Environment, a mod that added to the game several new climate zones, weather patterns and environment colors (i.e. sea and atmosphere colors). See DynEnv's old thread (http://www.subsim.com/radioroom/showthread.php?p=1639246#post1639246) if you are interested in the history of its development.

Back then, stoianm asked me to have a look into terrain files. I did, and that disclosed to me a whole range of new modding potentials, but at that time I lacked some important information for exploiting them. The most important piece of information came more than a year after the first release of DynEnv, with this post (http://www.subsim.com/radioroom/showpost.php?p=1957612&postcount=29) by Pintea on echolot's "Smaller trees" thread. By then, other personal and modding projects prevented me from explicitly working on anything related to SH5 terrain, but in the free time I started gathering the real world information needed for the planning of the "SH5 EcoMod".

More recently, I started helping my good friend Kendras with his SHIII Lighthouse mod (http://www.subsim.com/radioroom/showthread.php?p=2451598#post2451598) project, and during its making we found that some little islands off NW France were missing from the SHIII world. That prompted me to edit the terrain of the latter game and to create a new terrain type for one of the missing islands, but while working on those items I realized the limits of SHIII when it comes to terrain's 3D shape and textures (though I think it has a more flexible handling of trees than SH5). That observation prompted me to go back to SH5 and to apply to it the little experience I had learned in the meanwhile, starting the present mod.

Despite some important differences, the core systems defining terrain and tree textures in the two games are pretty similar; converting from a SH5 terrain type mod to a SHIII or SHIV one, shouldn't be impossible. Things are more complicated when it comes to terrain shape though. Again, the core files of the two games are pretty similar, but SH5 adopts a completely new method for adding more detail to the areas of biggest interest. So far, we had the inbuilt SH5 Terrain Editor for creating mountains, plains, shallow water basins and deep canyons directly in game, but while enabling us to manually perform very precise terrain changes, to the level of ± a few decimeters, that tool doesn't allow the overlapping of a grid system or real world topographic maps, needed for very accurate terrain modifications. This is where a raster graphics editor or a 3D modelling program could be preferable to TE.

This brings us to these days; as far as I know, no one before has tried to bring in game the level of terrain realism that I am aiming at. Jeff and Pintea are helping me to understand the file format used for storing SH5's detailed height maps and to build the tools needed for converting that data to a format that can be easily edited with Photoshop or 3dsMax and re-imported in game. Thanks to them, tings are finally starting to make sense; we have already a tool capable of extracting height information from the game files, work of Jeff, and I have created a rudimentary tool based on Excel for visualizing that information as a greyscale map. If there is anyone interested in knowing more in detail the ins and outs of our work, I will be glad to explain them when our tools will be finished and when we will be sure about the correctness of our method.

The reshaping of the game terrain so that if reflects the geography of some conspicuous areas, the re-making of some cities and harbors so that they resemble more closely their real counterparts, and the addition to the game of some new 3D landmarks where they were to be expected in the world of 80 years ago, are features that I didn't plan initially for this mod. They will slow down somewhat the main plan of adding in game new vegetation types and terrain textures, but I hope that at the end they will be worth your patience and all the efforts we are putting on them.

Stay tuned guys :salute:

[Jeff-Groves]

gap gives good credits to several of us but believe me! He's kicking out some killer ideas that will actually work for what We are doing.
:rock:

[gap]

gap gives good credits to several of us but believe me! He's kicking out some killer ideas that will actually work for what We are doing.
:rock:

You are too kind. I am just lucky because I know who to ask when I need help :|\\

[Jeff-Groves]

I think Pintea would be impressed with your latest idea.
I know I was!
:salute:

[THE_MASK]

OT
"Due to certain broken aspects of SH5's launcher, there is a possibility that even upon reaching campaign chapter ending date, campaign transfer still may fail to initialize"
Is this still the case ?

[Jeff-Groves]

:doh:
Sure your in the right thread?
:hmmm:

[gap]

OT
"Due to certain broken aspects of SH5's launcher, there is a possibility that even upon reaching campaign chapter ending date, campaign transfer still may fail to initialize"
Is this still the case ?

You should ask the same question in the TWoS thread sober; many valiant captains actively involved in the Battle of the Atlantic check that thread on a daily basis, and I am sure they will give you an answer. Here you will meet mostly mechanics and engineers, more interested in the inner workings of the game than in the progress of the war :D

[skipjack578]

Im glad you guys are doing this project, i don't know much about what you said in technical terms. :D
But iv been at sea a lot to know what you mean, i have seen every color there is on the sea, the sea has all the colors...so you can't go wrong.

Keep up the good work! :salute:

[Jeff-Groves]

Very early view of something special.
http://www.subsim.com/radioroom/picture.php?albumid=1069&pictureid=10110

This is the Alpha 1 Terrain Editor We are working on for SH5.
It has a long way to go but early testing shows the promise.
It is locked to only allow up and down edits.

[propbeanie]

Is the orange color of the overview your edit point, or the slope indicator? :salute:

[Jeff-Groves]

Is the orange color of the overview your edit point, or the slope indicator? :salute:

The Alpha is based on an indexed color map.
It's actually the source code from a long ago project for something not related to SH Games.
I had to do a color map based off the RAW files in SH5 to render this image. That will not work for our needs but the source code can be adapted.
Rather then build a new wheel? I plan to re-tread this old wheel.

[propbeanie]

Old wheels are like old guys: they can work pretty good. dependable??... :roll: :har:

[gap]

In addition to the anticipations by Jeff, here are three screenies illustrating what we have been working on in the last two weeks:

https://i.imgur.com/AXr3wz3.png

The picture on top is a in-game screenshot illustrating the NW tip od Heligoland, and it is here for visual reference, while the two bottom pictures are respectively a greyscale height-map and a colorized 3D mesh of the same area. Both of them were calculated thanks to a RAW-data exporter created by Jeff with the information kindly provided by Pintea (I will never acknowledge enough these amazing guys :sunny:), and with a spreadsheet I deviced for sorting the RAW elevation data extracted by Jeff and for putting it in a format that can be easily edited and re-imported in game.

The workflow I described above is already obsolete though: since I took those pictures Jeff has created a new script for 010 editor capable of quickly writing the SH5 elevation data directly in an obj file (with no need of my spreadsheet) and, as he said, he is already thinking about a standalone importer/exporter with more advanced features. The general idea is enabling us to edit the SH5 terrain using external programs like Gimp/Paint.Net and/or Blender/Wings3D, and to easily switch from a format/editing method to the other.

The advantages of each of the above editing methods, complementary to the inbuilt Terrain Editor, are many, from the ability of overlapping detailed maps to the terrain export the we want to edit, to the centimetric accuracy that some of them enable, to the faculty of putting in game the information of conveniently scaled real-world height maps. Within the limits of the SH5 world and of the map projection it uses, new realism frontiers are opening up for us, and they will only wait for us to exploit them! :yep:

[OldCoder]

If that is what I think it is, could we also be seeing possible future visual improvements to the Dover coastline from the English Channel?

[gap]

If that is what I think it is, could we also be seeing possible future visual improvements to the Dover coastline from the English Channel?

My answer is definitely yes!

That's something I have had in mind for a loooong time. The white cliffs of Dover and their French counterpart of Étretat, could already be modelled with the Terrain Editor, but with the new tools we can bring them to a new level of realism :cool:

[OldCoder]

Very exciting. As I live just a few minutes walk from such cliffs, it would be great to see the game able to render them better.

[gap]

Very exciting. As I live just a few minutes walk from such cliffs, it would be great to see the game able to render them better.

If you have topographic maps, heightmaps, or anything showing in detail the altitudinal features of the area you are living in, this is about the right time to share them here :)

[Seaowl]

WOW!! Fantastic! Great work.:Kaleun_Thumbs_Up:
So now it´s possible to import some Lighthouserocks directly into the map.
And this lighthouses will be desperatly needed, because the shelves will become real too...


Is there more or less the same map in SH3-4-5?

So that everything, that would be done for one part, could be used for the others too?

[OldCoder]

gap, is the en-gb.topographic-map.com site any use? For instance if you go there and type in Dover, you can see the height map for the coast line. I used to have printed nautical maps for the Ramsgate/Dover area showing height maps under the water but sadly I don't have those any more. :(

[Seaowl]

The "National Library of Scotland" has a very nice collection of maps (not only from scotland).


Here is the link:


https://maps.nls.uk/index.html

[Jeff-Groves]

What would be really nice is digital information on heights!

[hauangua]

What would be really nice is digital information on heights!

Maybe I wrong... But can help this? :

http://en-us.topographic-map.com/places/White-Cliffs-of-Dover-9575707/

[bstanko6]

Nice work. Works great. The 2x 3x trees slows things down a bit, so I don't use them. Good work.

[OldCoder]

Jeff, this may be your answer:

https://github.com/Jorl17/open-elevation/blob/master/docs/api.md

It's a public/free API for elevation queries. Examples are there to call it from curl, but obviously you could easily make a Python script/whatever to send queries.

For instance, in a web browser:

https://api.open-elevation.com/api/v1/lookup?locations=51.138584,1.366985|51.123826,1.31 7397

Response:
{"results": [{"latitude": 51.138584, "elevation": 105, "longitude": 1.366985}, {"latitude": 51.123826, "elevation": 7, "longitude": 1.317397}]}

The first set of coordinates are the cliffs to the eastern side of Dover port, returning a height of 105 metres.

The second set of coordinates are also Dover near the sands/seafront shopping area, returning 7m.

The API does respond quite slowly unfortunately (at least for me), so that might be a limiting factor.

[OldCoder]

Also Jeff, to solve the performance issue it should be possible to self-host the API (and data if you have large hard drives), which you probably already realised it being on github. this issue issue related to the above is worth a read. https://github.com/Jorl17/open-elevation/issues/27. It also mentions higher resolution map data at https://www.eorc.jaxa.jp/ALOS/en/aw3d30/index.htm

[Jeff-Groves]

Thanks OldCoder
:salute:

Some interesting information in the links. I'll do more research into them.

What I did find in some formats is that the scale is wrong for SH5.
The z or up/down is fine but the x y is wrong. I think it could be averaged for our use.
Say we needed 5 units from a data set to make one unit for SH5. We could just average that and be close.

[Jeff-Groves]

So far I see the DSM .asc format as the most promising to work with.
I'll admit I have not followed it up fully.

[gap]

WOW!! Fantastic! Great work.:Kaleun_Thumbs_Up:
So now it´s possible to import some Lighthouserocks directly into the map.
And this lighthouses will be desperatly needed, because the shelves will become real too...

That might depend on the size of each rock. SH5's height maps have a resolution of one point every fifty meters so, for little single rocks whose diameter is lesser than a few hundred meters, it might be more convenient resorting to external 3D meshes set as terrain objects. As for sandbanks, shoals, (semi-)submerged reefs, and other seabed natural features, indeed we can realistically model them directly in the terrain, as long as we have real bathymetric charts available.


Is there more or less the same map in SH3-4-5?

So that everything, that would be done for one part, could be used for the others too?

Yes and no.

I think the base terrain files are common to all the SH games. Demonstration of that is the fact that, as tested by Jeff, the tool used by the GWX team for editing terrain files also works with SH5. The resolution of those files is extremely low though, with elevation points set 200 meters apart. As far as I could understand from Pintea's detailed explanations, SH5 uses two methods for increasing that resolution:


in general areas, the terrain resolution is increased in memory to one point every 50 meters, and noise is applied to make the 'fake' heigh points not to look flat.


In areas of bigger interest, Terrain Editor can be used to create 'real' height changes. The editor saves the new terrain elevation values as RAW data files, which are exactly the ones I and Jeff have been working on. Each RAW file covers an area of 5 x 5 sq. km, and again it has a resolution of one height point every 50 meters. RAW files work as terrain patches: i.e. when the game finds those files for areas within rendering range from the camera, low -detail height data from base terrain files gets overwritten with data from the said RAW files.


Unfortunately SHIII (and probably SHIV) has nothing like that, so yes, in theory we could convert the detailed height data from RAW files into greyscale heightmaps, as used by SH5's predecessors, but due to their poor resolution, they wouldn't look not nearly as good as in SH5.

gap, is the en-gb.topographic-map.com site any use? For instance if you go there and type in Dover, you can see the height map for the coast line. I used to have printed nautical maps for the Ramsgate/Dover area showing height maps under the water but sadly I don't have those any more. :(

Maybe I wrong... But can help this? :

http://en-us.topographic-map.com/places/White-Cliffs-of-Dover-9575707/

Thank you guys, I knew that website. Their height maps look good, but they have a problem: they are colored, which makes the grabbing of their information more complicated for us (greyscale i.e. 8-bit maps are much more easy to handle, though they have a poorer vertical resolution), and they are overlapped with location names, road symbols etc. which make them unusable for our purposes, unless we find a way to toggle them off (which I didn't find so far).

NASA maps are much easier to use though not all the planet is covered and seabed depth information is not provided by them:

https://gdex.cr.usgs.gov/gdex/

The "National Library of Scotland" has a very nice collection of maps (not only from scotland).


Here is the link:


https://maps.nls.uk/index.html

Very nice, thanks, I will have a close look into that website! :up:

Nice work. Works great. The 2x 3x trees slows things down a bit, so I don't use them. Good work.

Okay, good to know. The next release will have also a couple of optional mods (work of Pintea) that will make trees 25% bigger or 25% smaller than in main mod. You might use bigger trees (as in stock game), to compensate for their sparser density :)

What would be really nice is digital information on heights!

Jeff, this may be your answer:

https://github.com/Jorl17/open-elevation/blob/master/docs/api.md

It's a public/free API for elevation queries. Examples are there to call it from curl, but obviously you could easily make a Python script/whatever to send queries.

For instance, in a web browser:

https://api.open-elevation.com/api/v1/lookup?locations=51.138584,1.366985|51.123826,1.31 7397

Response:
{"results": [{"latitude": 51.138584, "elevation": 105, "longitude": 1.366985}, {"latitude": 51.123826, "elevation": 7, "longitude": 1.317397}]}

The first set of coordinates are the cliffs to the eastern side of Dover port, returning a height of 105 metres.

The second set of coordinates are also Dover near the sands/seafront shopping area, returning 7m.

The API does respond quite slowly unfortunately (at least for me), so that might be a limiting factor.

Also Jeff, to solve the performance issue it should be possible to self-host the API (and data if you have large hard drives), which you probably already realised it being on github. this issue issue related to the above is worth a read. https://github.com/Jorl17/open-elevation/issues/27. It also mentions higher resolution map data at https://www.eorc.jaxa.jp/ALOS/en/aw3d30/index.htm

That's very nice, though for me the API is so slow that I got a timeout error most of the times.

I say sorry if my questions might sound a bit silly but can I change coordinates values in web browser-based query with whatever world coordinate I want? Where are those height values obtained from?

Thanks OldCoder
:salute:

Some interesting information in the links. I'll do more research into them.

What I did find in some formats is that the scale is wrong for SH5.
The z or up/down is fine but the x y is wrong. I think it could be averaged for our use.
Say we needed 5 units from a data set to make one unit for SH5. We could just average that and be close.

I don't quite get you here Jeff :06:

[Seaowl]

As you mentioned height points every 50 meters:
On this site I´ve found some 50m Digitalterrainmodels (also available in 25m, 1“ and 3“)

http://data.opendataportal.at/dataset/dtm-norway

I downloaded Denmark and Norway.
Into the folders, I found .prj, .tfw and .tif files.
The .tif files I was able to open with Gimp: They contain black and white pictures without inscriptions.
Unfortunatly no seabed data (mainland is white, sea is black)
Don´t know, if this is of any use for you?

As far as I´ve seen, there is a difference between digital terrain and digital surface models. The latter ones map the landscape including vehicles and buildings.

That might depend on the size of each rock. SH5's height maps have a resolution of one point every fifty meters so, for little single rocks whose diameter is lesser than a few hundred meters, it might be more convenient resorting to external 3D meshes set as terrain objects. As for sandbanks, shoals, (semi-)submerged reefs, and other seabed natural features, indeed we can realistically model them directly in the terrain, as long as we have real bathymetric charts available.


Ok.
Hence for lighthouses such as Fastnet lighthouse,
it would be the best solution to make a 3D rock model
like the one you already did for Kendras lighthouse in SH3.

[gap]

As you mentioned height points every 50 meters:
On this site I´ve found some 50m Digitalterrainmodels (also available in 25m, 1“ and 3“)

http://data.opendataportal.at/dataset/dtm-norway

I downloaded Denmark and Norway.
Into the folders, I found .prj, .tfw and .tif files.
The .tif files I was able to open with Gimp: They contain black and white pictures without inscriptions.
Unfortunatly no seabed data (mainland is white, sea is black)
Don´t know, if this is of any use for you?

As far as I´ve seen, there is a difference between digital terrain and digital surface models. The latter ones map the landscape including vehicles and buildings.

Thank you Seaowl :salute:

I have had a look into the height maps provided by opendataportal.at, and as far as I could understand its owner has made a great work in collecting and putting together world height data from many sources; I see two problems with them though:


The world of all the SH games is based on the equirectangular map projection (https://en.wikipedia.org/wiki/Equirectangular_projection)

According to the definition given by Wikipedia, this projection "maps meridians to vertical straight lines of constant spacing (for meridional intervals of constant spacing), and circles of latitude to horizontal straight lines of constant spacing (for constant intervals of parallels)". Due to its distorsions, this kind of map projection is not used in marine cartography, but (quoting again Wikipedia) "because of the particularly simple relationship between the position of an image pixel on the map and its corresponding geographic location on Earth", it is quite popular in thematic mapping and commonly used in Gis technology and... submarine simulations lol. For small areas it is possible for us to empirically convert other map projections to an equirectangular projection, as far as the original projection is a cylindrical projection (https://en.wikipedia.org/wiki/Map_projection#Cylindrical). The below charts are an example of that:

http://map.openseamap.org/
http://fishing-app.gpsnauticalcharts.com/i-boating-fishing-web-app/fishing-marine-charts-navigation.html?title=Helgoland+boating+app#12.09/54.1996/7.8663

Unfortunately, the height maps offered by the website you pointed out use another type of projection which maps parallels and meridians as curved lines and which is not easy to overlap to the SH world.


There are several free 3D editing programs capable of converting from 8-bit (greyscale) height maps to mesh coordinates (that we will be able to import in game when Jeff's importer/exporter will be ready) and vice-versa. Moreover greyscale maps can be easily edited in photoshop or in any other raster image editor though, having only 256 color shades for representing heights from pure black to pure white, their resolution is quite poor. Conversely, the height maps available at opendataportal.at adopt the 32-bit floating point deflate GeoTIFF format, capable of storing a much wider height range at excellent resolution, but not supported neither by photoshop nor by 3D editing programs (that's why when you open them you see only a B/W map). In order to extract their height data we would need another exporter, similar to the one Jeff is already working on.



Ok.
Hence for lighthouses such as Fastnet lighthouse,
it would be the best solution to make a 3D rock model
like the one you already did for Kendras lighthouse in SH3.

Exactly! :up:

[Jeff-Groves]

I don't quite get you here Jeff :06:

Say we have height information at a resolution of 1 m.
Say SH5 needs 25m resolution.
Take a 5mx5m (25m) area from the 1m information and just average that.
All you do is add all the values and divide by 25.
You'll get the average height value for that area.

It's like getting your average in bowling. LOL!

[gap]

Say we have height information at a resolution of 1 m.
Say SH5 needs 25m resolution.
Take a 5mx5m (25m) area from the 1m information and just average that.
All you do is add all the values and divide by 25.
You'll get the average height value for that area.

It's like getting your average in bowling. LOL!

Yep, I understand that but so far I thought that SH5 needs 25m resolution and... we have height information at a resolution of 25 m lol :hmm2:

[Jeff-Groves]

Yep, I understand that but so far I thought that SH5 needs 25m resolution and... we have height information at a resolution of 25 m lol :hmm2:
yes. But I see some information set at 2m, 30m, and so on depending where you get the info. Still learning how all this works and studying file formats.
Just as the info we got to do the RAW stuff to this point?
The more info on different formats I learn could lead to a better RAW creator/editer.

[Jeff-Groves]

OBJ2RAW script files sent.
:up:

It could be better but I'd rather work on the stand alone version as not many want to pony up for 010.

[OldCoder]

gap,

I say sorry if my questions might sound a bit silly but can I change coordinates values in web browser-based query with whatever world coordinate I want? Where are those height values obtained from?

Yes you can. The height values are obtained from 250m SRTM data. I found links in the project github issues to download that data, and then it would just be a case of cloning the open-elevation code so that it would run on a local web server (or query direct via python). Then there would be almost zero latency in the query response time. Problem is 250m may or may not be accurate enough for your needs.

[gap]

OBJ2RAW script files sent.
:up:

It could be better but I'd rather work on the stand alone version as not many want to pony up for 010.

Got them and they work, very well done Jeff, you are the man! :yeah: :sunny: :woot:

Now it is part of my job putting your scripts at work :D

yes. But I see some information set at 2m, 30m, and so on depending where you get the info. Still learning how all this works and studying file formats.
Just as the info we got to do the RAW stuff to this point?
The more info on different formats I learn could lead to a better RAW creator/editer.

I perfectly got your point now :up:

Real world height data with a resolution much lesser than 50 m would still be useful for general terrain profile, especially for the seabed whose visual aspect is not so important, and we could add some random noise to that data so to simulate a resolution to the resolution we have in game; anyway such a poor resolution would be of small use for recreating realistic coastal profiles and elevations, and if we used only that data we would end up with a smoothed out version of the world.

For height data with a resolution bigger than 50 m I see no problem: as you said we can average that data. Some manual adjustments might still be needed near the coasts for avoiding an undesirable "zigzags", but that kind of post-processing could be quickly performed in a 3D program (Wings3D 'sculpt' mode totally fits this need) or with the Terrain Editor that devs gave us.

gap,

Yes you can. The height values are obtained from 250m SRTM data. I found links in the project github issues to download that data, and then it would just be a case of cloning the open-elevation code so that it would run on a local web server (or query direct via python). Then there would be almost zero latency in the query response time. Problem is 250m may or may not be accurate enough for your needs.

Okay, I thought that that set of world elevation values had a better resolution; I compared some of the heights it provides with Google Earth's data, and they seem largely comparable if not identical.

For the rest, see my considerations above please: indeed 250m is not the best, but it is still of some use. The problem is that I have no idea on how to create a local query like the one you have described :doh:

[gap]

@ OldCoder, Jeff-Groves and all the concerned ones

I have found another STRM database promising high resolution (30-m) world height data:

https://dwtkns.com/srtm30m/

From what I have read from STRM documentation available elsewhere, the resolution given is relative to the equator. Due to the distorsions of the SH world, the in-game resulting resolution will be slightly lesser than that, especially near the poles, but still bigger than the maximum terrain resolution of SH5 which is equal to 50m.

There are two problems with that set of data:

1 - some areas are not covered, and data provided seem only relative to landmasses (no sea depth data)

2 - height data is provided in SRTMHGT format (https://www.gdal.org/frmt_various.html#SRTMHGT); I don't know how easy would be for us processing that kind of files. Moreover, in order to download the satellite data an username and password are required, but I didn't see any way to register into the site. Greyscale previews of each tile are freely accessible. They cover an area of 1 deg of latitude x 1 deg of longitude and they have a size of 3601x3601 pixels; realtive to our game, that makes an horizontal resolution of 120,000 ÷ 3,600 = 33.33m, more than we need, but being 8-bit images they can store only 256 shades of gray/height levels. Depending on the height range stored in each tile, this might involve a quite poor vertical resolution and an overall lack of accuracy considering that each tile might use its own greyshade-to-height scale, and that those scales are not provided :hmmm:

[OldCoder]

Inspecting the URL for the download link, and trying the web site, it seems that registering an account at https://urs.earthdata.nasa.gov/ may be possible. It looks free to anyone, though I haven't tried it.


EOSDIS data are openly available to all and free of charge except where governed by international agreements.

[gap]

Inspecting the URL for the download link, and trying the web site, it seems that registering an account at https://urs.earthdata.nasa.gov/ may be possible. It looks free to anyone, though I haven't tried it.

Thank you OldCoder, you are right! :up:

I had already registered on https://urs.earthdata.nasa.gov/, and logging in into it allowed me to download the .hgt file relative to the German Bight tile. Now we need to understand how we can convert such a file format into something that can be easily imported in game :hmmm:

P.S: this link might be of some use:
https://github.com/zbycz/srtm-hgt-reader

[OldCoder]

That 30m data and the hgt reader at github was a good find.

I've cooked up a little c++ exe that will read the hgt and dump the contents into output files. One output file is text format, and looks like:

51.999722 1.000000 47
51.999722 1.000833 47
51.999722 1.001666 46

where that's LAT/LON and height in metres.

Second format is a greyscale image in PGM format, documented here: https://en.wikipedia.org/wiki/Netpbm_format.

The image created looks exactly like the preview button image at the earthdata site, so I know it's correct (at least for the single tile I tested).

So, download the height.zip file and put it into an empty folder.
Then download a srtm hgt tile file and drop it into an srtm subfolder.

Then run it like:

height.exe 51 1 3 200


51 1 are the LAT/LON from the tile filename.
3rd param is the stepping value. Specifying 3 results in a 1201x1201 image file. Specifying 1 is the highest resolution of 30m 1 arc-second data, and generates output at 3601x3601.
The 4th param is calibration of black to white, mapped to max height. So if you specify 200, then 0 is black and anything at 200+m will be drawn (and clipped if necessary) in white. You could specify 1000 instead and then a lot of low lying ground will be black or very dark, and only the highest mountains would be white.

The text file output scales similarly in size according to the 3rd param.


Also, you can run the exe with only 2 params just to return height at a specific LAT/LON if you want:

height.exe 51.389057 1.384060 and it will return interpolated height of 9.3m


I've only tested it briefly, so it may be a bit buggy lol.


Unfortunately the forum won't let me upload the exe zip to this message, but when I find out a way to upload it I will do so.

[Jeff-Groves]

Nice link to github.
May compile that in VS 2008 and see what needs done to adapt for RAW.
:hmmm:
Oh OldCoder beat me to it! LOL!
I can send you my e-mail address and you can send it to me.

[Jeff-Groves]

Also, you can run the exe with only 2 params just to return height at a specific LAT/LON if you want:

height.exe 51.389057 1.384060 and it will return interpolated height of 9.3m

This part interests me.
Say we build a matrix to hold just the height data for a given grid that matches what we need in SH5 then save that info as a float for each height. Output a proper format and you have a RAW file.

We can edit the RAW files now in a 3D program and convert back to RAW.
I butchered up some scripts for 010 that do the RAW to obj and back pretty well so far.
A stand alone program is in the works.

[gap]

Thank you OldCoder! :Kaleun_Applaud:

Jeff is right in suggesting to adapt the output of your executable so that the exported terrain height values cover exactly one of the SH5 RAW maps.

Each data tile provided in the 30-m STRM database is 1deg of longitude by 1deg of latitude in size, and it has a resolution of 1 arc-second. Conversely SH5 height maps measure 1/24 deg of lat/long, and they have a resolution of 1.5 arc-seconds.

Knowing that, I suppose that it shouldn't be too complicated modifying your code so that for each export session it will extract from the appropriate hgt file only the elevation values we need for one of the RAW maps, down-sampling them to the desired resolution. Ideally, the exe should accept three user imputs. The first two onputs would be numbers comprised between 0 and 23, indicating the position of the data array we want to extract relative to the big hgt tile, while the third parameter should determine the resolution of the exportd data (1.5 if omitted).

What do you think guys? :D

[OldCoder]

Hi, yes I just sent Jeff a link to the mediafile link for the exe.

If it works for Jeff I can easily upload sources so you guys can adapt as needed (or just do your own thing based on what's in github). The github code works (with a few fixups), and I then modified it to generate the output formats I was interested to see.

[gap]

Hi, yes I just sent Jeff a link to the mediafile link for the exe.

If it works for Jeff I can easily upload sources so you guys can adapt as needed (or just do your own thing based on what's in github). The github code works (with a few fixups), and I then modified it to generate the output formats I was interested to see.

Loking forward to your and Jeff's news :up:

You are amazing guys :sunny:

[gap]

@ OldCoder

I am trying to run your tool as you said:

I created a new folder in C:\, I dropped your executable into it, I created a subfolder of the first folder, I named it 'strm' and I dropped there one of the hgt files, but when I try running height.exe from the command prompt I get a missing VCRUNTIME14D.dll error even though I have the latest Visual C++ 2017 runtimes installed (x86 and x64).

This happens in Windows 8.1. Is there anything I am doing wrong? :hmmm:

[OldCoder]

Try searching google/MS for Microsoft Visual C++ 2015 Redistributable

That should sort it (I think).

[Jeff-Groves]

OldCoder answered quicker then me.
And you need VCRUNTIME140.dll not VCRUNTIME14D.dll
Get the full install as there may be other dll's you need.

https://www.microsoft.com/en-us/download/details.aspx?id=52685

[gap]

Try searching google/MS for Microsoft Visual C++ 2015 Redistributable

That should sort it (I think).

OldCoder answered quicker then me.
And you need VCRUNTIME140.dll not VCRUNTIME14D.dll
Get the full install as there may be other dll's you need.

https://www.microsoft.com/en-us/download/details.aspx?id=52685

The missing dll is actually VCRUNTIME140D.dll, sorry for mispelling its name, and unfortunately installing the Microsoft Visual C++ 2015 Redistributable didn't fix the problem

https://i.imgur.com/4WdL5Sr.png

I am now trying to install the Microsoft Visual C++ 2015 Redistributable updates and to re-installe Microsoft Visual C++ 2017 Redistributables (I had to uninstall the latter in order to install the 2015 version) :hmmm:

[Jeff-Groves]

OldCoder,

Can you send me your source code or could you add the needed code to do a area search function? Say something where we can search all heights for a given square?

I could add the code but it may be awhile. Real life business has me REAL busy just now.

Picture doing bids for 4 jobs all over the U.S.A. and trying to get motels and man power in line!
That runs the range from Down town Cleveland to Dallas, Texas.

[gap]

OldCoder,

Can you send me your source code or could you add the needed code to do a area search function? Say something where we can search all heights for a given square?

That would be awesome... :up:

...if only I could get that code to work lol :-?

So here are all the Visual C++ redistributables I have installed on my system (I suppose that VC++ 2015 red. and VC++ 2015 red. update 3 get overwritten by VC++ 2017 red, because I don't see them in the list even though I had installed them today:

https://i.imgur.com/XwqMBEj.png

...and this one was my last attempt to run the exe: :doh:

https://i.imgur.com/O7cHEnO.png

[gap]

Okay, I have probably found the culprit:

I have VCRUNTIME140.dll in my windows\system32 folder. Nonetheless OldCoder's program throws a missing VCRUNTIME140D.dll error (see that litthe D?).

VCRUNTIME140D.dll is the debug version of VCRUNTIME140.dll, and it is not redistributable, that's why it is missing from my system.


Possible solutions:

@ OldCoder would it be possible for you to compile your executable with release settings?

If not, I will try and install a basic version of Visual Studio, to see if it registers on my computer the DLL(s) required by your exe. :salute:

[OldCoder]

Ok I just sent you both a link to the source code so you can adapt it to requirements, but also inside the zip is a recompiled Release cfg exe which you can try gap.

[gap]

Ok I just sent you both a link to the source code so you can adapt it to requirements, but also inside the zip is a recompiled Release cfg exe which you can try gap.

Thank you very much!

I have received your files. The recompiled exe works now, though, whatever stepping value is used as its argument, the resulting .txt output contains 1,442.401 (i.e. 1,201 x 1,201) height points, which is equal to one third of the original resolution (stepping factor = 3).

For our purposes we need to apply a stepping factor of 1.5. Using that value I would expect an output containing 5,764,801 (i.e. 2,401 x 2,401) height points.

Any idea on why the stepping factor doesn't work and how to fix it? I have the source code that you sent me, but I know nothing about C++, and even if I started learning it today, I am afraid it would take long before I could figure out the problem by myself :hmm2:

[bstanko6]

I placed the mod in the following order:

main
Season Fine tuning
Full resolution tree textures
full resolution detail masks
x2 trees

No lag, but I can see with my bino little black spots in the sky above the trees. probably has to do with that 3d turn in place issue you are talking about. Is that normal?

[gap]

I placed the mod in the following order:

main
Season Fine tuning
Full resolution tree textures
full resolution detail masks
x2 trees

No lag, but I can see with my bino little black spots in the sky above the trees. probably has to do with that 3d turn in place issue you are talking about. Is that normal?

Do you mean tiny lines along the borders of the tree "billboards"?

[bstanko6]

They are little lines floating above the trees so I think yes?

I think we are talking about the same thing.

[gap]

They are little lines floating above the trees so I think yes?

I think we are talking about the same thing.

Okay, I don't think that this is something related with the "3d turn in place issue", but let me guess: do you have a Nvidia gfx card?

In any case: try disabling the 'full resolution tree textures' submod and, if that doesn't solve your problem, try enabling in its place the "low resolution tree textures" one. By using lower resolution tree textures you shouldn't loose too much: for memory saving reasons, SH5's hard-coded tree shading/downsampling/transparency sorting methods are pretty "coarse", and increasing texture size doesn't increase the final tree rendering quality in any significant matter :yep:

[bstanko6]

I will test later. Just for your info the little lines only appear when I use Binos not regular view from bridge

[gap]

I will test later. Just for your info the little lines only appear when I use Binos not regular view from bridge

Okay, you didn't answer my question though: do you have a Nvidia gfx card?

[bstanko6]

I’m sorry I typed my answer but did not save. AMD 580 8gig I. Playing h now and low resolution worked!

[bstanko6]

But I also changed the detail mask to low. I’m going to play around with it

[bstanko6]

It was the full resolution detail mask! Now I run with full tree resolution and low detail mask and it looks absolutely awesome. Thanks

[bstanko6]

In sorry update. So in the bunker no line, but as soon as I exit... line still there. More testing.

The little line is not a problem… It’s when there’s a bunch of trees, all of those trees have lines and it becomes an issue.

I took out all tree and mask resolution and same problem

[bstanko6]

https://i.ibb.co/Htdcmyj/SH5-Img-2019-02-13-12-51-46.png (https://ibb.co/rxsBnhV)

in the picture just above the trees looks like black lines or birds to some, but that appears with this mod.

What ive tested:
-Main mod with fine tuning only
-main with fine tune, high res trees only
-main with fine, high res trees, low res masks
-main with fine, hi res trees, hi res masks
-main, fine, low res tree, hi res mask
-main fine, low res tree, low res mask
-main, fine, hi res mask

always comes up. it must be in main or fine tune. but i want the fine tune, because that makes themost sense. This issue is not that big of a deal since i only spend so much time in port. just wanted to make you aware

AMD - 6300 processor
AMD 580 8gig
16 ram

Im pretty sure my rig is good.

[gap]

Okay, I think I know where those lines are coming from, and I know how to get rid of them. I don't go into details now because explaining the likely cause of the problem is probably more complicated than removing it, but if you just want to know it in short, it is the miplevels.

A fix will be released with the next version of the mod. Thank you for reporting the problem bstanko6, your patient feedback is highly appreciated :salute:

[bstanko6]

No way I thank you! It’s a great mod!! Love it.

Colors are vibrant and the trees from a great distance are so life like!

[gap]

No way I thank you! It’s a great mod!! Love it.

Colors are vibrant and the trees from a great distance are so life like!

In these days I am working on more vegetation types. Stay tuned :salute:

[Jeff-Groves]

Real life back in Cleveland has me busy again.
I do have the source code now (Thanks Mike!) and a 2nd 500 Gig SSD is installed and working. (Finally!)

I have a few more LONG days here but will be looking at the stuff ASAP.

[ReallyDedPoet]

Nicely done here :up:

[gap]

Real life back in Cleveland has me busy again.
I do have the source code now (Thanks Mike!) and a 2nd 500 Gig SSD is installed and working. (Finally!)

I have a few more LONG days here but will be looking at the stuff ASAP.

Back in the apartment with a nice view :03:

Take your time on our project Jeff, real life has the priority. In the meanwhile I am working on some new terrain types, and while we wait for a proper srtm data exporter I might extract the height values I need, from the greyscale previews they also provide at the strm website.

The previews have full planar resolution (33 m), more than we need. Their only disadvantage is that they store height values as gray shades from 0 to 255; for step coastlines that would involve a relatively poor height resolution but, after all, converting SH5 RAW maps into greyscale height maps was our initial plan, so I don't complain too much :)

Nicely done here :up:

Thank you mate, I am glad that you like the stuff we are cooking for you :salute:

[Jeff-Groves]

It's not all bad here in Cleveland. Got this really neat shot today of a Police Officer on a horse.
That's not something I have seen in a very long time! And it's near the Statue of Liberty to!
:up:
http://www.subsim.com/radioroom/picture.php?albumid=1069&pictureid=10188

Something about this picture just means a lot when your away from home in a strange City.

[Ashikaga]

The mod is looking great! Question, could you also add ports in that Mod, for instance in the pacific? Or is the game map hard-coded in that way so nothing can be added?






.

[gap]

It's not all bad here in Cleveland. Got this really neat shot today of a Police Officer on a horse.
That's not something I have seen in a very long time! And it's near the Statue of Liberty to!
:up:
http://www.subsim.com/radioroom/picture.php?albumid=1069&pictureid=10188

Something about this picture just means a lot when your away from home in a strange City.

So are you native from NYC? Your location info here says 'Nevada, Ohio', which is not too far from Cleveland.

Nice picture, by the way. Keep them coming :up:

The mod is looking great!

Thank you mate. Wait to see what is coming next :03:


Question, could you also add ports in that Mod, for instance in the pacific? Or is the game map hard-coded in that way so nothing can be added?

The game map is not hard-coded. From the SH5 release day, devs gave us editors for modifying the terrain profile and for adding any sort of object to the game map, including cities, villages, ports, U-boat pens, etc.
Now, thanks to Pintea, Jeff Groves and OldCoder, we have some new terrain editing tools that will enable the attaining of an higher level of accuracy. Unless we decide to create a fleet boat megamod, I am not sure on the utility of adding to the game Pacific ports, but definitely some new and more realistic ports are in the todo list of this mod :yep:

[gap]

I can see with my bino little black spots in the sky above the trees.

@ bstanko6

Please enable this patch (https://www.mediafire.com/file/jv9vmbp55m9ivp9/EcoMod_patch_0.2.2.rar/file) on top of the latest EcoMod version and tell me if that solves the issue for you. It actually should, but some finetuning might be needed depending on the GFX card and graphic settings (especially screen resolution) used.

Thank you in advance for you help :salute:

[Ashikaga]

The game map is not hard-coded. From the SH5 release day, devs gave us editors for modifying the terrain profile and for adding any sort of object to the game map, including cities, villages, ports, U-boat pens, etc.
Now, thanks to Pintea, Jeff Groves and OldCoder, we have some new terrain editing tools that will enable the attaining of an higher level of accuracy. Unless we decide to create a fleet boat megamod, I am not sure on the utility of adding to the game Pacific ports, but definitely some new and more realistic ports are in the todo list of this mod :yep:

Yeah I asked that so those of use who can edit missions could make Pacific missions like they have in SH4 Wolves of the Pacific. That is something I have always missed in this game. That way we can maybe extend the campaign years till 1945 and even add more difficulty. And if there would be very proficient ship editors here one might even be able to add a ship armed with hedgehogs!



.

[gap]

Yeah I asked that so those of use who can edit missions could make Pacific missions like they have in SH4 Wolves of the Pacific. That is something I have always missed in this game. That way we can maybe extend the campaign years till 1945 and even add more difficulty. And if there would be very proficient ship editors here one might even be able to add a ship armed with hedgehogs!

I understand and I share your same enthusiasm, but...

1 - SH4 has Pacific campaign and missions because, unlike SH5, it revolves around the US submarine campaign of WWII. The Pacific was only a marginal theater of the U-boat campaign.

2 - The SH5 campaign has already been extended to 1945 with Open Horizons II (now part of TWoS), and it already includes an Indian Ocean and Western Pacific campaign (i.e. Monsun Gruppe) which is what you probably have in mind. Indeed, this mod will cover the areas of U-boat operation in the Far East, as well as the East coasts of USA and Canada, the Mediterranean, the Black Sea, the Caribbean, the South Atlantic and other minor war theaters (from the Ubootwaffe perspective), but the North and Western coasts of Europe for now have the highest priority. :)

3 - There already are hedgehogs in game. Maybe you meant squid launchers (http://www.subsim.com/radioroom/showpost.php?p=2399929&postcount=80). :D

[propbeanie]

https://www.youtube.com/watch?v=Fk3o8VAp3os

:salute:

[bstanko6]

Ok i just tried it and here are my observations:

Definitely an improvement.

Weird though. Trees up close or closest to the Uboat, the line is gone, but Trees further away in a group still have the line, but the line blinks really fast where it has the illusion it is toned down, not bad.

But the non improvment is i only saw the line when using my bino, now the line is there in regular view, but for far away trees in a group only if that makes sense. again not a terrible issue.

trees up close in regular or bino view are fine. no issue.

[bstanko6]

im using 1920 x 1080

AMD RX 580 8 gig

[gap]

Okay, on my rig, the patch I have uploaded this afternoon fixed for good all the tiny lines; I play the game at the same resolution as you, but as I said video card and graphic settings might also play a role.

Reproducing your exact conditions would be impossible for me, so may I ask you a favour?

This is the code of the shader (ForestVS.fx) I have edited for patch 0.2.2:

float4x4 g_matWVP : register(c0);
float3 SunLightDir : register(c4);
float3 g_eyePos : register(c5);

#ifdef OUTPUTDEPTH
float3 g_world : register(c6);
#endif

struct VS_IN
{
float4 pos : POSITION;
float3 texCoord : TEXCOORD0;
};

struct VS_OUT
{
float4 projPos : POSITION;
float2 texCoord : TEXCOORD0;
float4 pos : TEXCOORD1;
};

const static float3 g_texCoords[36] =
{
{0, 0, 0},
{-0.0002, 0, 1},

{0.125, 0, 0},
{0.1248, 0, 1},

{0.25, 0, 0},
{0.2498, 0, 1},

{0.375, 0, 0},
{0.3748, 0, 1},

{0.5, 0, 0},
{0.4998, 0, 1},

{0.625, 0, 0},
{0.6248, 0, 1},

{0.75, 0, 0},
{0.7498, 0, 1},

{0.875, 0, 0},
{0.8748, 0, 1},

{1, 0, 0},
{0.9998, 0, 1},

///////////////////////////

{0, 1, 0},
{-0.0002, 1, 1},

{0.125, 1, 0},
{0.1248, 1, 1},

{0.25, 1, 0},
{0.2498, 1, 1},

{0.375, 1, 0},
{0.3748, 1, 1},

{0.5, 1, 0},
{0.4998, 1, 1},

{0.625, 1, 0},
{0.6248, 1, 1},

{0.75, 1, 0},
{0.7498, 1, 1},

{0.875, 1, 0},
{0.8748, 1, 1},

{1, 1, 0},
{0.9998, 1, 1},
};

static const float decompressVal = (32767.0 / 2048.0);

VS_OUT main ( VS_IN In )
{
VS_OUT Out;

const float3 up = float3(0, 1, 0);
#ifdef OUTPUTDEPTH
float3 viewDir = normalize(SunLightDir - In.pos);// depth from the light's perspective - works only

for shadow maps
#else
float3 viewDir = (In.pos - g_eyePos);
#endif
float3 right = normalize(float3(-viewDir.z, 0, viewDir.x));//normalize(cross(viewDir, up));//
//float3 up = normalize(cross(right, viewDir));// this can be skipped if a top

perspective is not allowed

// decompress verts
In.texCoord.x *= 32767;
In.texCoord.yz *= decompressVal;

float3 texCoord = g_texCoords[(int)In.texCoord.x];

// move position based on texture coordinates
float3 outPos = In.pos;

#ifdef OUTPUTDEPTH
outPos += g_world;
#endif

outPos += right * ( (texCoord.y - 0.5f) * In.texCoord.y );
outPos += up * ( texCoord.z * In.texCoord.z );

Out.projPos = mul(float4(outPos, 1), g_matWVP);
Out.texCoord = texCoord.xy;
Out.pos.xyz = In.pos.xyz;
Out.pos.w = Out.projPos.z;

return Out;
};


You should open the same shader on your computer (you can use notepad for that) and replace the decimal numbers I have marked above in red with some smaller numbers.

Maybe you could start by using the same values on top of each red number (0, 0.125, 0.250, etc.) minus 0.001.

- If that is not enough, use even smaller numbers;

- if it works, use bigger values but still smaller that the ones I used for my patch.

- keep increasing/decreasing those numbers until you find the values closest to my settings that fix the lines issue for you.

I might have explained myself poorly, so don't hesitate asking in case you need further explanations :salute:

[gap]

https://www.youtube.com/watch?v=Fk3o8VAp3os

:salute:

That's it! :D

[bstanko6]

I’ll play with it thanks.

[bstanko6]

Are you saying you don’t see lines on your pc?

If not, what on my graphics card would cause that? I would have no clue?

[gap]

I’ll play with it thanks.

Thank you too!

Are you saying you don’t see lines on your pc?

After enabling my last patch I don't. Neither looking through bino's nor in free camera view, no matter distance/magnification levels.


If not, what on my graphics card would cause that? I would have no clue?

Well, that might require a long explanation.

For each terrain type, eight trees are stacked one on top of the other in the same texture. Where one tree ends, the next tree starts. In theory the demarcation between a tree and the next one should be sharp, but in practice some blending happens in game. The lines you see on top of the trees, are actually a tiny portion of stem of other trees.

On my computer, this blending mostly happened when I looked at trees from very close range (that's why I thought that the glitch could be ignored); what I did with my patch was "clipping" the very top of each tree, thus removing the lines blended on it.
Nnetheless, I think that in your case the problem comes from the mipmaps (https://en.wikipedia.org/wiki/Mipmap); that might explain the fact that you only see those tiny lines when you look at trees from a certain distance. Mipmaps are calculated automatically and, depending on the resampling method used, some shading happens. That worsens the in-game blending, making the lines you see at long range, thicker than they were on my computer. The shading is so faint, that the shaded lines are considered transparent by my video card, but not by your one. The difference might be caused by a bunch of factors, but I think making a comparison between different gfx cards would be a waste of time.
What matters here is finding a compromise that will work with all the cards.

One solution could be re-saving my textures, but using a sharper re-sampling method, or editing the contrast of their alpha (opacity) channels so that a bigger portion of them will result black (=transparent). This approach would have two con's though: each terrain type features four tree textures (one per season); multiply this number times three (one main resolution plus two optional resolutions), times six (number of vegetation/terrain types I have modelled so far), that makes 72 textures that I would need to edit. Moreover, a sharper opacity mask might mean a poorer texture quality, especially for the smaller mipmaps.

The other fix is "cutting" a bigger portion of texture from the top of each tree. Which is exactly what I asked you to test. I am sorry, but since I don't have your same glitch, I need your help for fixing it :)

[bstanko6]

@ Gap I did it my friend no problem now. Here are the numbers:

const static float3 g_texCoords[36] =
{
{0, 0, 0},
{-0.0001, 0, 1},

{0.125, 0, 0},
{0.1230, 0, 1},

{0.25, 0, 0},
{0.2490, 0, 1},

{0.375, 0, 0},
{0.3730, 0, 1},

{0.5, 0, 0},
{0.4980, 0, 1},

{0.625, 0, 0},
{0.6230, 0, 1},

{0.75, 0, 0},
{0.7480, 0, 1},

{0.875, 0, 0},
{0.8730, 0, 1},

{1, 0, 0},
{0.9980, 0, 1},

///////////////////////////

{0, 1, 0},
{-0.0001, 1, 1},

{0.125, 1, 0},
{0.1240, 1, 1},

{0.25, 1, 0},
{0.2480, 1, 1},

{0.375, 1, 0},
{0.3730, 1, 1},

{0.5, 1, 0},
{0.4980, 1, 1},

{0.625, 1, 0},
{0.6230, 1, 1},

{0.75, 1, 0},
{0.7480, 1, 1},

{0.875, 1, 0},
{0.8730, 1, 1},

{1, 1, 0},
{0.9980, 1, 1},

Perfect for us AMD users! I never played with shaders before. Thanks!

[gap]

@ Gap I did it my friend no problem now. Here are the numbers:

const static float3 g_texCoords[36] =
{
{0, 0, 0},
{-0.0001, 0, 1},

{0.125, 0, 0},
{0.1230, 0, 1},

{0.25, 0, 0},
{0.2490, 0, 1},

{0.375, 0, 0},
{0.3730, 0, 1},

{0.5, 0, 0},
{0.4980, 0, 1},

{0.625, 0, 0},
{0.6230, 0, 1},

{0.75, 0, 0},
{0.7480, 0, 1},

{0.875, 0, 0},
{0.8730, 0, 1},

{1, 0, 0},
{0.9980, 0, 1},

///////////////////////////

{0, 1, 0},
{-0.0001, 1, 1},

{0.125, 1, 0},
{0.1240, 1, 1},

{0.25, 1, 0},
{0.2480, 1, 1},

{0.375, 1, 0},
{0.3730, 1, 1},

{0.5, 1, 0},
{0.4980, 1, 1},

{0.625, 1, 0},
{0.6230, 1, 1},

{0.75, 1, 0},
{0.7480, 1, 1},

{0.875, 1, 0},
{0.8730, 1, 1},

{1, 1, 0},
{0.9980, 1, 1},

Perfect for us AMD users! I never played with shaders before. Thanks!

Thank you very much bstanko6, well done! :up:

I have noticed that you didn't subtract a constant vale from stock vertex coordinates (e.g. 0.125 - 0.123 = 0.002, but 0.25 - 0.249 = 0.001). Is there a reason for that? At the first attempt, did you try doing as I had said yesterday? :)


Maybe you could start by using the same values on top of each red number (0, 0.125, 0.250, etc.) minus 0.001.

[bstanko6]

Exactly! I’m not too good at playing with programs and stuff so I stay simple and do little changes at a time. So went the red numbers you suggested and changed them ever so slightly but consistently. I hope you can publish this as a fix for those that have similar issues like I did. Now it’s absolutey perfect! I tried several times in several testing missions!

[gap]

Exactly! I’m not too good at playing with programs and stuff so I stay simple and do little changes at a time. So went the red numbers you suggested and changed them ever so slightly but consistently. I hope you can publish this as a fix for those that have similar issues like I did. Now it’s absolutey perfect! I tried several times in several testing missions!

Adding the fix to the next EcoMod version is exactly my intention, and you deserved a special mention in its release noted, but before we release the fix, could you please test the numbers below? :D

{0, 0, 0},
{-0.001, 0, 1},

{0.125, 0, 0},
{0.124, 0, 1},

{0.25, 0, 0},
{0.249, 0, 1},

{0.375, 0, 0},
{0.374, 0, 1},

{0.5, 0, 0},
{0.499, 0, 1},

{0.625, 0, 0},
{0.624, 0, 1},

{0.75, 0, 0},
{0.749, 0, 1},

{0.875, 0, 0},
{0.874, 0, 1},

{1, 0, 0},
{0.999, 0, 1},

///////////////////////////

{0, 1, 0},
{-0.001, 1, 1},

{0.125, 1, 0},
{0.124, 1, 1},

{0.25, 1, 0},
{0.249, 1, 1},

{0.375, 1, 0},
{0.374, 1, 1},

{0.5, 1, 0},
{0.499, 1, 1},

{0.625, 1, 0},
{0.624, 1, 1},

{0.75, 1, 0},
{0.749, 1, 1},

{0.875, 1, 0},
{0.874, 1, 1},

{1, 1, 0},
{0.999, 1, 1},

[bstanko6]

Sure thing. Give me an hour

[bstanko6]

testing completed with your numbers and it is exactly the same, or at least no change.

Tested all Gap testing missions with all seasons - successful.

Changed weather, for more ship movement, and sighted no black lines - successful.

Looks great, and I will just keep your new numbers for the Eco patch. Good Job friend.

[Sjizzle]

gap my good old m8 u are still doing really amazing jobs with graphics and many more... keep up the good works cheers m8 :Kaleun_Cheers:

[Jeff-Groves]

Nice work Guys!
:salute:

[bstanko6]

Thank Gap! He has nice work. I was privileged to help test. It’s a fantastic mod. I have never been one about harbors or the way they looked before, but leaving the harbor and seeing these real trees looks really incredible! And the proper seasonal colors makes it even better!

[gap]

testing completed with your numbers and it is exactly the same, or at least no change.

Tested all Gap testing missions with all seasons - successful.

Changed weather, for more ship movement, and sighted no black lines - successful.

Looks great, and I will just keep your new numbers for the Eco patch. Good Job friend.

Thank Gap! He has nice work. I was privileged to help test. It’s a fantastic mod. I have never been one about harbors or the way they looked before, but leaving the harbor and seeing these real trees looks really incredible! And the proper seasonal colors makes it even better!

Good job bstanko6, the privilege is mine :yep:

gap my good old m8 u are still doing really amazing jobs with graphics and many more... keep up the good works cheers m8 :Kaleun_Cheers:

Sjizzle matey, thank you for the encouragement, I am glad to see you around. Let's keep in touch :sunny:

Nice work Guys!
:salute:

Thank you Jeff. This afternoon I played a bit with the height maps available (as data previews) on the 30-Meter SRTM Tile Downloader website (https://dwtkns.com/srtm30m/). As I said in one of my previous post, each preview has a size of 3601 x 3601 pixels, thus offering the full data resolution of one arcsecond stored in the hgt files.

The image below, is the (scaled down) combination of two tiles relative to the western coast of Jutland:

https://i.imgur.com/7ii48Qp.png

Each tile preview used its own grey/height scale, so I had to match them by modifying their tone curves, making their lowest point (= the sea level) black, and reducing the contrast of the tile having the narrower height range. Performing the latter task based on numbers was easy, as OldCoder's exe reports for us the highest point of each tile. Note that the same can be done directly with just one command, using OldCoder excutable, though, due to its high "stepping factor" data exported from it has a too low planar resolution (exactly one third of the original resolution).

The next step, was creating an overlay which will help me locating the small SH5 tiles on each big STRM tile:

https://i.imgur.com/aU1F5LA.png

With the help of that overlay, selecting only the portion of map we desire, copying it into a new file and scaling it to the desired size of 101 x 101 pixels is quite easy. Then we can save it to the PGM format (https://en.wikipedia.org/wiki/Netpbm_format#PGM_example) pointed out by OldCoder (for that task, I have found a good Portable PixMap Importer Exporter (http://richardrosenman.com/shop/portable-pixmap-importer-exporter/)).

From there, converting 0-255 values in actual heights (using the conversion factor we had calculated before) and importing them into the obj file like the one exported/imported by you OBJ-RAW converter, it's a short road. There is a problem though.

At first, I had underestimated the effects of poor vertical resolution. You will understand better from the following pictures, where I have marked the zeroes (= black = sea level) in blue:

https://i.imgur.com/rK2Hf2n.png

If you compare it with a satellite view of the same area, you will easily spot the problem:

https://i.imgur.com/nx83Ps4.png

The height approximation caused by having only 256 height levels, would cause beaches and lowlands to literally disappear below the sea. Useful for previewing the future disasters of the greenhouse effect, but not for our game. :hmmm:

Of course I could adjust those areas manually, but still, having a proper hgt exporter would greatly ease / speed-up our work on the game map :yep:

[Seaowl]

Altought there are troubles

the project looks great so far !


:Kaleun_Thumbs_Up:
Due to lack of insight, I can´t be much of a help.

But I found this blog: https://www.sigterritoires.fr/index.php/en/make-a-map-topographic-with-qgis/

There SRTM-data are used together with QGIS:
https://www.qgis.org/en/site/

Perhaps there is some useful information....

[gap]

Altought there are troubles

the project looks great so far !


:Kaleun_Thumbs_Up:
Due to lack of insight, I can´t be much of a help.

But I found this blog: https://www.sigterritoires.fr/index.php/en/make-a-map-topographic-with-qgis/

There SRTM-data are used together with QGIS:
https://www.qgis.org/en/site/

Perhaps there is some useful information....

Good point! I will install QGIS and see if it can export height data in a format that can be easily imported in an obj file :up:

[gap]

Interesting web page:

https://gisgeography.com/free-global-dem-data-sources/

According to it, the most accurate world height data is offered by the Japanese Space Agency with the denomination of ALOS Global Digital Surface Model "ALOS World 3D - 30m (AW3D30)"

This is the main website of the project:

https://www.eorc.jaxa.jp/ALOS/en/aw3d30/index.htm

According to it, the data they provide is in GeoTIFF (https://earthdata.nasa.gov/user-resources/standards-and-references/geotiff) format, but a link to a script based on GDAL (https://www.gdal.org/index.html) technology is also provided for conversion from GeoTIFF to the SRTM HGT format that by now we are more accustomed to. :03:

Last note: for downloading their data registration is required. I applied for it, but I still haven't received any activation e-mail...

EDIT: I have just found another website that offers 1", 3" and 15" word height data, with a coverage that includes zones not covered by other data sets:

http://viewfinderpanoramas.org/dem3.html#images

Download from it doesn't require any registration, and data offere is in the usual HGT format :)

[Jeff-Groves]

Grey Scale is never going to work in SH5 for detail such as Helgoland.
We already know that so why waste time with it?
The 256 shades stuff is for SH3/4 not SH5.
Only the main terrain files use that old system and we know the detailmasks use a different system.
Now why do the old files still use the old system? Because they didn't want to waste time creating a whole new system for a Game
built on something from 2004 or so! They just improved some areas and left the old stuff in place because it works.
You don't replace something that works. You just improve it in areas that make sense and saves development time and money.
I find the 'fix' brilliant!

[gap]

Grey Scale is never going to work in SH5 for detail such as Helgoland.
We already know that so why waste time with it?

The answer it simple: because we haven't found a way to export 1" resolution height data contained in those hgt files into a numerical format, and I am impatient to feed real world data into your RAW to OBJ and OBJ to RAR importer/exporter :D


The 256 shades stuff is for SH3/4 not SH5.
Only the main terrain files use that old system and we know the detailmasks use a different system.
Now why do the old files still use the old system? Because they didn't want to waste time creating a whole new system for a Game
built on something from 2004 or so! They just improved some areas and left the old stuff in place because it works.

Sounds reasonable :up:

P.S: I have seen you had tried to send me a PM, but my inbox was full. I have made some space, just in case :salute:

[Jeff-Groves]

Our Friend did a fantastic job re-coding the game to give us detailmasks in the way he did.
We are finding that many things are highly mod able. Even more so then with the tools they gave us.
We are finding ways to do better and faster work and never use the crazy tools built into the software.
Which I find right next to useless for sane people and would bet money the Devs never really used them to build SH5!
I'm not slamming the Devs. They only did what they were told they were allowed to do!
Much like my job? I can not do just any thing I want. I have to follow rules and budgets also.

[gap]

Our Friend did a fantastic job re-coding the game to give us detailmasks in the way he did.
We are finding that many things are highly mod able. Even more so then with the tools they gave us.
We are finding ways to do better and faster work...

I agree, the idea of patching the old SH terrain with higher resolution detail mask only where it was actually needed, was a great idea, and so far we have done an outstanding job. We only miss a little puzzle piece before we can start showing our subsim buddies the fruits of our struggles.


...and never use the crazy tools built into the software.
Which I find right next to useless for sane people...

Well, not totally useless. I think it is a great tool for smoothing out things, adding some embellishments and making the last touch-ups, but you can't rely on it for exploiting the full potential of the game.


...and would bet money the Devs never really used them to build SH5!

That I can tell for sure: near all the stock ports and wherever there are detail masks added, there are slight mismatches between a tile and the next one. This is something that could never happen if devs had used only the Terrain Editor :yep:


I'm not slamming the Devs. They only did what they were told they were allowed to do!
Much like my job? I can not do just any thing I want. I have to follow rules and budgets also.

Neither do I. If there is someone to be complained, that is Ubisoft.

[Ashikaga]

You guys are doing an awesome job! Say, how does the landscape buildup/world building of Sh5 compare with for instance IL2 Cliffs of Dover?


.

[Sjizzle]

gap m8 if u need a tester for your work just let me know...
btw i just installed this mod in my mod list damn look awesome :D....

[Jeff-Groves]

Thank you very much!

I have received your files. The recompiled exe works now, though, whatever stepping value is used as its argument, the resulting .txt output contains 1,442.401 (i.e. 1,201 x 1,201) height points, which is equal to one third of the original resolution (stepping factor = 3).

For our purposes we need to apply a stepping factor of 1.5. Using that value I would expect an output containing 5,764,801 (i.e. 2,401 x 2,401) height points.

Any idea on why the stepping factor doesn't work and how to fix it? I have the source code that you sent me, but I know nothing about C++, and even if I started learning it today, I am afraid it would take long before I could figure out the problem by myself :hmm2:
Not following you here.
Height points in a RAW file is 10201
It's 101 floats x 101 floats

From looking at the source code I see this.......

const int secondsPerPx = 3; //arc seconds per pixel (3 equals cca 90m)

So changing = 3 to = 1 would give us 30m (Also saw that commented out so maybe we can't do that?)

Resolution for SH5 is 25m per height value is it not?
:hmmm:

[gap]

You guys are doing an awesome job! Say, how does the landscape buildup/world building of Sh5 compare with for instance IL2 Cliffs of Dover?

I have never messed with any of the games of the IL-2 Sturmovik franchise, so I can't say for sure, but I am afraid that comparing a submarine simulator with a flight simulator wouldn't lead to any conclusion: in one case we have a world made for being observed from the sea level, while in the latter case the privileged view is obviously from top...

gap m8 if u need a tester for your work just let me know...
btw i just installed this mod in my mod list damn look awesome :D....

ROGER that. Thank you mate, the next release shouldn't take too long.
I will let you know if I need you to do some testing :up:

Not following you here.
Height points in a RAW file is 10201
It's 101 floats x 101 floats

true


From looking at the source code I see this.......

const int secondsPerPx = 3; //arc seconds per pixel (3 equals cca 90m)

So changing = 3 to = 1 would give us 30m (Also saw that commented out so maybe we can't do that)

Resolution for SH5 is 25m per height value is it not?
:hmmm:

Forget about meters. Due to distance stretching caused by cylindrical map projections, and to the arbitrary scale used in SH5, we better use angles than meters as unit of measure.

The 30-Meter SRTM dataset that we want to use in game, has a resolution of 1" (= 1 deg / 3600) whereas SH5 RAW files have a resolution of 1.5" (= [1 deg / 24] / 100).

Theoretically, using github/OldCoder's exe for downsampling SRTM data to the SH5 resolution, we should apply a stepping factor of 1.5.
If a similar factor was applied correctly, for a tile of 1 deg x 1 deg, I would expect an output containing 5,764,801 height points (= 2401 x 2401) rather than the full resolution 12,967,201 (= 3601 x 3601) height values.

Yet, no matter what stepping factor I apply (tried with 1, 1.5, 2 and 3), I always obtain an output with 1,442,401 height values (= 1,201 x 1,201) :yep:

I hope I made myself clearer now :)

[Jeff-Groves]

Could be this line in the source.............
const int totalPx = 1201;

I'm reading the code as I do back ground checks on new applicants for jobs.

[gap]

Could be this line in the source.............
const int totalPx = 1201;

I'm reading the code as I do back ground checks on new applicants for jobs.

Maybe try using the data extractor yourself first... maybe I am doing something wrong though I doubt that :hmmm:

[Jeff-Groves]

Send the files your working with to my e-mail and I'll work with them.

[gap]

Send the files your working with to my e-mail and I'll work with them.

Okay, thanks. Check your email :salute:

[Jeff-Groves]

Had to go back to Cleveland for a few days but I have the files.
Now I just got word I have to go to California for a few days.
:o

[gap]

Had to go back to Cleveland for a few days but I have the files.
Now I just got word I have to go to California for a few days.
:o

California is not exactly around the corner :doh:
how is your jet lag :D

In the last few days I have been experimenting with Ref's Terrain Extractor-Importer, and I can confirm that it works for SH5 as well as for SHIV, though I think there is a little glitch with the inbuilt ZHF to RAW conversion utility.

Low-detail height maps extracted from TerrainData.BFD are greyscale images with a size of 601x601 pixels and covering an area of 1 deg x 1 deg. That means that they have a resolution of six seconds: four times lesser that the high-detail height masks generated by Terrain Editor.

Being able to edit the low-detail game terrain tiles is very important, especially for the sea bottom and for areas that we don't visit that often and/or we only see from far away; using 1-sec elevation data for the whole SH5 world would be a waste of time and resources.

The GEBCO website has bathymetric data that can be freely downloaded:

https://www.gebco.net/data_and_products/gridded_bathymetry_data/

Its maps have a resolution of 30 arc-seconds; this is five times lesser than we need, but while better resolution public domain maps would be welcomed (suggestions anyone?), the lack of resolution is not a big problem either, as underwater everything looks blurred and smoothed out anyway. Data offered by GEBCO is in many formats. The ones I have found more useful are GeoTiff (which can be opened and converted in a regular raster format using softwares as QGIS, thank you for pointing me into this program Seaowl!), and asc, which is a plain ASCII format with heigh values sorted in an array.

What I did, was creating an Excel spreadsheet which converts from metric height values contained in the asc files, to 8-bit grey shades calculated according to SH5's height table (http://www.subsim.com/radioroom/showpost.php?p=2587605&postcount=87); the calculated grey values are then sorted by the same spreadsheet in the order required by the portable bitmap format (https://en.wikipedia.org/wiki/Netpbm_format) (eternal gratitude to OldCoder for making me aware of it).

The next steps are importing the output maps in photoshop, re-scaling them to the desired size (some post-processing is needed for hiding the pixelization caused by magnification), replacing unacceptably blurred landmasses (where they apply), with the greyscale version of SRTM 30m (https://dwtkns.com/srtm30m/) data, (similarly edited to match the in-game height table) and, finally, replacing pixels along the borders of each map so that they match the pixels along the adjoining borders of the next maps (for avoiding abrupt height changes: as usual there is some overlapping between nearby tiles).

When I finish with those tasks I export to RAW/ZHF format, and the maps are ready to be used in game. The result I get is visually convincing: at least for the tiles I have edited so far, the edited seafloor has a lot more depth changes than the stock one, with trenches, sandbanks and shoals which obviously resemble the real bathymetric profile of the same areas. There is something I have found out though: in spite of the general similarity, and of all the care I have placed into matching they grey values of those maps with real heights, there will always be some important discrepancies, caused by the fact that there is not a simple relation between grey values and height/depth at any given point: due to the blending of an hardcoded noise mask, even if a set an area totally flat, there will always be a slope of some sort, and the height of that area might vary considerably relative to the height expected according to table I linked above.

If/when we need that level of accuracy, we will have to edit the 1.5 arsecond raw maps using Jeff's importer/exporter and/or terrain editor :yep:

https://i.imgur.com/z773DEZ.png

The Channel Islands seen in Mission Editor. Left: as they look in stock game; right after the edits I described above :sunny:

[Jeff-Groves]

In the last few days I have been experimenting with Ref's Terrain Extractor-Importer, and I can confirm that it works for SH5 as well as for SHIV, though I think there is a little glitch with the inbuilt ZHF to RAW conversion utility.



I don't think there is a glitch.
The compressed tiles are 601x601 and have been since SH3.
That's one of the main reasons the Kiel canal is like it is in SH3/4.
Limited detail terrain work can be done in SH3/4 because of the restriction.

SH5 we can go to greater details thanks to the Detailed RAW files that use actual height values over a smaller area rather then a crude 0-255 scale over a much larger area.

Given the size exported by Ref's program and based on info we have on the details RAW files? The Ref Exported files are 6 time larger then a true SH5 Detail RAW. (Not a fault of his program. That's how it is stored in the files)

Given We know the difference? I'm thinking We could cut the exported files down to the needed detail RAW size with one problem!
It will still be just a 0-255 limited output.

On California? I have a few friends very near there that I'm hoping to spend a weekend with to shoot some nice Air Guns.

[gap]

I don't think there is a glitch.

Maybe not exactly a glitch. I din't check if the uncompressed RAW files work as well as ZHF files and RAW files saved by photoshop, probably yes, but if you want to understand what I mean try doing the following: extract one tile from TerrainData.BFD and open the resulting RAW file in photoshop. The RAW importing menu will select for you the best import options and once opened everything will look in order.
Now try converting the same RAW file to the ZHF format, and then uncompress it again, or if you prefer uncompress one of the stock ZHF files. On opening the resulting RAW file, you are no longer suggested the optimal dimensions. If you try selecting anything bigger than 601x601 obviously you will be told that the image you are trying to open is bigger than the file and you are not allowed to go on with the import; if you select 601x601 you are told that the image is actually smaller than the file, and once opened, five or six columns of pixels in the map are shifted from left to right or vice versa, I don't remember exactly now.


The compressed tiles are 601x601 and have been since SH3.
That's one of the main reasons the Kiel canal is like it is in SH3/4.
Limited detail terrain work can be done in SH3/4 because of the restriction.

Well, SHIII terrain limitations are actually bigger than in SHIV/5: height maps are five times smaller in SHIII than they are in SHIV, i.e. 126 x 126 pixels, with an overlapping between neighbouring tiles of six pixels rather than one pixel. That means one height point every 30 seconds / kilometer as opposed to the resolution of one point every 6 seconds / 200 m of SHIV and the maximum resolution of one point every 1.5 seconss / 50 m attainable in SH5. Nonetheless, due to the extremely poor height resolution of SHIII, the devs used a trick: they introduced 1-bit masks similar in size to the SHIV height maps (602x602 pixels, with an overlapping of two pixels) that are used for determining the shape of islands, coast lines or canals like the Kiel canal :know:


SH5 we can go to greater details thanks to the Detailed RAW files that use actual height values over a smaller area rather then a crude 0-255 scale over a much larger area.

Given the size exported by Ref's program and based on info we have on the details RAW files? The Ref Exported files are 6 time larger then a true SH5 Detail RAW. (Not a fault of his program. That's how it is stored in the files)

Given We know the difference? I'm thinking We could cut the exported files down to the needed detail RAW size with one problem!
It will still be just a 0-255 limited output.

If they will occupy a smaller space on file and the game will read them more readily, why not? Sure, they would still have an 8-bit output, but I suppose that that's a limit of the game code and, as we said, for very accurate terrain changes (where actually needed) we can always use the 1.5 sec RAW maps :yep:


On California? I have a few friends very near there that I'm hoping to spend a weekend with to shoot some nice Air Guns.

Enjoy :up:

[Jeff-Groves]

If you need to work with a file saved to ZHF with Ref's program?
Extract it to RAW again and then delete the first 4 bytes with 010.
:03:
Not sure why that is added but I'll look closer at those 4 bytes.

I can do a script for 010 to remove them if you'd like.

[propbeanie]

Using the GWX Terrain Extractor, SH4 tiles are 601x601 when opened in PS for the first time, and the tiles do overlap by 6 pixels also. If you do NOT get the transitions between tiles edited exactly correct, you often end up with pillars of rock six pixels wide and hundreds of ?? pixels high at the corner junctions. You also have to do all edits in one session. You cannot go back in and re-edit any mistakes you make, else you might end up with 6 pixel wide walls or trenches at the borders, depending upon your edits. Therefore, it's best to print out the original tile on paper, and noodle with that and a pencil first, then use the software. There is an issue with either the PS edits, the Save back to RAW, or the conversion from RAW to the game's ZHF, and you end up with the blank dimensions in the Open dialog if trying to edit a 2nd time. If you just do one edit on the RAW file, then it will display in the game correctly after a single edit session with PS, else you get that shift of six pixels, and other areas near the 6 pixle "border" area that have changed their values. Anymore, if editing terrain for SH4, I export to RAW each time I do an edit, instead of trying to re-edit. Very frustrating... :salute:

Edit: oooohhh, as I was typing Jeff posts about those first four bytes... I will be doing some experiments with that in a few days! I could try that 010 script for you Jeff!... thank you! :salute

[gap]

If you need to work with a file saved to ZHF with Ref's program?
Extract it to RAW again and then delete the first 4 bytes with 010.
:03:
Not sure why that is added but I'll look closer at those 4 bytes.

I can do a script for 010 to remove them if you'd like.

Okay, now it makes more sense though, like you, I am curious to know the function of the extra bytes added by Ref.

Deleting them manually is something I should be able to do myself but, while not impelling, a script automating the process would be nice indeed :up:

[Jeff-Groves]

Those first 4 bytes are in the ZHF files before the first extraction with Ref's tool. On the first extraction they are not there. If you save to ZHF then extract again? They are there.
I'd think a slight mistake on Ref's part of course. Now the question is does that mistake show in Game?
I suspect it's a magic header that is ignored or tells the engine to unzip the file.
Since we never use the unzipped version? We can just remove it for editing reasons.

I'll do some more research on the re-compression Ref's tool does and see if that is added to the files.
If it is? We many need to do a whole new tool to fix that issue.
ZHF is just a part of a compression library so should be easy enough to do.

What I will do is use a ZHF in stock SH4. Extract it then save again to ZHF and compare.

[propbeanie]

I wonder if the first four bytes are a "This file has been tampered with!" for the game?

[Jeff-Groves]

I wonder if the first four bytes are a "This file has been tampered with!" for the game?
No. That is something Ref never did. Aside from the Jammu he did to replace Kashmir in headers of files? There was only one other branding he did at my request. That is in an area of dat files that have no impact on the Game that anyone has ever proven.
It is also coded so only the program that places the information can read it.
That was done in response to others claiming ownership of some type for files.
We quickly caught them using GWX files as the base. Thus ended that.
Sneaky bastards aren't we?
:haha:

[gap]

@ propbeanie

Thank you for the suggestions mate. Yes, definitely there is an overlapping that needs to be observed between one tile and the next ones. Reason for that is that each pixel on those maps doesn't represent a quadrant in a grid, but a verted in a continuos mesh, and at least one row/column of vertices must be in common for two neighbouring terrain tiles for them to be treated as a continuous surface. If their heigh/grey values are not identical, an abrupt height change or a step is created, that from the game perspective are like a trench ir a wall.

That said, I think that SH5 tiles have an overlapping of just one pixel. Why I think that?

1 - by comparing stock tiles with real world terrain height maps, if a 6-pixel overlapping was to be applied, land masses on those tiles should look a bit smaller than they do, whereas they actually show the same land proportions as seen in the big (3601x3601 pixels) srtm maps (https://dwtkns.com/srtm30m/)

2 - as I said before, 601 pixels - 1 pixel of overlapping makes a resolution of exactly one height point every six arcseconds (= 200 m in SH5). A bigger overlapping would involve a decimal resolution which seems not likely to me.

3 - Before starting with my editing I had exporter two adjoining maps and overlapped them using the "divide" blending mode. The division of two identical pixels is equal to pure white (any number divided by itself makes 1 which, in image processing, is equal to white). The only way I could get all white pixels was by setting an overlap of only one pixel :yep:

I will do more test, but I think that my working method is correct for SH5. Yes, editing those maps requires a general plan. The error is around the corner, but as far as you keep track of all the changes made and you save every change in separate layers (that's easy in photoshop), keeping backup copies of the edited files, You don't necessarily need to edit everything at once. The four extra bits added by Ref's exporter are a bit annoying, but if you have saved copies of your working files in psd format, that's not a big problem either :)

[Jeff-Groves]

I'd say compression format is not the same when compressed with Ref's Tool. SH will read it however so that is not a bad thing.
Compression is based on zlib. Just check the main SH folder and you see that.

[gap]

Those first 4 bytes are in the ZHF files before the first extraction with Ref's tool. On the first extraction they are not there. If you save to ZHF then extract again? They are there.

That's exactly what happens to me too :yep:


I'd think a slight mistake on Ref's part of course. Now the question is does that mistake show in Game?

I suspect no. I think that the possible mistake becomes evident when you want open the RAW file in photoshop, but I am not 100% sure; I need to check that.


I suspect it's a magic header that is ignored or tells the engine to unzip the file.
Since we never use the unzipped version? We can just remove it for editing reasons.

Do you mean something added to the ZHF file whose remnants are found in the ouptput RAW file in the form of those extra bits? That seems unlikely to me, and I don't see the point of adding any header to those files, since the only reason for converting from ZHF to RAW is because RAW files can be edited in photoshop (but the supposed header would make the editing harder) :hmmm:


I'll do some more research on the re-compression Ref's tool does and see if that is added to the files.
If it is? We many need to do a whole new tool to fix that issue.
ZHF is just a part of a compression library so should be easy enough to do.

What I will do is use a ZHF in stock SH4. Extract it then save again to ZHF and compare.

Okay, good idea, though as I said I keep psd copies of my working files with all the editing layers, and in SH5 there are only two stock ZHF files that we might need to edit.

[propbeanie]

No, I didn't mean ref put that in there Jeff, but that maybe it looks like an unused portion of the file, and ref didn't touch it, but the game sees and uses it, though if it sees the ZHF in the folder, it already knows that that tile is altered... :hmmm: - maybe PS is putting it in there to note the edit?

gap, I'd forgotten I had saved in psd format and did that... but once you convert RAW to ZHF, you cannot then convert the edited ZHF to RAW for further edits, and expect good results... :roll: sorry for my addled brain... :salute:

[gap]

No, I didn't mean ref put that in there Jeff, but that maybe it looks like an unused portion of the file, and ref didn't touch it, but the game sees and uses it, though if it sees the ZHF in the folder, it already knows that that tile is altered... :hmmm: - maybe PS is putting it in there to note the edit?

I can be wrong, but I think that the most likely answer is a little oversight by Ref (see my considerations below), though I am a bit surprised that no one made Ref aware of it, when he released his tool... :hmm2:


gap, I'd forgotten I had saved in psd format and did that... but once you convert RAW to ZHF, you cannot then convert the edited ZHF to RAW for further edits, and expect good results... :roll: sorry for my addled brain... :salute:

No need to say sorry, mate. You are absolutely right though, going by what Jeff has said, the fix could be easy :)

[Jeff-Groves]

What I suspect is those 4 bytes are identification bytes.
Does Ref's tool put them into the file when recompressed to ZHF?
I suspect so.
Does SH Game engine ignore them then?
Maybe but We need to prove that before we say 100% Yes.
If Ref's tool does in fact throw in the extra data to the ZHF files (Which tests show me it does) Should we remove that issue or just work around it by removing the extra 4 bytes on re-export from a modded file.
Since SH4 depends more on those type files? The SH4 modders should let us know about issues and we can go from there.

[Jeff-Groves]

I've downloaded all the source code for zlib so I can check that 4 bytes.

[gap]

Okay, just tested:

https://i.imgur.com/QGpYoKy.png

Do you see the vertical line to the very left of the map? I didn't actually check in game, but if ME "sees" it, there is an high chance that the game engine will see it too.

That should end all our doubts: that's a little mistake by Ref.

Yes, probably SHIV modders could tell us more on that, though I think that propbeanie's experience is relative to the latter game and he knows as much as us on the topic.

On a side note: why the GWX name for the Terrain Extractor-Importer, since it is a tool eminently done for SHIV? :doh:

[Jeff-Groves]

GWX 4 for SH4 was well under way even as SH5 was being worked on.
There was even a Type 7A done for SH4 before SH5 ever came out.
It's a long story about the GWX Team. Some of it is not happy stuff.
Ref wrote many tools and all were for the GWX Team. That's how he wanted it.
If he released them at the time? They were named as he saw fit.

[Jeff-Groves]

Now. If you re-export a ZHF saved with ref's tool. Do nothing to it then re-ZHF it THEN export again?
It is all kinds of messed up.

My conclusion is the original export to RAW is fine.
The compression to ZHF is flawed thus any further decompression and recompression only increases the flaws leading to a useless file.
I have not tested deleting those bad 4 bytes with multi compression-decompression.

[gap]

GWX 4 for SH4 was well under way even as SH5 was being worked on.
There was even a Type 7A done for SH4 before SH5 ever came out.
It's a long story about the GWX Team. Some of it is not happy stuff.
Ref wrote many tools and all were for the GWX Team. That's how he wanted it.
If he released them at the time? They were named as he saw fit.

I understand. I am afraid that starting from the next version of the mod, the acknowledgement list will be actually bigger than the list of features lol :doh: :D
Definitely Ref deserved a place in it :yep:

Now. If you re-export a ZHF saved with ref's tool. Do nothing to it then re-ZHF it THEN export again?
It is all kinds of messed up.

My conclusion is the original export to RAW is fine.
The compression to ZHF is flawed thus any further decompression and recompression only increases the flaws leading to a useless file.

No, I have used ZHF-compressed files; they are fine.
RAW files exported from TerrainData.BFD are fine too.
The error is introduced with the ZHF to RAW conversion.

[Jeff-Groves]

The error is introduced with the ZHF to RAW conversion.
Did you remove the first 4 bytes?

[gap]

Did you remove the first 4 bytes?

No, I didn't, let me check if it works. I will report back in a short while :up:

[gap]

Okay, I think it actually works.

I used XVI32 for that, because I am more accustomed to its basic interface.
I removed the bytes marked in green in the picture below, and after that I could open the file in photoshop without further problems

https://i.imgur.com/jyh6yad.png

That was easy, thanks :yeah:

[Jeff-Groves]

Ref's decompression is putting those bytes in. It's a magic number that should have been excluded from the raw file output.
Makes me wonder a little about the SH3 Terrain files in GWX.

Now to test if Ref's tool messes with terrain in Game? We'd need to find a land area that it can show up on.
Or build a Deep Sea Explorer Sub. :)

[gap]

Ref's decompression is putting those bytes in. It's a magic number that should have been excluded from the raw file output.

Yes, that sounds plausible a flag or a marker of some sort that was needed by its code and he forgot to remove at the end of its cycle.

It is 4 am here. Time to rest.

Thank you for your inputs that are always illuminating, and let's continue this discussion tomorrow.

gap's off for now :salute:

[Jeff-Groves]

Sleep Good Mate!
:)

Your questions and observations feed me and give me directions to check that I don't think of on my own. Every small bit of the Game we learn helps us go further into other areas never imagined just a year ago!

You started with some terrain modding ideas and have led us to a whole new way to do terrain modding!

Not just for SH5. But has impact on SH4 as We have discovered, and possibly for SH3!

That is not an everyday, all kinds of people do it everyday, thing that happens!

:salute:

[propbeanie]

Art Anker encountered what he called "Lot's Super Wives" in an altered Carotio_SH4_TrukCoralHarbour (http://www.subsim.com/radioroom/downloads.php?do=file&id=4925) mod, at what I called the "corners" of the Truk Atoll tile. I can do some digging tomorrow for the actual tiles in the SH4 Terrain / Data folder. I also encountered similar when I was attempting to alter Pearl Harbor to make room for a more accurate Submarine base in the Southeast corner of the East Loch, near the Navy Yard. I wrote one row into the 6 pixel overlap, and ended up with a wall from as deep as you could go, to as high as you could go, a solid reddish / brown wall... There are more than one way to get errors when editing the files... Attempting to re-edit an already edited ZHF file, or editing within the 6 pixel overlap in SH4.

You are correct gap that what you see in the ME does show in the game - at least in SH4 it does... I was trying to Search for Art's old post, but failed to find it, and I was trying to find the "Morse Code" post that someone reported having seen "dots" & "dashes" in FotRSU. It was an odd looking bunch of "dots" and "dashes" in the ocean of deep and shallow "water", probably caused by an edit that I believe I had done with GWX & PS, but I can't remember if I did a re-edit or got into the 6-pixel overlap... Most of the edits I did in FotRSU were kind of done like the Army Corps of Engineers coming in and dredging channels and parking spots for the ships in a lot of the harbors, that while historically deep enough for BB and CV, but in which the game just can't even handle a CL, due to the way SH4 draws a ship's hull. You need about an extra 4 to 6 meters underneath, else you get all that shtinking crashing and grinding noise of the ship hitting "bottom"... :salute:

[gap]

Your questions and observations feed me and give me directions to check that I don't think of on my own. Every small bit of the Game we learn helps us go further into other areas never imagined just a year ago!

That's true. I am glad that, besides keeping us awake until late, my silly questions have unexpectedly positive side effects too :O:


You started with some terrain modding ideas and have led us to a whole new way to do terrain modding!

Not just for SH5. But has impact on SH4 as We have discovered, and possibly for SH3!

That is not an everyday, all kinds of people do it everyday, thing that happens!

:salute:

One lesson I learned since I started messing with SH5 files, back in 2011, is that the distinction between SH5, SHIV and SHIII modding is somewhat artificial, and detrimental to the development of the SH modding community as a whole.

Personally I tend to see the three games as different specimens of the same thing, and when I have a modding idea I often think about its feasibility within the three of them. Indeed, some important changes have been brought in SHIV over SHIII, and in SH5 over SHIV and SHIII; each game has its own character, its pro's and its con's, and not all that is possible in one of them is doable in the others too, but all the SH games also have many striking similarities that we can exploit. That's obvious for someone who started from SH5, because this game was the last in the series and it owes many of its features to its predecessors, so it is natural for him to look back at them, but we should also admit that there are things about SHIII that you learn better when you dig into its younger brothers.

Unfortunately, at least in the past, there are has been an hostility between different modding communities and groups that has not eased the sharing of knowledge and ideas, but I am convinced that in our small way we should do our best to bypass this narrow-minded mentality and encourage the knowledge-sharing, and I think this is exactly the spirit of our conversation on this mod :up:

Art Anker encountered what he called "Lot's Super Wives" in an altered Carotio_SH4_TrukCoralHarbour (http://www.subsim.com/radioroom/downloads.php?do=file&id=4925) mod, at what I called the "corners" of the Truk Atoll tile. I can do some digging tomorrow for the actual tiles in the SH4 Terrain / Data folder. I also encountered similar when I was attempting to alter Pearl Harbor to make room for a more accurate Submarine base in the Southeast corner of the East Loch, near the Navy Yard. I wrote one row into the 6 pixel overlap, and ended up with a wall from as deep as you could go, to as high as you could go, a solid reddish / brown wall... There are more than one way to get errors when editing the files... Attempting to re-edit an already edited ZHF file, or editing within the 6 pixel overlap in SH4.

Yes, though to a much lower scale, what you are talking about is also visible in stock SH5 where 'high-detail' terrain tiles interconnect with each other and with 'regular' tiles. In theory there are ways to make tile borders seamless (by 'overlapping' nearby tiles, or by smoothing their seams in TE), but probably devs didn't have time to do it, or the judged that no one would have noticed the slight height mismatches around the borders of all the main ports, canals and choke points, or wherever they needed an higher terrain detail level.

Besides not observing the 6-pixel overlapping, what could have caused the problems you have described above is probably the little glitch that we have discussed yesterday (concerning the ZHF to RAW conversion) and, maybe, the lossy nature of the ZHF compression.

In any case, this morning I checked my edit of the SH5 terrain around the Channel Islands, and everything looks okay on its borders. What I had done to keep the 1-pixel overlapping I had found in stock files, and to ensure a smooth transition between my edited terrain and the stock terrain that surrounded it, was pasting the stock height map for the same area as a new layer on top of of the height map I had previously edited, and applying to it the following layer mask:

https://i.imgur.com/JXdbKxN.png


You are correct gap that what you see in the ME does show in the game - at least in SH4 it does... I was trying to Search for Art's old post, but failed to find it, and I was trying to find the "Morse Code" post that someone reported having seen "dots" & "dashes" in FotRSU. It was an odd looking bunch of "dots" and "dashes" in the ocean of deep and shallow "water", probably caused by an edit that I believe I had done with GWX & PS, but I can't remember if I did a re-edit or got into the 6-pixel overlap...

Okay, let me know if you can find the old posts describing those problems...


Most of the edits I did in FotRSU were kind of done like the Army Corps of Engineers coming in and dredging channels and parking spots for the ships in a lot of the harbors, that while historically deep enough for BB and CV, but in which the game just can't even handle a CL, due to the way SH4 draws a ship's hull. You need about an extra 4 to 6 meters underneath, else you get all that shtinking crashing and grinding noise of the ship hitting "bottom"... :salute:

Yep, that's a known problem;in practice collisionable units collide with the sea bottom several meters before they actually hit it. Add to that the other issue of AI units getting stuck in low waters before they actually collide with anything, and the picture is complete.

I don't remember whether the cause for the first problem are collision spheres protruding from the keel or what else but, IIRC, back in the days TDW had created a patch that solved the issue for SH5; it should be included in the last version of the patcher.

[Jeff-Groves]

This what you referring to?
http://i1157.photobucket.com/albums/p583/icthird623/Hailstorm/SH4Img2012-06-16_132323_049Custom3.jpg

[bstanko6]

Is that Supermans fortress of solitude forming? It's beautiful!

[Jeff-Groves]

That is an anomaly spotted in SH4 near Turk.
We believe it is caused by extra bytes added to the modified terrain files when recompressed to ZHF files.
Or bad editing of edge pixels before recompression.

[propbeanie]

... Yep, that's a known problem;in practice collisionable units collide with the sea bottom several meters before they actually hit it. Add to that the other issue of AI units getting stuck in low waters before they actually collide with anything, and the picture is complete.

I don't remember whether the cause for the first problem are collision spheres protruding from the keel or what else but, IIRC, back in the days TDW had created a patch that solved the issue for SH5; it should be included in the last version of the patcher.
I am going to have to look into that... It seems sometimes like the game "drops" the ships into the water when it spawns them, and they sometimes "bounce" and then run shallow, or "bounce" and get damaged and then ride low in the water.

That is an anomaly spotted in SH4 near Turk.
We believe it is caused by extra bytes added to the modified terrain files when recompressed to ZHF files.
Or bad editing of edge pixels before recompression.
Since I'm so-oh good at bad edits, if I get time tomorrow or Tuesday, I'll try a little of each, purposefully making "mistakes", and see what each type of those errors do to the in-game display. As for the pillar, other than color (I thought the ones I had were brown), they can also be jagged on the edges of the pillar, and not smooth looking at all when up close to them. That might depend upon the type of error created also... :salute:

[bstanko6]

Here are some pictures during my passage through the Kiel canal. Ecomod in full affect. When I started in the pens, the trees were green. Did not take a photo, but as I passed, the tree colors changed. Is that the mod working? because it is seriously awesome! I never really cared about harbor mods, but this mod is so damn cool!

Here I am halfway through the canal when I noticed the trees changing color. It is mid September.
https://i.ibb.co/m64bBC2/SH5-Img-2019-02-26-03-18-30.png

Passing Brunsbuttel.
https://i.ibb.co/px9wRPg/SH5-Img-2019-02-26-12-56-44.png

A closer look at the trees in Brunsbuttel.
https://i.ibb.co/2Yp7LvP/SH5-Img-2019-02-26-03-18-21.png

[gap]

Did you start from Kiel and what was your in-game date when you took those screenies? :)

[bstanko6]

Started in Kiel 19SEP39 @ 0700
Headed for Scapa Flow

[bstanko6]

Good stuff gap!

[gap]

Started in Kiel 19SEP39 @ 0700
Headed for Scapa Flow

Okay, that makes sense: the Kiel end of the canal is located on a different terrain type than the Brunsbuttel end. That's why you get to see different vegetation types while you cross the canal.
The alpha version of the mod that you are using only features two new terrain types, so the trees you are seeing in Kiel (and in several other locations) are part of the stock "temperate alpina vegetation and their dark green color doesn't change much throughout the year, thencr the striking contrast with the red/yellow colors I have chosen for autumn foliage of the new trees.

Each terrain type can have up to eight different tree types (I wish we could add more, but that's an hardcoded limit). The new terrain types that I will add to the mod - right now I have four more ready on my HD, waiting for the next release, and I am planning many more - will have about half of their trees in common with the previous terrain type, so to make the landscape not to change as abruptly as you have experienced during your last patrol, and forest types to gradually transition to other types as you travel along coasts, canals, etc... :)

[bstanko6]

Honestly I did not think it was an issue! I thought I was watching seasons change!!!