Almagest. Strict Import for SH5.

[Jeff-Groves]

One of the biggest problems when importing with TDW's program is the loose import.
File size grows in an astronomical way!

1st thing you'll ask is "Why that Name?" Look it up!
:har:

2nd thing you'll ask is "How do you know it will work?"
:hmmm:

There are many posts in the past where I TRIED to explain this all. Those are hard to find and it's time to teach a new crowd.

This is going to be a stand alone program ANYONE can use! No math. No 010 editor. Just run it and your files will be adjusted to a well known standard.
Well know by me any way!

Neal and Jimbuna helped solve my connection issues so this is my way of giving back to ALL my friends here at SubSim.
:salute:

[Jeff-Groves]

Why THAT name?

In the second century, Claudius Ptolemy, an Alexandrian astronomer and mathematician, wrote Mathematike Syntaxis (in Greek) or The Mathematical Compilation, a treatise on the apparent motions of the stars and planets. This work soon became known as The Greatest Compilation and it established the model of a geocentric universe, a scientific schema that would be followed for the next thousand years. When Ptolemy’s Greek work was adopted in the Islamic world, its title in Arabic was shortened to The Greatest, which when transliterated into Latin became Almagest.
:har:

People come up with some of the craziest names so this is my name for the program!

[Jeff-Groves]

Let us talk about details.
1st thing one needs to have is a complete understanding of the obj format.
It's a text file yes but do you REALLY understand how it works?
Then you need to have a complete understanding of how those files are adjusted and stored in a GR2 file.

Pretty sure very few reading this do know. You see what is exported and probably don't even look at the files in a text editor.
Even if you do it's Greek.
(Another salute to the Name! :haha:)

[Jeff-Groves]

I'm not going to explain the object file format. You can look that up for a basic understanding.

What I AM going to explain is HOW a GR2 file uses the format to store the information and how it does that!
Knowing that information tells me HOW to write a Converter!

Some of the information can also be used to Ghost S3D!

[gap]

Of all the mad scientists and inventors, you are by far the one with the craziest names. :har:

Following this thread.

Long life to the almighty Almagest :up:

P.S: if someone wants to know more on the wavefront (obj) format, I can provide the basics :salute:

[Jeff-Groves]

RAD Games GR2 format for storing the 3D object files is basically an obj file.
The mystical part is how they did it. (To most people)
Why do they export with so many faces and verts and such?

That is where a FULL understanding of the obj file format comes in SO important!

RAD took full advantage of the format in a way that just confuses most!
I'll call it a shuffling of the deck type approach.
We take 2 decks with different faces, discard what we don't need, then shuffle the rest but in an organized way with a template of where each card is.

Each card is recorded in the Faces section of the obj format with a number.
Each number in the Faces section is a line number.
So if we want vert 75? It will be in that catalog as 75.

[Jeff-Groves]

Knowing all this? How did RAD do it in GR2?
Easy! They read the files, drop what they don't need, add what is needed, make sure file placement numbers are equal, then adjust the faces section to call the proper lines!

Being obj files are text files? Pretty easy to shuffle the deck in our favor!

[Jeff-Groves]

A seemingly magical process of transformation, creation, or combination.

In all my years watching GR2 editors and such? The best I have seen is TDW's version. I rate it the best as it does not use the granny2.dll.

He did miss the one critical thinking about imports.
Now even S3D missed the mark so not the first time!

It is easier to work around S3D's faults but maybe I'll include that during programming.
:hmmm:

Now I am concerned about the transformation\combination in this Alchemy.
I'll need to read 2 separate files, program to find the differences, then adjust everything to create 2 new files that meet the requirements.

Being doing that 2 files at a time would be slow?
Gonna need to make it a mass Alchemy type program.
So put all the files in a folder, click a button, and stand back!

[Jeff-Groves]

I will take files from people to experiment on.
I'll need both the normal file and any AO files.

Better to test what you have then play with something that may not match
what you are working in.

[kapuhy]

I will take files from people to experiment on.
I'll need both the normal file and any AO files.

Better to test what you have then play with something that may not match
what you are working in.

First of all, thanks for making the effort to create Almagest :up: It will be invaluable for anyone planning to create new content for SH5.

Now, if I read you correctly re:files, you'd like examples of files we've been importing using TDW's tool in order to experiment on them? In that case, here's a sample:

Plane model (https://drive.google.com/file/d/14uSJ1-aEtaLf4e6VCWYeDg1l0PnBFnpW/view?usp=sharing)

That's a plane I've been working on recently, upon importing it using TDW's program and following settings:
- Loose import
- Vertex data, uv1 data, normal data, calculate tangent and binormals: yes
- uv2 data, force subset ordering, update materials - no

Resulting GR2 file of that 8K vertices model is 1683 KB, while - for comparison - stock Avro Anson (7,1 K vertices) is 657 KB. I packed into linked file both .obj files before importing and the .gr2 file after import.

[gap]

@ Jeff

a quick question: if I got you correctly the core of your idea, when using your program we will need to check that the model we want to import in game has a number of of vertices and faces equal or lesser than the stock GR2 template chosen for the import. Is that correct?

If that was true, it would be a big incentive to keep our models as simple as possible...

[Jeff-Groves]

Strict import doesn't matter about size of the imported file.
Strict means all verts and such match on their numbers.
So if you have 100 verts you need to have 100 text coords and 100 normals (If included)
Both the diffuse obj and AO obj MUST match those numbers!

AND the faces must match! That is the tricky part.
:03:

I'll need to test the different versions of TDW's program as I'm not sure how he's defining Strict at this time.

[Jeff-Groves]

@gap, I hope you down load both mine and kapuhy's files to examine!




@kapuhy,

Using nothing but 010 editor to change your rudder,
I imported your rudder to the Avro GR2 file.
There is NO file size change to the GR2 file!
:D

Look at the rudder size in your edited GR2 compared to mine!
I also opened my GR2 with Goblin.

Testing TDW's program I did remember a few things I forgot!
So this is going to be easier then I recalled!

I'll upload the test GR2 so you can see it for yourself.
:up:


https://www.mediafire.com/file/vypnt2b9aiquo5s/test1.GR2/file

[Jeff-Groves]

@ Jeff

a quick question: if I got you correctly the core of your idea, when using your program we will need to check that the model we want to import in game has a number of of vertices and faces equal or lesser than the stock GR2 template chosen for the import. Is that correct?

If that was true, it would be a big incentive to keep our models as simple as possible...

So far I'd say stay smaller then the target GR2.
As I stated above? I'd forgotten a few things about Ghosting TDW's program.
It's all coming back now!
:har:

[Jeff-Groves]

So 3D object import as Strict works.
I still need to work on the texture part but it's looking good so far!

What I'll do now is to export the rudder and check the textures UV's in 3DS max.
That should give me the information I need to adjust things.

[Jeff-Groves]

I think I need to flip tex coords on import.
:hmmm:

PERFECT! It's on like Donkey Kong now!
(According to the Urban Dictionary (www.urbandictionary.com) “Its on like Donkey Kong” is: a phrase to denote that it's time to throw down or compete at a high level; something is about to go down.)
https://www.subsim.com/radioroom/picture.php?albumid=1069&pictureid=11850

[Jeff-Groves]

I have the basic stuff done.
I need something that uses a Diffuse obj and an AO obj file now.
:03:

[kapuhy]

Using nothing but 010 editor to change your rudder,
I imported your rudder to the Avro GR2 file.
There is NO file size change to the GR2 file!
:D

Look at the rudder size in your edited GR2 compared to mine!
I also opened my GR2 with Goblin.

Great :up:

Just checked - 656 KB both files, and for comparison - if I import rudder into Anson using loose import file grows to 660. Might not seem much for just this one part, but it adds up - when all parts are imported with loose import files grow to about twice original size.

[Jeff-Groves]

Great :up:

Just checked - 656 KB both files, and for comparison - if I import rudder into Anson using loose import file grows to 660. Might not seem much for just this one part, but it adds up - when all parts are imported with loose import files grow to about twice original size.
Yes. File sizes are crazy under the Loose Import option!
It's very easy to Ghost and import with TDW's program.
But I don't expect people to do it by hand as I can thus I'll write a program to do it for you!
As I am testing I see I'll need a way to tell you if a targeted GR2 will work.
I'll get the conversion function working then look into that problem.

Here's the rudder obj file I adjusted.
Compare it to your original rudder obj.
You'll see what I did if you look at them in Notepad or 010.
Plus you can import it yourself and actually see the results!

https://www.mediafire.com/file/0p1njtaj3uggnqh/test-Ruder.obj/file

[Jeff-Groves]

So a Strict import MUST match existing information in a GR2 file as I can see.
Not a problem as long as you select a file with 3D models below or the same as your intended imports.

Lots of ways around that. It's going to be up to you to figure out somethings.
Say you have a model with larger sizes for a 3D object.
Cut it into pieces and use a GR2 that will accommodate more parts if need be.

Say the selected GR2 has to many parts? Throw in a totally zeroed part!
Or just add something special.

Important part is to reduce the ridiculous Loose import option and create files of a nominal size!

I don't need any Tool but 010 editor to do anything I want with a GR2 file.
But that's just me.
You need and use other programs so I'm just giving you what I talked about years ago.
You have now seen the proof of concept by example files.
It's not a theory. It's a proven way to reduce file sizes using an existing Program by TDW.
I'm not criticizing his work! It's a fantastic program!
I'm just addressing flaws as I have done with S3D in the past.

Should TDW make a magic appearance here? I'd gladly tell him how to adjust his Program.

[kapuhy]

I have the basic stuff done.
I need something that uses a Diffuse obj and an AO obj file now.
:03:

Here's another test subject - this is one of relased ships from my coaster mod, has diffuse, ao as well as damage and lod models. I imported it using loose import setting into NF_boat_1 (Coastal Boat) gr2 file (both obj before import and imported files attached).

Test subject - motor coaster (https://drive.google.com/file/d/1sG2mB_9PESK1VzTlWj4Jyr11qAQ6TSvF/view?usp=sharing)

[Jeff-Groves]

@kapuhy

Those files are exactly what I wanted to look at.
They show the problem I'm working to fix to allow them to be adjusted and imported as Strict.
:salute:

If you look at the 2 files for the rudder you can see the problem when you look close at the faces section of the obj files.
Same number of faces but they do not match! That is the shuffling part that I spoke about.

[Jeff-Groves]

One of the things I'm working on is to test different sized obj imports under the Loose option.

I'm trying to see if that option can be used to create custom sized perimeters
that We could then use for a strict import to keep file size as low as possible.

The adjustments to the files one creates are figured out and tested.
Just need to work on the code for that.

It's actually really simple based on how the GR2 file is constructed and how the obj format works.

[Jeff-Groves]

Yes. We can shrink the meshes in a GR2 file with a loose import.
Not a perfect way to do so since the Loose import is based on false assumptions and a total misunderstanding of how the GR2 meshes work with the obj format.
But a good work flow option to take advantage of.

[gap]

Yes. We can shrink the meshes in a GR2 file with a loose import.
Not a perfect way to do so since the Loose import is based on false assumptions and a total misunderstanding of how the GR2 meshes work with the obj format.
But a good work flow option to take advantage of.

Thank you Jeff :yeah:

Currently working on my model's diffuse mapping (AO mapping already done). I am eager to see the result of my efforts in game, and I hope your program will help me to optimize the import.

[Jeff-Groves]

Let's look at the 3D objects stored in a gr2 file.
Verts, normals, textures, etc ALWAYS have the same count.
Faces are totally separate from that.

So here's the magic and simplicity.
We can match the verts and such by just copying existing verts or whatever and pasting so each has the same count.

500 texture points? Make sure verts is 500 etc.

The Faces will ONLY USE what they call!
Thus no loading of the extra duplicate information!
So I can have 500 verts and textures, etc BUT the faces say only render 100!

Don't believe me?
Export a mesh from a GR2, open it in any 3D program then save it.
If your program optimizes an obj file? Compare them in notepad or any other text reader.

[Jeff-Groves]

How will this program work?

It will look at both a base obj file and if selected the AO version.
Then it will adjust the files to have the same counts.

Once saved? You find a candidate GR2 file.
Say you need 500 verts and 95 faces to do a Strict import.
Your files have 400 verts and 35 faces.
We run the program and tell it the verts and faces needed.
It will adjust them for a strict import from there.

This is just Ghosting both TDW's program AND the GR2 files based on knowledge of how both work!

A similar approach will Ghost S3D also!

I prefer to call it Ghosting instead of Faking it till you make it.

[kapuhy]

Once saved? You find a candidate GR2 file.
Say you need 500 verts and 95 faces to do a Strict import.
Your files have 400 verts and 35 faces.
We run the program and tell it the verts and faces needed.
It will adjust them for a strict import from there.

Forgive layman question :) If my files have 600 verts and 115 faces, I'll need to find a template gr2 with higher count than mine, right? So instead of having to find template with identical count like is now the case (which is pretty much impossible unless we're just making minor edits to stock model), we'll use templates with <equal or lower> count?

[Jeff-Groves]

You will need to find a GR2 with as close a count as possible but ALWAYS higher then your adjusted files.

You will still get a smaller ending GR2 then loose import will ever give you!

Remember this point!
Say you find a nice GR2 but it just has to many parts.
We can use the loose import to shrink them to nothing thus reducing the size even more!

[gap]

Let's look at the 3D objects stored in a gr2 file.
Verts, normals, textures, etc ALWAYS have the same count.
Faces are totally separate from that.

So here's the magic and simplicity.
We can match the verts and such by just copying existing verts or whatever and pasting so each has the same count.

500 texture points? Make sure verts is 500 etc.

The Faces will ONLY USE what they call!
Thus no loading of the extra duplicate information!
So I can have 500 verts and textures, etc BUT the faces say only render 100!

Don't believe me?
Export a mesh from a GR2, open it in any 3D program then save it.
If your program optimizes an obj file? Compare them in notepad or any other text reader.

One of the biggest problems I had in the past, was matching vertex order in the main and uv2 meshes. There is a bunch of reasons why various obj exporters might change that order, but the final result is the same: after importing in GR2 Editor, the AO mapping is messed up even though the two imported objects are apparently identical in their vertex/face structure.

In theory, if all the vertices in an object had different coordinates, we could easily create a tool which will automatically sort uv-2's vertices according to main object's template. The problem is that, after edge-splitting hard edges (for correct smoothing in game), most import candidates will have a lot of duplicated vertices with identical xyz and uv coordinates, but placed on different meshes. Making our tool to automatically discern them wouldn't be that easy.

You will need to find a GR2 with as close a count as possible but ALWAYS higher then your adjusted files.

You will still get a smaller ending GR2 then loose import will ever give you!

Remember this point!
Say you find a nice GR2 but it just has to many parts.
We can use the loose import to shrink them to nothing thus reducing the size even more!

If I got you correctly, the above criterion should be applied not only to the main object, but also to all the subparts. Finding a GR2 file which will match the requirements for all the model parts, or shrinking all the parts in our models so that they won't exceed the vert count of the GR2 template they will be imported in, might be a little PITA, but I definitely see the advantages of your program and of the working method you are suggesting :salute:

[Jeff-Groves]

Don't worry about Vert count nor texture count. That is easily sorted!
On face count? Those should be the same as far as count but can be adjusted.
They may be different but the GR2 file accounts for that!

[AMZ]

Hello Jeff-Groves,

Following your feedback

With just a little text editing of the obj files you can do a strict import
and get a much smaller file size.


Personally?
I use a script I wrote for 010 that looks at the new obj files and adjusts them for a strict import.

Hello Jeff-Groves,

Sorry to answer you so late, I admit I did not understand the meaning of your advice.
Today I am less stupid, thanks to your writings elsewhere.

In this regard, you say that you have made an application: "Almagest".
Is she available? Where can we find it?

Thank you




On the other hand, maybe you have an idea of what concerns me?

Click Here ; Hi to all After the first advice, and thank you for that, I thought I'd manage on my own, but I have some problems with the realization I am coming back to you in the hope of having a solution. My problem: Creating an earthly object as a characteristic Seamark Example: a church, a bell tower, a lighthouse, etc. Here are my questions: 1 - TDW editor for GR2 Files seems a great tool, but for lacking of information I am not an expert in its use. Can we create from nothing, from zero *. the skeleton bones and a hierarchy of these bones? *. Initial parent with his own name then parentage for the Meshs (Diffus - AO) then Meshs Collision? 2 - I got around the No. 1 problem for my first realization by copying an existing object Either in C:... / ... 'Silent Hunter 5 with TWOS'data-Terrain-Locations'CustomAreas-germany The Church object: church. GR2 and associated .sim Note: This one as is in the game, is not visible if placed with the mission editor ... The 3D model is very far from the reference point, about 1 km after Gmax or Blender. So with Blender I remade the church at the reference point, so that an placement in the mission editor corresponds to the place pointed and seen in the game, map or TAI . Of course, I transcribed the exported files in .obj format using the Note Block. I checked the correct transcription by importing and exporting again in the Wings3D editor. Using TDW-editor, I imported the renamed files Church_AMZ.obj Church_AMZ_AO.obj And Collision_ Church_AMZ.obj They appear well at Zero Point ... It's ok. But since this is a clone of the GR2 file with 3D replacement, I decided to replace the names of the skeleton and bones, then the parents Meshs, in order to have the identifiers remapped. Not expert with the TDW, I did it using a Hexa editor. Here's my second question: *. Can we in TDW change the names of skeleton, bones etc? *. In the same way for the Meshs, can we edit the names of the Parents masters? Sometimes this is possible for certain filiations. ??? Not being a programmer, I just replaced Letters by Letters the name without changing the length of the GR2 file (Hexa). Of course, I remapped the Church_AMZ.sim file accordingly. The Church_AMZ.GR2 file thus modified opens correctly in the TDW, after having automatically corrected the byte counter ... Control by opening everything in GobinEditor (_.GR2 and _.sim merge) ... Everything seems correct! I did a mission (MissionEditor) in order to have the visual in the game ... There's nothing??? Please, can you give me a post where I could refer Can you help me in my process With all my gratitude.[/QUOTE]"]Hi to all


After the first advice, and thank you for that, I thought I'd manage on my own, but I have some problems with the realization

I am coming back to you in the hope of having a solution.
My problem: Creating an earthly object as a characteristic Seamark
Example: a church, a bell tower, a lighthouse, etc.


Here are my questions:

1 - TDW editor for GR2 Files seems a great tool, but for lacking of information I am not an expert in its use.

Can we create from nothing, from zero

*. the skeleton bones and a hierarchy of these bones?
*. Initial parent with his own name then parentage for the Meshs (Diffus - AO) then Meshs Collision?

2 - I got around the No. 1 problem for my first realization by copying an existing object

Either in C:... / ... 'Silent Hunter 5 with TWOS'data-Terrain-Locations'CustomAreas-germany
The Church object: church. GR2 and associated .sim

Note: This one as is in the game, is not visible if placed with the mission editor ...
The 3D model is very far from the reference point, about 1 km after Gmax or Blender.

So with Blender I remade the church at the reference point, so that an placement in the mission editor corresponds to the place pointed and seen in the game, map or TAI .
Of course, I transcribed the exported files in .obj format using the Note Block.
I checked the correct transcription by importing and exporting again in the Wings3D editor.

Using TDW-editor, I imported the renamed files
Church_AMZ.obj
Church_AMZ_AO.obj
And Collision_ Church_AMZ.obj
They appear well at Zero Point ... It's ok.

But since this is a clone of the GR2 file with 3D replacement,
I decided to replace the names of the skeleton and bones, then the parents Meshs, in order to have the identifiers remapped. Not expert with the TDW, I did it using a Hexa editor.

Here's my second question:

*. Can we in TDW change the names of skeleton, bones etc?
*. In the same way for the Meshs, can we edit the names of the Parents masters?
Sometimes this is possible for certain filiations. ???

Not being a programmer, I just replaced Letters by Letters the name without changing the length of the GR2 file (Hexa).
Of course, I remapped the Church_AMZ.sim file accordingly.
The Church_AMZ.GR2 file thus modified opens correctly in the TDW, after having automatically corrected the byte counter ...
Control by opening everything in GobinEditor (_.GR2 and _.sim merge)
...
Everything seems correct!
I did a mission (MissionEditor) in order to have the visual in the game ... There's nothing???



Please, can you give me a Thread where I could refer ?
Can you help me in my process ?

With all my gratitude.

[Jeff-Groves]

A few months down the road and several Alpha versions to sort problems and all is GOOD!
Mostly real life obligations have kept me away from the final Coding parts.
Final parts are to actually write the Faces and texture sorting.
That I wrote in my head while driving all over the USA.
Then debugged much of it in my head!
:o

[kapuhy]

A few months down the road and several Alpha versions to sort problems and all is GOOD!
Mostly real life obligations have kept me away from the final Coding parts.
Final parts are to actually write the Faces and texture sorting.
That I wrote in my head while driving all over the USA.
Then debugged much of it in my head!
:o

Good to hear it! I have a queue of ship models on the slipways waiting for import to SH5 and I'm looking forward to use Almagest on them. Keep up the good work as BdU uses to say :up:

[Jeff-Groves]

I did figure out a lot that allows me to 'kind of' brand any files.
:haha:

It does NOT have any effect on SH5 files using Almagest.
And you'd need the Key files I also created to decode the hidden information!

Call it an Easter Egg! I'm well known for those!
:03:

[gap]

@ Jeff

Are you still interested into testing your importer on my Clyde puffer model?
I am still working on textures (AO and diffuse map are WIP and specular and normal maps are missing), but 3D models and UV maps (both diffuse and AO) should be OK for import. Drop me a PM or an e-mail if you want to check them :salute:

diffuse map:
https://i.imgur.com/FJD9VbN.png

AO map:
https://i.imgur.com/yJn28Uh.png

diffuse + AO map:
https://i.imgur.com/O11E8k1.png

[Jeff-Groves]

@ Jeff

Are you still interested into testing your importer on my Clyde puffer model?
I am still working on textures (AO and diffuse map are WIP and specular and normal maps are missing), but 3D models and UV maps (both diffuse and AO) should be OK for import. Drop me a PM or an e-mail if you want to check them :salute:



Send me a link by PM! Would love to test them!

[gap]

Send me a link by PM! Would love to test them!

Excellent, thank you Jeff! I have all the parts in one Wings file. Can I send them in their original format or do you prefer me to convert them as obj files?

[Jeff-Groves]

obj files please.
I've only coded to read and adjust obj files at this time.
Now I may change that to read Wings files also at some point.

[gap]

obj files please.

OK, in that case exporting all the parts one by one will take some time. Please wait until tomorrow :)


I've only coded to read and adjust obj files at this time.
Now I may change that to read Wings files also at some point.

That indeed would be a nice for a lazy boy like me, but not a priority :up:

[Jeff-Groves]

That's fine. I'm not in a rush.
I'll explain ONE MORE TIME how this works.

GR2 stores 3D objects in a known way.
We are only interested in the
V
VN
VT
since TDW calculates the Tangents, Bi-tangents on import.

Here's where he went TOTALLY off on some unknown tangent!
All the V, VN, VT counts must match in how many there are.
75 V's? You have to have 75 VN and VT's. Simple to add junk for that.

Faces? Whole different story! And thus the sorting needed!

Only one set of Faces is stored in a GR2 file.
So what may be texture face 2 in a main obj file may be texture face 5 in the AO version!
We MUST do a swap on the main obj files textures (VT) placement!

Now. Say We add to the V, VN to match VT Sizes. Will that show in Game or be read?
NOPE! If the extras ARE NOT listed in the Face section? You will never see them and they will be ignored!

You can't see an object that has no faces! Nor will that information be read by the Granny system!

[tonschk]

Thank you for your hard work Jeff-Groves :salute::up::yeah:

[Jeff-Groves]

OK, in that case exporting all the parts one by one will take some time. Please wait until tomorrow :)



That indeed would be a nice for a lazy boy like me, but not a priority :up:
I took a look at the source code for Wings3D to see if it would be possible to do a direct conversion from the .wings format.
What I found is that will very probably not happen.

Still looking at the source to see if a plug-in could be done for a direct export from Wings3D to do what Almagest does.
I'll say MAYBE.
:hmmm:

[Jeff-Groves]

To Vector or NOT To Vector. That was the question. For a while anyway.
Calling on the Dark arts and thoughts of Technomancy?
I Vectored. But only to get iterators. Then I pushed Vectors into Arrays.
After that I brought in the Strippers! They did their work and set everything up to switch things around by my Managers! The Bouncers did some work also.
:har:

[gap]

Hey Jeff, I am dropping this message just to inform you that I didn't forget about my commitment with you.

During the last weeks I have been rather busy with family and work stuff. In the meanwhile I have been slowly working on some final touches, like bulk cargo and debris getting visible when a well aimed gun shell rips away the cargo hold cover. :D :arrgh!:

https://i.imgur.com/GNvoCm1.png

There will be five types of cargo configurable via .cfg file: coal, stones, gravel, sand and grains. These should reflect the most common kinds of goods our sturdy puffers carried in reality.

Wait for my files before the next year :salute:

[Jeff-Groves]

The basic code already will do things with no AO mappings.
The newest code will do all the work to correct the obj files in a blink of an eye!
Just need real world test files from different 3D programs as not all are the same once exported to the obj file format.

[gap]

The basic code already will do things with no AO mappings.
The newest code will do all the work to correct the obj files in a blink of an eye!
Just need real world test files from different 3D programs as not all are the same once exported to the obj file format.

All my models are created in Wings3D, but the files I am going to send you will probably be exported from Blender. Reason for that is that Blender has ways to quickly split hard edges. Split edges are needed for correct model smoothing, as neither the GR2 format nor the dat format support hard edges.

Unless you advise otherwise, my workflow will be: export from Wings3D as obj => import in Blender => apply edges split modifier => export from Blender as obj.
I will repeat this process for all the meshes composing my model, both for main mesh and UV2 mesh.

[Jeff-Groves]

I believe kapuhy uses blender and his files are what I would call proper.
I'd still like to see Wings files also.
You and I both know that SOMEONE SOMEWHERE will bitch that my program don't work with what ever they are using to make models!
I want to cut them off at the knees!

[gap]

Unless you make your importer to split hard edges automatically, the Blender step is needed, otherwise the game will try smoothing edges which it should not.

A good AO map can hide smoothing errors to a certain degree. Moreover, I suspect the GR2 import process to split edges which are not contiguous in the UV map. Yet, splitting edges before import is still advisable.

Doing that manually in Wings would take ages. If you want to test your program on Wings3D object files I could import the models exported from Blender back in Wings and export them again. Problem is (don't ask me how/why) Wings' obj importer won't recognize edges split in Blender. It simply welds them back.

Something has changed either in Blender's obj exporter or in Wings's obj importer, because this problem started only a couple of years ago. If you prefer I can try installing some old version of either program, and see if I can bypass the problem.

[Jeff-Groves]

I'd prefer both versions. Wings and Blender.
The way the GR2 saves files don't give a crap about things like that.
What goes in is what comes out so to say.

[gap]

I'd prefer both versions. Wings and Blender.
The way the GR2 saves files don't give a crap about things like that.
What goes in is what comes out so to say.

So are you okay if I send you two sets of files? One exported directly from Wings 3D with hard edges still in place, and the other exported from Blender with split edges?

If memory serves, Skwas' dat importer does not handle smoothing groups (it throws an error when it meets them), but TDW's GR2 importer will just ignore them.

[Jeff-Groves]

2 sets would be great. Then I can compare imports to see the effects.
I basically just ignore any smoothing group statements.

[gap]

2 sets would be great. Then I can compare imports to see the effects.
I basically just ignore any smoothing group statements.

Hey Jeff, please check your inbox :D

@ ALL

Belated wishes of happy new year buddies! After so many patrols I hope 2022 found you in port and enjoying your leaves :()1: :woot:

[gap]

@ Jeff

I have updated my files (corrected a few Wings3D exports which had some hard edges wrongly set as soft, and added Blender-exported models with split hard edges). The download link of the updated files is the same as the one I sent you yesterday by PM. If you had already downloaded it, please discard the first download and download again :salute:

[Jeff-Groves]

Just downloaded the file.
:up:

[Jeff-Groves]

A quick look at both Wings and Blender versions tells me a few things.

I can grab information from the Wings version that will make those a tiny bit faster to process. Basically? Since the counts of faces, verts, and such are listed? I don't have to run a loop to count all that on Wings files.

Blender files I'll have to do a loop just to count the stuff.
Probably won't be able to notice it but annoying none the less.

3DMax files also don't need that extra 'Blender' loop.

Blender files you sent seem to show a 'Better' way to organize faces so may reduce sort times. Going to be interesting to see the times for both.
Again it will probably be so fast one really won't notice!


Ahh. Wings also does not list a Normals or texture count so a partial count loop is needed.

I'd also check your settings for Wings to see if you can reduce the number of output to 6 digits.

[gap]

A quick look at both Wings and Blender versions tells me a few things.

I can grab information from the Wings version that will make those a tiny bit faster to process. Basically? Since the counts of faces, verts, and such are listed? I don't have to run a loop to count all that on Wings files.

Blender files I'll have to do a loop just to count the stuff.
Probably won't be able to notice it but annoying none the less.

3DMax files also don't need that extra 'Blender' loop.

Blender files you sent seem to show a 'Better' way to organize faces so may reduce sort times. Going to be interesting to see the times for both.
Again it will probably be so fast one really won't notice!


Ahh. Wings also does not list a Normals or texture count so a partial count loop is needed.

Do normals stored in obj files really matter? I think TDW's editor can calculate them if they are not found in the imported files, and Silent3ditor has even an option for ignoring them. When working on dat objects for SHIII I couldn't spot any visual difference. Maybe the game calculates them on the fly if no normal vectors are stored on file (?) :hmm2:

[Jeff-Groves]

I'd have to play with TDW's importer again to answer that.
I know it does the Tangent and BiNormal stuff.
I recommend seeing if Wings can export with just a 6 precision after the decimal point though. GR2 files only go to that.
SH3 doesn't do Normals except under special things. No Units actually use them.
SH4 does in SOME areas.

[Jeff-Groves]

OK. Let's talk about hard edges.
What is a hard edge? I'll start with how SH3 and SH4 handles them.
Say you make a Box. You NEED to have each side as a separate object, so to speak, to avoid the Games built in smoothing. That's been a well known for years.
How do you do that? In 3D Max I select the face for a wall and detach.
That produces a separate object as I suspect the Blender option to rid hard edges does.

I'm not sure if GR2 is smart enough to do that on it's own.

[gap]

I'd have to play with TDW's importer again to answer that.
I know it does the Tangent and BiNormal stuff.

Since in the 3D space normals, binornals and tangents are orthogonal to each other, only two of them should be needed. Calculating/storing the third vector on file would be pleonastic. Isn't it?


I recommend seeing if Wings can export with just a 6 precision after the decimal point though. GR2 files only go to that.

Unfortunately I don't see any export option for rounding numbers to only six decimal digits. Those are all the options in Wings3D's obj export dialog:

https://i.imgur.com/Oz8HFSY.png

If it is important, I can ask in Wings3D's forum if there is a way to do that.


SH3 doesn't do Normals except under special things. No Units actually use them.
SH4 does in SOME areas.

That might explain why SHIII/IV's shading looks so flat compared to SH5...

[Jeff-Groves]

It's not important to limit the output since the TDW importer does round it down.
Not even sure He actually checks Normals from the Main object file to the AO object file.

[gap]

OK. Let's talk about hard edges.
What is a hard edge? I'll start with how SH3 and SH4 handles them.
Say you make a Box. You NEED to have each side as a separate object, so to speak, to avoid the Games built in smoothing. That's been a well known for years.

Exactly!


How do you do that? In 3D Max I select the face for a wall and detach.
That produces a separate object as I suspect the Blender option to rid hard edges does.

That's the hard way, but depending on the model complexity this process can take a long time and it is not perfect. I give you an example. Let's say that we want to model a prism with a tear shaped base. Only one of its longitudinal edges is supposed to be hard, but there is no way you can have only one hard edge by detaching faces manually. This is where Blender Edge Split modifier comes in handy. It can detach even a single edge within an otherwise perfectly smooth surface, and it does that in one click :yep:

[gap]

It's not important to limit the output since the TDW importer does round it down.
Not even sure He actually checks Normals from the Main object file to the AO object file.

I can't say for sure, but since the only use of the AO object is providing a secondary UV map, it makes sense that only UV coordinates are read from file. All the other information should be ignored. It is also my understanding that TDW's GR2 importer assumes faces in the AO object being in the same order as in the main object. Thence the buggy AO mapping if face order is not exactly the same in the two objects.

I know 3dsMax supports multiple UV sets for the same object. This is the best way to ensure that, once exported, main and AO objects are identical in all the respects but UV coordinates, and I bet devs exploited this feature to ease their job.

[gap]

Yesterday I have had an insight into a couple of files, and I definitely see some problems.

This is a quick comparison of the two files:

https://i.imgur.com/nRX8aua.png

Vertices (v)

As expected, 3D vertex coordinates are identical in number, order and value between the two files.


Texture vertices (vt)

For each file, there are more UV vertices than 3D vertices. This is expected because when you unwrap a closed surface for texturing it, you need to "cut" it in pieces so to make it as flat as possible and to avoid texture distortions. As a consequence texture vertices along the boundaries of "cropped" UV regions are duplicated.


There is a mismatch in number of texture vertices between the two files, the main object having a bigger number of UV coordinates than its UV2 copy. This is probably not good for TDW's importer, and it might be the consequence of me cutting further some UV regions when creating the main UV map from the AO one.


Vertex normals (vn)

Vertex normals are identical in number, order and value between the two files. Again, this was expected as the main and AO objects have the same 3D geometry.


There are more vertex normals than vertex coordinates. That should be the consequence of my models having hard edges in place. Since the vertices along those edges are considered as part of two or three surfaces (each pointing in its own direction) rather than one smooth surface, there is a need for the shared vertices to have more than one normal.
If I had removed hard edges before exporting the models, I am sure that the number of normals would have equated the number of vertices, and if I split the hard edges (as I did with the Blender exports), there would be as many vertices as the current number of normals.

Faces (f)


There is an equal number number of face definitions between the two files. Again, this is the direct consequence of the two objects being identical in their 3D geometry, thus having the same number of faces.


In a triangulated model each face definition is composed of three triplets of numbers, each triplet corresponding to one of the three vertices composing each face. Within each triplet, the first number should be the index to each of the 3D vertices connected to form a triangle; the central member of each triplet should be for texture vertices, and the third one for vertex normals.



Between my two models, there is a partial mismatch in face order relative to the first (v) and third (vn) indices of each triplet. This mismatch might be caused by the fact that the main model has three materials assigned - one for the main texture, one for the railing and one for the rigging - whereas the AO model got only one material. Ungrouping materials in the AO model might have scrambled face order relative to the main model, since faces of the latter model are ordered on file depending on the material they use.


There is a total mismatch of texture vertex indices, i.e. the second numbers in each triplet of face definitions. This mess might have been caused by the removal of "redundant" materials from the AO model (as above), but also by the cutting of some edges in the UV map of the main model (as explained before, there are more vt coordinates in the main model than there are in the AO model).



If Almagest could obviate to my mistakes, sorting out texture coordinates and faces in one click, that would be terrific. I realize that setting up the needed automated routines might be more complicated than it sounds though. Alternatively, I could try to manual edit my AO model in the hope to correct at least in part the problems above.
It that doesn't work, redoing the main UV mapping on the base of the AO model is still an option. That should not be a complicated task either. A little boring but not too complicated. For this particular task I think I would resort to Blender.
Blender's advantage is that, like Max, it supports multiple UV sets, and it got advanced and configurable shader setting. Very useful for previewing the combined effect of diffuse, specular, normal and AO maps. Unfortunately I am not as proficient in Blender as I am in Wings3D, but learning it better would be an investment for future projects.

[Jeff-Groves]

All that stuff gets sorted automagically.
:D

The nice thing about how Granny stores the files is that We can INSERT stuff that does NOT load! Extract a 3D model from a GR2 file and look at it!
You'll see repeats of verts or textures that NEVER actually get called by any Face!

So I balance the counts to match the largest of the input.
In your case? V and VT will equal VN.
The faces will be sorted which moves the VT placement of textures in the obj file.
Thus one set of Faces that point to properly sorted information!
Now since We NEVER call the extra information? It never gets loaded!

I did try to point all this out to TDW several times. But I'm not sure he caught the idea so went with his way of doing the Loose Import.
That, as We all know, causes HUGE increases in file size. Not only that but EACH line needs to be loaded to display properly in Game.

[gap]

All that stuff gets sorted automagically.
:D

I like your automagic lol :D


The nice thing about how Granny stores the files is that We can INSERT stuff that does NOT load! Extract a 3D model from a GR2 file and look at it!
You'll see repeats of verts or textures that NEVER actually get called by any Face!

So I balance the counts to match the largest of the input.


In your case? V and VT will equal VN.

Well, in the case we are discussing, vn's are not an actual problem.
As I said, as far as hard edges are removed or split before GR2 import, vn's count will match v's count :yep:


The faces will be sorted which moves the VT placement of textures in the obj file.
Thus one set of Faces that point to properly sorted information!

Doing this shouldn't require the addition of any extra information either. The good thing about my otherwise buggy AO model is that its vertices are in the same order as in the AO model, so we can easily track down which vt corresponds to which v in both files and, from there, which vt in one file corresponds to which vt in the other file. What is actually needed is sorting vt coordinates in the AO object so that they are in the same order as used by face definitions in the main model. All the other lines in the AO object can be identical to the ones of the main model.

I actually thought of a way for doing that in Excel, but it will work only for this specific set of files. Doing that spreadsheet generic would involve more work than it is probably worth. Nonetheless, while you work on a proper object sorter, I might give my stop-gap Excel tool a try... I am just curious to see my model in Blender with the two UV maps and connected textures applied, but I can't import the secondary UV map until the mismatch is fixed... (and I can't create a decent LOD model).



I did try to point all this out to TDW several times. But I'm not sure he caught the idea so went with his way of doing the Loose Import.
That, as We all know, causes HUGE increases in file size. Not only that but EACH line needs to be loaded to display properly in Game.

Well, we should admit that these concepts are not so easy to understand even if someone explains them to you one hundred times. Moreover one I can grasp the basics of someone else's discoveries, but putting them in practice would be another matter. TDW is brilliant at math, programming and debugging. He demonstrated his talents in many ways, but probably 3D geometry wasn't his piece of cake. In all honesty I can't blame him for that. I myself have a hard time trying to figure out the public domain and well known waveform format, le alone the granny one lol :doh:

P.S: a big limit to performing only strict import with TDW's tool, will be vertex count. I am not even sure I can fit my puffer in the template given by the stock fishing boat...

[Jeff-Groves]

I'm not criticizing TDW's work. It's BRILLIANT!
And the whole thing is kind of complicated if one does not have a full understanding of both the GR2 file storage and how obj files can be manipulated to match the GR2 format.

I'll give you a quick example you can do.
Cut say half the Faces in any object file and paste them to the top of the faces section of any obj file.

So if you have 500 faces? Cut 250 and paste them at the top so you still have all 500.

Then copy and paste say half your verts at the bottom of the verts section.
Open the file in any 3D program.
:haha:

[Jeff-Groves]

If you read back through this thread?
i've already told you HOW to adjust file sizes!
Use the Loose import to grow or shrink things!

It's going to take some work to determine how big a file to import under the loose setting to shrink or grow but it does work!

[gap]

I'm not criticizing TDW's work. It's BRILLIANT!
And the whole thing is kind of complicated if one does not have a full understanding of both the GR2 file storage and how obj files can be manipulated to match the GR2 format.

I'll give you a quick example you can do.
Cut say half the Faces in any object file and paste them to the top of the faces section of any obj file.

So if you have 500 faces? Cut 250 and paste them at the top so you still have all 500.

Then copy and paste say half your verts at the bottom of the verts section.
Open the file in any 3D program.
:haha:

To the best of my understanding of the obj format, face order within a mesh doesn't really matter (it matters in our case though), and the redundant vertices would be isolated vertices. The obj format doesn't support isolated vertices though. Depending on the program they might be ignored or the might show up, but they would be considered a topological error. Probably they wouldn't survive after a save/reload cycle.

If you read back through this thread?
i've already told you HOW to adjust file sizes!
Use the Loose import to grow or shrink things!

It's going to take some work to determine how big a file to import under the loose setting to shrink or grow but it does work!

Good to know! I had missd that piece of information, sorry :oops:

[Jeff-Groves]

Redundant information is ignored with most programs.
And you can see that by exporting from a GR2 and then saving again in a program if you have optimize on save.

Now how can you have 16150 faces but still have 25665 V, VT, VN?
That's because the GR2 files SKIP some garbage information!

It's simple math! 9515 lines are ignored.
And that is directly from the NAGC_C2Appalachian GR2!

I'm exploiting the way Granny works! I don't give a crap about Wings, Blender, nor 3D Max and how they may react once the files are run through Almagest.

[gap]

Redundant information is ignored with most programs.
And you can see that by exporting from a GR2 and then saving again in a program if you have optimize on save.

Now how can you have 16150 faces but still have 25665 V, VT, VN?
That's because the GR2 files SKIP some garbage information!

It's simple math! 9515 lines are ignored.
And that is directly from the NAGC_C2Appalachian GR2!

I'm exploiting the way Granny works! I don't give a crap about Wings, Blender, nor 3D Max and how they may react once the files are run through Almagest.

Okay, looking forward to your progress with the automagic, almighty Almagest :) :up:

[Jeff-Groves]

Nice thing about the AO mapping is that it don't care about multiple textures for the main object file.
Now Almagest will determine WHICH faces structure it needs to use.
Does it use the AO faces structure or the Main Object faces structure?
It needs to scan the files, see if it's multi textured and choose the right way to sort things.

A single mapped main object? It may be better to use the AO face structure.
Multi mapped main object? Probably better to use that face structure THEN maybe fake some Faces.

It's kind of like the Street Game where you try to follow the balls under cups!
Everything has to be manipulated at just the right time, in the right order, to pass for magic!

[gap]

Nice thing about the AO mapping is that it don't care about multiple textures for the main object file.
Now Almagest will determine WHICH faces structure it needs to use.
Does it use the AO faces structure or the Main Object faces structure?
It needs to scan the files, see if it's multi textured and choose the right way to sort things.

A single mapped main object? It may be better to use the AO face structure.
Multi mapped main object? Probably better to use that face structure THEN maybe fake some Faces.

It's kind of like the Street Game where you try to follow the balls under cups!
Everything has to be manipulated at just the right time, in the right order, to pass for magic!

In the meanwhile I finished processing my AO model so that its texture vertices are sorted in the same way as in the main model.

First, I sorted AO faces so that they matched faces in the main obj file. For accomplishing this task I set my spreadsheet to compare vertex triplets in both files, as I knew that one triplet in one file corresponded with one triplet, and only one, in the other file.
Matching face order in both files gave me information on which UV2 vt corresponded with which vt in the main object (the second numbers in each triplet), so I updated vt's in the AO model according to the order used in the main model.

Now I need to copy the sorted AO vt's in main model's obj files, and I need to check in a 3D editor if I didn't make any mistake. If, as I hope, everything is in order I will send to you the new UV2 file so you can test your program on it too :salute:

[gap]

Now I need to copy the sorted AO vt's in main model's obj files, and I need to check in a 3D editor if I didn't make any mistake. If, as I hope, everything is in order I will send to you the new UV2 file so you can test your program on it too :salute:

It works!

There are only a few errors in the "new" UV2 map which I mostly know how to correct :up:

[Jeff-Groves]

There you go!
You looked behind the curtain and saw the Magic.
:har:
https://www.youtube.com/watch?v=ubIpoPjBUds&t=28s

Now you can hand edit the rest of the file to be proper for GR2 format.
Just copy and paste V or VT or VN's to MATCH which ever is highest in counts!
You could also just add with 0's. So V 0.0 0.0 0.0 will work. Your just filling space.
Since no FACE calls them? You'l never see them!



I do suggest using Blender files as they are way smaller but produce the same results! Plus you get the modifier you like.

Not sure WHY the Wings files have so many Normals compared to the Blender files?
:hmmm:

[Jeff-Groves]

Well. Did some code modifications and just could not get the proper out put!
Spent several days and wasted those days!

Seems I Forgot that the basic code looks for a certain file in a very specific location!
:/\\!!

I haven't added the UI to select a file yet.
:oops:

Once I placed the file and named it properly? It worked perfectly.

I've also to add a routine so one can select if this is for TDW's SH5 import or Skywas's S3D import.
There are different things to consider for each.

[Jeff-Groves]

I did move the Code to Visual Studio 2010 Express C++.
Not sure if that is a smart move as yet but We will see.

[Mister_M]

So, things are nearly finished ? :D

[gap]

There you go!
You looked behind the curtain and saw the Magic.

Hahahah, as of now I did not see any magic, sorry, but I am trying to figure out the logic and I must add that, complicated as it is with all those v, vt, vn and f lines to track down, most of what I see in the obj format finally makes sense to me.

Now I know that everything must be in the right order for us to be sure that UV2 coordinates are imported correctly in game. I know what I did wrong in the past and how to avoid those mistakes. Even so, the error is around the corner. A single, little mistake can mess up a whole UV map. For this reason, a program tracking down the most common causes of secondary UV map corruption, and correcting them where possible, would still be a valid help :yep:

From this perspective, Blender is an useful editing and diagnostic tool, as it allows us to import the UV map from one model to another identical model. If the two models have the same geometry but v/vt sorted in a different order, it will be immediately apparent that the imported UV map is buggy, and so it will be in game. I think kapuhy exploited this feature, and this is why he never had similar problems importing his AO-mapped units in game.
Nonetheless, Blender can't debug wrong AO models created with other programs, and this is where Almagest might show its AutoMagic powers :|\\ :D


I do suggest using Blender files as they are way smaller but produce the same results! Plus you get the modifier you like.

That's something I have noticed too. I believe Belnder to store the same information as Wings3D, but in a more "compact" way. That shouldn't have any effect on GR2 file size though.

There is a con to Blender-exported obj files though. If memory serves, GR2 Editor doesn't recognize one of the ASCII character it uses as line separator. If you open one of those files in Mircrosoft notepad you will immediately notice the difference as all the lines are shown one after the other, as if they were in the same paragraph. For GR2 Editor to read those files one needs to export/re-export the files in another program, or Notepad++ must be used to replace one character with the other. Kapuhy might know more on this topic.


Not sure WHY the Wings files have so many Normals compared to the Blender files?
:hmmm:

If you are referring to the files I sent you the answer is easy: I think my Blender exports have as many normals as vertices because I split hard edges before exporting. Provided that this is the correct answer, Wings-exported files have not a bigger number of vertex normals compared to Blender exports, but the other way around: they have a lesser number of vertices.

[Jeff-Groves]

On the differences of Wings and Blender files.

The hexadecimal 0a, a control character as opposed to a printing character, is called a line feed. The hexadecimal 0d is called a carriage return.
Pretty much all the programs on the Windows platform understand and expect the hexadecimal 0a0d pair in text.
The 0d0a pair of characters is the signal for the end of a line and beginning of another.

Almagest accounts for that where TDW may not have caught that based on what files he was working with as test files.
I've ran Wings files, Blender files, and 3d Max files through the basic code and all report proper out put.

[gap]

So, things are nearly finished ? :D

Please, check my PM :03:

On the differences of Wings and Blender files.

The hexadecimal 0a, a control character as opposed to a printing character, is called a line feed. The hexadecimal 0d is called a carriage return.
Pretty much all the programs on the Windows platform understand and expect the hexadecimal 0a0d pair in text.
The 0d0a pair of characters is the signal for the end of a line and beginning of another.

Yes, exactly that!

I was looking for the right words, but I missed them. Sorry for my bad phrasing and thank you for correcting it.


Almagest accounts for that where TDW may not have caught that based on what files he was working with as test files.
I've ran Wings files, Blender files, and 3d Max files through the basic code and all report proper out put.

:up:

[Jeff-Groves]

One thing I am doing is reading each line as a string. Not each section of that string like one would to import the obj format.

I'm not changing that information so strings should stay as in the obj file.
All I'm doing is "Shuffling the cards" so to speak.

Now that does involve "Stacking the deck" in areas but it works for our needs.
:haha:

[Mister_M]

Yesterday I have had an insight into a couple of files, and I definitely see some problems.

This is a quick comparison of the two files:

https://i.imgur.com/nRX8aua.png

Vertices (v)

As expected, 3D vertex coordinates are identical in number, order and value between the two files.


Texture vertices (vt)

For each file, there are more UV vertices than 3D vertices. This is expected because when you unwrap a closed surface for texturing it, you need to "cut" it in pieces so to make it as flat as possible and to avoid texture distortions. As a consequence texture vertices along the boundaries of "cropped" UV regions are duplicated.


There is a mismatch in number of texture vertices between the two files, the main object having a bigger number of UV coordinates than its UV2 copy. This is probably not good for TDW's importer, and it might be the consequence of me cutting further some UV regions when creating the main UV map from the AO one.


Vertex normals (vn)

Vertex normals are identical in number, order and value between the two files. Again, this was expected as the main and AO objects have the same 3D geometry.


There are more vertex normals than vertex coordinates. That should be the consequence of my models having hard edges in place. Since the vertices along those edges are considered as part of two or three surfaces (each pointing in its own direction) rather than one smooth surface, there is a need for the shared vertices to have more than one normal.
If I had removed hard edges before exporting the models, I am sure that the number of normals would have equated the number of vertices, and if I split the hard edges (as I did with the Blender exports), there would be as many vertices as the current number of normals.

Faces (f)


There is an equal number number of face definitions between the two files. Again, this is the direct consequence of the two objects being identical in their 3D geometry, thus having the same number of faces.


In a triangulated model each face definition is composed of three triplets of numbers, each triplet corresponding to one of the three vertices composing each face. Within each triplet, the first number should be the index to each of the 3D vertices connected to form a triangle; the central member of each triplet should be for texture vertices, and the third one for vertex normals.



Between my two models, there is a partial mismatch in face order relative to the first (v) and third (vn) indices of each triplet. This mismatch might be caused by the fact that the main model has three materials assigned - one for the main texture, one for the railing and one for the rigging - whereas the AO model got only one material. Ungrouping materials in the AO model might have scrambled face order relative to the main model, since faces of the latter model are ordered on file depending on the material they use.


There is a total mismatch of texture vertex indices, i.e. the second numbers in each triplet of face definitions. This mess might have been caused by the removal of "redundant" materials from the AO model (as above), but also by the cutting of some edges in the UV map of the main model (as explained before, there are more vt coordinates in the main model than there are in the AO model).



If Almagest could obviate to my mistakes, sorting out texture coordinates and faces in one click, that would be terrific. I realize that setting up the needed automated routines might be more complicated than it sounds though. Alternatively, I could try to manual edit my AO model in the hope to correct at least in part the problems above.
It that doesn't work, redoing the main UV mapping on the base of the AO model is still an option. That should not be a complicated task either. A little boring but not too complicated. For this particular task I think I would resort to Blender.
Blender's advantage is that, like Max, it supports multiple UV sets, and it got advanced and configurable shader setting. Very useful for previewing the combined effect of diffuse, specular, normal and AO maps. Unfortunately I am not as proficient in Blender as I am in Wings3D, but learning it better would be an investment for future projects.

What I can tell you for sure is that if the main and the UV2 model are identical in all the respects except vt coordinate values, there is a 100% probability of success. With this idea in mind I have created an Excel spreadsheet which sorts vt coordinates for us. It is a rather crude tool, it requires a lot of manual data filtering, cutting and pasting; any mistake will cause wrong results, but if used correctly it does its job just fine.

If you are interested into testing it, I can upload it somewhere and post its link in the Almagest thread where it belongs more than here.

Couldn't be the problem coming from the fact that sub-objects (composing the final exported "diffuse object") are combined in a different order than in the "AO object" ?

If your excel sheet can solve the problem, let's try it. :up: Just explain to me how this works. :doh:

[Jeff-Groves]

I know from reading the other thread that a question was raised about how the information on mapping is stored in a dat file.
A dat with only 1 map channel is stored in a simple block of data.
When We add say an AO channel? A TMAP section is added to the dat.
I'll have to refresh my memory on how the TMAPs actually work since it's been so long since it had to be done by hex editing. (Pre 2007!)
BUT! Only faces and texture coords are saved to the TMAP sections if I am recalling correctly.

I actually still have a program from 'back in the day' that exports and imports the different TMAP channels. So one could change or add just a TMAP and not have to re-import the complete obj like S3D does.

https://www.subsim.com/radioroom/picture.php?albumid=1069&pictureid=12276

[Jeff-Groves]

One thing I don't do is try to 'Guess' what went wrong.
Send me the files and I'll look at them.
I've probably forgotten more then most people know.
Not trying to be a smarmy butt head but I have been around the block more then once.

[Mister_M]

One thing I don't do is try to 'Guess' what went wrong.
Send me the files and I'll look at them.

Here are .obj files which don't render correctly in game with the AO map : https://www.mediafire.com/file/ut1sswyfu0e3xgf/Objects_with_broken_AO_map.zip/file

We are trying to understand why...

[gap]

Couldn't be the problem coming from the fact that sub-objects (composing the final exported "diffuse object") are combined in a different order than in the "AO object" ?

Possible, but that's not the main suspect in my list of possible culprits.

What for sure can mess up the order of sub-objects (and faces) is material assignation. The main model and the AO model should have materials with similar naming (Wings3D sorts them in alphabetical order) assigned to corresponding faces. If, for example, a group of faces in the main model is given a material whose name is let's say 'wood', the equivalent faces in the AO/UV2 object should be given a material named 'wood_AO', 'wood_2', or something similar.


If your excel sheet can solve the problem, let's try it. :up: Just explain to me how this works. :doh:

Yes, it can. As I said, by using the output of my spreadsheet we can create an AO/UV2 obj file which is the exact copy of the main obj file, exception made for the values of texture coordinates.

Before I upload it, I need to be sure that it has enough rows for processing your model. Can you tell me the number of f and vt lines in the obj files you are processing? You can easily count them by pasting their content in an empty Excel document and by using the data filtering options or the COUNTIF function.


I'll have to refresh my memory on how the TMAPs actually work since it's been so long since it had to be done by hex editing. (Pre 2007!)
BUT! Only faces and texture coords are saved to the TMAP sections if I am recalling correctly.

Thank you for your inputs Jeff!

Does that mean that faces and texture coordinates can be in a different order in the secondary map channel than in the primary one?


I actually still have a program from 'back in the day' that exports and imports the different TMAP channels. So one could change or add just a TMAP and not have to re-import the complete obj like S3D does.

https://www.subsim.com/radioroom/picture.php?albumid=1069&pictureid=12276

in future this tool might come in handy :up:

[Jeff-Groves]

Well that was easy.
I compared the main obj and the AO obj in 010.
Your vertices do not match between the Main Hull and AO Hull.
(I suspected that but as I said. I'll not play a guessing game)
So anything else is going to be All screwed up on import!

What S3D does when importing the Main diffuse object and the AO object is this!

IF number counts are the same for the vertices? It ignores what those are in the AO object and only imports the faces and texture coords!

So basically S3D IS importing garbage you created.
That is the flaw AND the opening in S3D that I am working to fix!

[Mister_M]

Can you tell me the number of f and vt lines in the obj files you are processing?

Hull1.obj :
2018 f
868 vt

Hull1-uv2.obj :
2018 f
2729 vt

It already seems problematic... :D

Well that was easy.
I compared the main obj and the AO obj in 010.
Your vertices do not match between the Main Hull and AO Hull.

Ah ! You replied while I was writting this...

[QUOTE=Jeff-Groves;2788565]So basically S3D IS importing garbage you created.

Sorry, you mean : garbage that Wings3D created... :D

That is the flaw AND the opening in S3D that I am working to fix!

You are trying to fix garbage created by Wings3D when exporting ?

[Jeff-Groves]

Well. In the dat file format you CAN HAVE different counts on that information!
But the VERTS MUST BE EXACTLY THE SAME!

I'd suspect Wings re-ordered the Verts placement in the exported obj file thus faces and textures are screwed.

And You are the one that used Wings. It didn't to it by itself!
LOL!

[Mister_M]

Well. In the dat file format you CAN HAVE different counts on that information!
But the VERTS MUST BE EXACTLY THE SAME!

I'd suspect Wings re-ordered the Verts placement in the exported obj file thus faces and textures are screwed.

And You are the one that used Wings. It didn't to it by itself!
LOL!

I don't create garbage by myself... :rotfl2:

[Jeff-Groves]

Just playing with you Mate.
:har:

It is strange to many that somethings do not work as expected with programs like S3D or TDW's program for SH5.

Both missed things and based their codes on false assumptions.
Sometimes that was a Ego thing on their parts to NOT listen to suggestions.
Sometimes it was a lack of understanding EXACTLY how the import target is constructed!

S3D used ALOT of information from the Grey Wolves in it's construction.
TDW's program aslo credits a Grey Wolf as soon as you open it!
(Guess WHO privateer is!)

[Mister_M]

Just playing with you Mate.
:har:

:up:

Guess WHO privateer is!

YOU

:Kaleun_Smile:

https://www.youtube.com/watch?v=EEFsJ-43pnA

Sorry... :D :haha: :har: :rotfl2:

[Jeff-Groves]

I may be the LAST Wizard left here at SubSim.
:haha:

I look at the files in a way that others do not.
With nothing but 010 Hex editor? I opened and changed the Bunker so one can walk all around!

I'm probably the ONLY one left active that can change things and create a better future for other Modders.
I am in debt to the ones that taught me.

[gap]

Well that was easy.
I compared the main obj and the AO obj in 010.
Your vertices do not match between the Main Hull and AO Hull.

If you are talking about my puffer model vertices in the main model do match vertices in the AO model, confirmed by WimMerge comparison and by the fact that my spreadsheet (which supposes an exact match in vertex order) generates a correct output :yep:

I think you meant texture vertices. Didn't you?


IF number counts are the same for the vertices? It ignores what those are in the AO object and only imports the faces and texture coords!

So basically S3D IS importing garbage you created.
That is the flaw AND the opening in S3D that I am working to fix!

in an obj file 3D faces are denoted by the vertices composing them. So in a triangulated model you need three numbers to denote a face, e.g. 3 31 15 (this is: vertices # 3, 31 and 15 in the list of vertices are connected to compose a face).

This is for non-UV-mapped objects. For UV-mapped objects you must add three more numbers. Following the previous example: 3/10 31/7 15/25.
This line tells us that vertices #3, #31 and #15 in the vertex list are connected to compose a face in the 3D space, and that vertices #10, #7 and #25 in the texture vertex list compose a face in the UV space, where vt #10 corresponds to v #3, vt #7 to v #31 and vt #25 to v #15.

The errors I and Mister_M have been experiencing lead me to think - but this is only my guessing - that face definitions are written in dat/gr2 files only once, that secondary texture coordinates are stored in the same order as main UV coordinates, and that neither S3d nor GR2E read all the information stored in secondary obj file's face definitions in order to sort texture coordinates appropriately.

[gap]

Hull1.obj :
2018 f
868 vt

Hull1-uv2.obj :
2018 f
2729 vt

It already seems problematic... :D

Oh my :doh:

I was expecting some unbalance in vt count between the two files, but such a huge unbalance was really unexpected.

Did you mess with secondary UV map? I mean: cutting edges which were originally joined or vice-versa?

And since you are at it, can you check that vertices (v lines) are sorted in the same order in the UV2 obj file as in the main obj file? That's a quick comparison in WinMerge :)

[kapuhy]

Oh my :doh:

I was expecting some unbalance in vt count between the two files, but such a huge unbalance was really unexpected.

If its my tramp steamer MisterM was writing about, this might also be my "fault" for using "lightmap pack" function for smaller faces to avoid problem I had where small and long faces like ropes and railings were omitted in AO baking. Lightmap pack uv-maps faces as separate rectangles in 2d space so there would be many more edge cuts in ao uvmap than in diffuse uv map.

[Mister_M]

Oh my :doh:

I was expecting some unbalance in vt count between the two files, but such a huge unbalance was really unexpected.

:/\\!!

Did you mess with secondary UV map? I mean: cutting edges which were originally joined or vice-versa?

No

And since you are at it, can you check that vertices (v lines) are sorted in the same order in the UV2 obj file as in the main obj file? That's a quick comparison in WinMerge :)

I guess I have to copy/paste all the lines into 2 .txt files ?
EDIT : So, order is different, a long block (nearly the first 1/3) has been put at the end...

If its my tramp steamer MisterM was writing about

Yes

[Jeff-Groves]

I guess I'm gonna need some crayons to explain things.
:har:

We ARE talking about dat file import of the files Mister_M sent me.
I have not seen the ORIGINAL FILES! Just what I got today.

In a DAT file? number of texture coords and faces does NOT matter!
Unlike SH5 where it DOES matter! (Thus the addition of false info to balance a strict import)

In Dat files, like GR2 files, ALL verts in ALL files MUST MATCH!

The files I have show that the COUNT of verts is fine but the placement does not match between the main and AO obj files.
Thus the Faces are off! Hell! For S3D and TDW's Tool? You could zero every Vert in the AO and as long as counts match? It will import!
Because they DO NOT read and save the verts in the AO obj!

Send me the ORIGINAL files.

[Mister_M]

Send me the ORIGINAL files.

Well, Kapuhy sent me the obj files of his nice tramp steamer. If Kapuhy doesn't mind, I can send you his D/L link.

If not imported/exported via Wings3D, it seems that there isn't any glitch with AO-map in game.

[gap]

I guess I'm gonna need some crayons to explain things.
:har:

:rotfl2:

If its my tramp steamer MisterM was writing about


We ARE talking about dat file import of the files Mister_M sent me.
I have not seen the ORIGINAL FILES! Just what I got today.

Sorry for the misunderstanding guys, I wasn't aware that Mister_M had already sent his own version of kapuhy's steamer to Jeff :oops:


this might also be my "fault" for using "lightmap pack" function for smaller faces to avoid problem I had where small and long faces like ropes and railings were omitted in AO baking. Lightmap pack uv-maps faces as separate rectangles in 2d space so there would be many more edge cuts in ao uvmap than in diffuse uv map.

Okay, that explains the huge difference in vt coordinate count between the two files. It also proves that I was wrong in thinking that GR2 editor does not read UV face information from the AO object file. If it didn't, the unbalance/different order of vt lines between the two files would have meant a bad secondary UV map, but this is obviously not the case :up:


I guess I have to copy/paste all the lines into 2 .txt files ?
EDIT : So, order is different, a long block (nearly the first 1/3) has been put at the end...

I am sorry to say that this is a huge problem. One of the prerequisites of my spreadsheet is vertices in the main and in the UV2/AO objects to be in the same order. I could identify and sort vertices according to their coordinates. This method would work for simple objects, but I could not easily discern vertices with the same coordinates, a rather common occurrence with edge-split models. In theory I could discern overlapping vertices by checking which faces they are part of, but that would be an overly complicated method; before I come up with a semi-decent routine, you could redo ten times your changes on kapuhy's model :yep:

My suggestion is to switch to Blender for final model editing. This program supports object multi-mapping, i.e. the assignation of multiple UV maps on the same object. You can even import an UV map from one object to another identical object, so whatever changes you do to the model, you are sure that the two exported obj files will be identical in vertex/face structure.
Else, if you want to stick to Wings3D, be more careful. You want to ungroup the main model? Immediately after, ungroup the AO model too. You want to regroup a group of sub models? Make sure that you regroup exactly the same objects for both models. Your actions on each model should be specular and you should perform them exactly in the same order for having a chance of success. :yep:



In a DAT file? number of texture coords and faces does NOT matter!
Unlike SH5 where it DOES matter! (Thus the addition of false info to balance a strict import)

In Dat files, like GR2 files, ALL verts in ALL files MUST MATCH!

The files I have show that the COUNT of verts is fine but the placement does not match between the main and AO obj files.
Thus the Faces are off! Hell! For S3D and TDW's Tool? You could zero every Vert in the AO and as long as counts match? It will import!
Because they DO NOT read and save the verts in the AO obj!

It all makes sense :up:

[Mister_M]

I have checked the original Hull1.obj and Hull1_AO.obj of Kapuhy's model.

- in main model = vt = 2102
- in AO model : vt = 13772

- v lines are identical and in the exact same order

No glitch in game.

[gap]

I have checked the original Hull1.obj and Hull1_AO.obj of Kapuhy's model.

- in main model = vt = 2102
- in AO model : vt = 13772

- v lines are identical and in the exact same order

No glitch in game.

Good news! That means that texture coordinates' count/ordering is not critical for UV2 import neither in granny models nor in .dat ones :up:

[kapuhy]

I have checked the original Hull1.obj and Hull1_AO.obj of Kapuhy's model.

- in main model = vt = 2102
- in AO model : vt = 13772

Wow. I might look for another way to make AO texture bake in future :)

Through in SH5 Goblin editor, it shows Armora having 69k verts and 28 k tris total while (comparable in size) stock N3SA1 is 27k verts and 11 k tris, so i'd say proportion is similar.

EDIT: btw, Mister_M, if you'd like to make some changes to model in Blender I can send you .blend files so there's less importing/exporting.

[gap]

Wow. I might look for another way to make AO texture bake in future :)

Why don't you bake your AO textures in Mod Tools (i.e. the free version of Autodesk Softimage)?
I started using that program after Jeff's advise. Its texture baking utility features a lot of useful options for customizing the output. They can be a bit confusing at the beginning, but imo they are worth the effort of learning them.
On medium settings, my bakes never missed a single texel even on very minute parts, but I must add that I always unwrap my models manually.

[kapuhy]

Why don't you bake your AO textures in Mod Tools (i.e. the free version of Autodesk Softimage)?

Never knew such a thing existed :) Thanks for heads up, will check it out.

[gap]

Never knew such a thing existed :) Thanks for heads up, will check it out.

My pleasure. Let me know if you need any help with it. If you want I can also tell you my favourite settings. Depending on model complexity and final pixel resolution a good bake might take 20 to 40 minutes, but if you have a more modern computer it might take even lesser :salute:

[Mister_M]

So, finally, where does the glitch I'm experiencing come from ? Only a difference in the order of vertices between main and AO objects ?

Mister_M, if you'd like to make some changes to model in Blender I can send you .blend files so there's less importing/exporting.

My suggestion is to switch to Blender for final model editing. This program supports object multi-mapping, i.e. the assignation of multiple UV maps on the same object. You can even import an UV map from one object to another identical object, so whatever changes you do to the model, you are sure that the two exported obj files will be identical in vertex/face structure.

I don't have Blender installed, and don't know how to use it.

Else, if you want to stick to Wings3D, be more careful. You want to ungroup the main model? Immediately after, ungroup the AO model too. You want to regroup a group of sub models? Make sure that you regroup exactly the same objects for both models. Your actions on each model should be specular and you should perform them exactly in the same order for having a chance of success. :yep:

:up:

[gap]

So, finally, where does the glitch I'm experiencing come from ? Only a difference in the order of vertices between main and AO objects ?

Yes, at this point we can conclude that this is the main, and probably the one problem :yep:


I don't have Blender installed, and don't know how to use it.

I think no one knew how to use Blender before installing it and starting to mess with it :O:

At the beginning the learning curve can be step, but on the internet there are a lot of guides and video-tutorials covering even the most advanced of its functionalities. I myself am still learning them. I think I will stick to Wings3D as my main modelling tool because I feel more proficient with it, but there are a lot of cool things you simply can't do in Wings and you can do in Blender. Why giving them up? :03:

[Jeff-Groves]

So, finally, where does the glitch I'm experiencing come from ? Only a difference in the order of vertices between main and AO objects ?


Yes. And there is a VERY easy way to see this in Wings3D, Blender, 3DS Max, or any other 3D program.

Copy the Faces section in the AO obj file.
(Obj files are just text files!)

Paste that over the faces section of the main obj file then open it!

If it looks like garbage? It is Garbage!

For GR2 strict import? We need to sort the faces and texture coords.
For SH3/4? The faces and textures are stored in a different TMAP section!

[Mister_M]

So, in short, it's impossible to realise (unless checking everything with your trick) that AO objects are incorrect before seeing it in game... because S3D only checks if there are the same number of vertices between main and AO obj...

I think no one knew how to use Blender before installing it and starting to mess with it :O:

I was sure you were about to say something like that... :DL

[Jeff-Groves]

So, in short, it's impossible to realise (unless checking everything with your trick) that AO objects are incorrect before seeing it in game... because S3D only checks if there are the same number of vertices between main and AO obj...

Correct!
But you can just look at the first few lines of verts in each file and see that you have a problem.

[gap]

I was sure you were about to say something like that... :DL

Hahahah, I am predictable and you know me well. My point still stands though :03: :salute:

[Jeff-Groves]

I had to do some research but I'll explain how a 3D model is stored in a dat file.
I'd figured this out long ago but just some details I had forgotten.

I have a Template for 010 that I'd forgotten about so I used it to refresh my memory. I guess I'm getting old!
:o
Anyway! Here's how a dat stores the information and how I figured it out.

All verts are stored in a chunk of information easy to follow with my Template.
As are Faces and texture coords. And here's where it gets tricky!
Any extra texture coords, for say the AO mapping, get stored right along with the Main objects coords in one chunk of data.

That's where the TMAP sections come in! They point to the Verts and the texture coords in the main chunk of data! So they are nothing more then faces calling the proper lines from the main chunk of data.

S3D just puts out a proper obj file on export then on import?
Just stacks everything in it's proper place.

How I do it is to export a 3D model as a rawchunk with S3D then run my Template. It takes a few minutes to look at things and see that texture coords are GREATER then the same obj files exported by S3D.

Should anyone want to see the Template and the files I worked with let me know.

SH5 does it a bit differently.

[Jeff-Groves]

Now! back to the dat information!

There's one more small bit of information to know on reading the textures in hex!

Say I have this in the obj file.....
vt 2.548912764 0.04277324677

But my Template shows this!
float U_coord 2.548913
float V_coord 0.9572268

Don't make sense? yes it does! Subtract the float V_coord from 1!

I'll have to edit my Template to get the proper read out.
:/\\!!

I also have to Credit my Friend Anvart (Alex) for most of the Template.

[Jeff-Groves]

Like I said before. I'd forgotten more then most even know.
Going over the information in a dat file I remembered even more!
I'm working on fixing the Template to show EXACTLY what you'd see when exported with S3D.

One area is the faces and how they are stored in a dat file.
One needs to read the first part then read the second.

It's kind of crazy but here's what you see.
We read 3 Verts THEN read the 3 Textures!
So if a Face is 3/1 2/3 5/4?
You will read 3 2 5 in Hex THEN 1 3 4!

Now I know most don't give a crap about that kind of stuff. But I do for a reason!
I am working on an Exporter to rip 3D models from SH3/4 that ACTUALLY places the parts in their positions just as you see them in S3D or in Game!
There is NOTHING that does that!

[vickers03]

I am working on an Exporter to rip 3D models from SH3/4 that ACTUALLY places the parts in their positions just as you see them in S3D or in Game!

that would be awesome:salute:

[tonschk]

I am working on an Exporter to rip 3D models from SH3/4 that ACTUALLY places the parts in their positions just as you see them in S3D or in Game!
There is NOTHING that does that!

Amazing, thank you Jeff-Groves :salute::up::yeah:

[gap]

I am working on an Exporter to rip 3D models from SH3/4 that ACTUALLY places the parts in their positions just as you see them in S3D or in Game!
There is NOTHING that does that!

That would be nice :up:

Indeed, one can arrange model parts in their desired position by manually adding/subtracting node translation/rotation values, but performing this task on model of a certain complexity is a PITA because nested objects inherit position/rotation from their parent objects :doh: