SUBSIM Radio Room Forums - [WIP] Almagest. Strict Import for SH5.

SUBSIM Radio Room Forums (https://www.subsim.com/radioroom/index.php)

- SH5 Mods Workshop (https://www.subsim.com/radioroom/forumdisplay.php?f=249)

- - [WIP] Almagest. Strict Import for SH5. (https://www.subsim.com/radioroom/showthread.php?t=248714)

Quote:

Originally Posted by Mister_M (Post 2786691)

So, things are nearly finished ? :D

Please, check my PM :03:

Quote:

Originally Posted by Jeff-Groves (Post 2786761)

On the differences of Wings and Blender files.

The hexadecimal 0a, a control character as opposed to a printing character, is called a line feed. The hexadecimal 0d is called a carriage return.
Pretty much all the programs on the Windows platform understand and expect the hexadecimal 0a0d pair in text.
The 0d0a pair of characters is the signal for the end of a line and beginning of another.

Yes, exactly that!

I was looking for the right words, but I missed them. Sorry for my bad phrasing and thank you for correcting it.

Quote:

Originally Posted by Jeff-Groves (Post 2786761)

Almagest accounts for that where TDW may not have caught that based on what files he was working with as test files.
I've ran Wings files, Blender files, and 3d Max files through the basic code and all report proper out put.

:up:

One thing I am doing is reading each line as a string. Not each section of that string like one would to import the obj format.

I'm not changing that information so strings should stay as in the obj file.
All I'm doing is "Shuffling the cards" so to speak.

Now that does involve "Stacking the deck" in areas but it works for our needs.
:haha:

Quote:

Originally Posted by gap (Post 2786275)

Yesterday I have had an insight into a couple of files, and I definitely see some problems.

This is a quick comparison of the two files:

https://i.imgur.com/nRX8aua.png

Vertices (v)

As expected, 3D vertex coordinates are identical in number, order and value between the two files.

Texture vertices (vt)

For each file, there are more UV vertices than 3D vertices. This is expected because when you unwrap a closed surface for texturing it, you need to "cut" it in pieces so to make it as flat as possible and to avoid texture distortions. As a consequence texture vertices along the boundaries of "cropped" UV regions are duplicated.
There is a mismatch in number of texture vertices between the two files, the main object having a bigger number of UV coordinates than its UV2 copy. This is probably not good for TDW's importer, and it might be the consequence of me cutting further some UV regions when creating the main UV map from the AO one.

Vertex normals (vn)

Vertex normals are identical in number, order and value between the two files. Again, this was expected as the main and AO objects have the same 3D geometry.
There are more vertex normals than vertex coordinates. That should be the consequence of my models having hard edges in place. Since the vertices along those edges are considered as part of two or three surfaces (each pointing in its own direction) rather than one smooth surface, there is a need for the shared vertices to have more than one normal.
If I had removed hard edges before exporting the models, I am sure that the number of normals would have equated the number of vertices, and if I split the hard edges (as I did with the Blender exports), there would be as many vertices as the current number of normals.

Faces (f)

There is an equal number number of face definitions between the two files. Again, this is the direct consequence of the two objects being identical in their 3D geometry, thus having the same number of faces.
In a triangulated model each face definition is composed of three triplets of numbers, each triplet corresponding to one of the three vertices composing each face. Within each triplet, the first number should be the index to each of the 3D vertices connected to form a triangle; the central member of each triplet should be for texture vertices, and the third one for vertex normals.
- Between my two models, there is a partial mismatch in face order relative to the first (v) and third (vn) indices of each triplet. This mismatch might be caused by the fact that the main model has three materials assigned - one for the main texture, one for the railing and one for the rigging - whereas the AO model got only one material. Ungrouping materials in the AO model might have scrambled face order relative to the main model, since faces of the latter model are ordered on file depending on the material they use.
- There is a total mismatch of texture vertex indices, i.e. the second numbers in each triplet of face definitions. This mess might have been caused by the removal of "redundant" materials from the AO model (as above), but also by the cutting of some edges in the UV map of the main model (as explained before, there are more vt coordinates in the main model than there are in the AO model).

If Almagest could obviate to my mistakes, sorting out texture coordinates and faces in one click, that would be terrific. I realize that setting up the needed automated routines might be more complicated than it sounds though. Alternatively, I could try to manual edit my AO model in the hope to correct at least in part the problems above.
It that doesn't work, redoing the main UV mapping on the base of the AO model is still an option. That should not be a complicated task either. A little boring but not too complicated. For this particular task I think I would resort to Blender.
Blender's advantage is that, like Max, it supports multiple UV sets, and it got advanced and configurable shader setting. Very useful for previewing the combined effect of diffuse, specular, normal and AO maps. Unfortunately I am not as proficient in Blender as I am in Wings3D, but learning it better would be an investment for future projects.

Quote:

Originally Posted by gap (Post 2788535)

What I can tell you for sure is that if the main and the UV2 model are identical in all the respects except vt coordinate values, there is a 100% probability of success. With this idea in mind I have created an Excel spreadsheet which sorts vt coordinates for us. It is a rather crude tool, it requires a lot of manual data filtering, cutting and pasting; any mistake will cause wrong results, but if used correctly it does its job just fine.

If you are interested into testing it, I can upload it somewhere and post its link in the Almagest thread where it belongs more than here.

Couldn't be the problem coming from the fact that sub-objects (composing the final exported "diffuse object") are combined in a different order than in the "AO object" ?

If your excel sheet can solve the problem, let's try it. :up: Just explain to me how this works. :doh:

I know from reading the other thread that a question was raised about how the information on mapping is stored in a dat file.
A dat with only 1 map channel is stored in a simple block of data.
When We add say an AO channel? A TMAP section is added to the dat.
I'll have to refresh my memory on how the TMAPs actually work since it's been so long since it had to be done by hex editing. (Pre 2007!)
BUT! Only faces and texture coords are saved to the TMAP sections if I am recalling correctly.

I actually still have a program from 'back in the day' that exports and imports the different TMAP channels. So one could change or add just a TMAP and not have to re-import the complete obj like S3D does.

https://www.subsim.com/radioroom/pic...ictureid=12276

One thing I don't do is try to 'Guess' what went wrong.
Send me the files and I'll look at them.
I've probably forgotten more then most people know.
Not trying to be a smarmy butt head but I have been around the block more then once.

Quote:

Originally Posted by Jeff-Groves (Post 2788552)

One thing I don't do is try to 'Guess' what went wrong.
Send me the files and I'll look at them.

Here are .obj files which don't render correctly in game with the AO map : https://www.mediafire.com/file/ut1ss...O_map.zip/file

We are trying to understand why...

Quote:

Originally Posted by Mister_M (Post 2788544)

Couldn't be the problem coming from the fact that sub-objects (composing the final exported "diffuse object") are combined in a different order than in the "AO object" ?

Possible, but that's not the main suspect in my list of possible culprits.

What for sure can mess up the order of sub-objects (and faces) is material assignation. The main model and the AO model should have materials with similar naming (Wings3D sorts them in alphabetical order) assigned to corresponding faces. If, for example, a group of faces in the main model is given a material whose name is let's say 'wood', the equivalent faces in the AO/UV2 object should be given a material named 'wood_AO', 'wood_2', or something similar.

Quote:

Originally Posted by Mister_M (Post 2788544)

If your excel sheet can solve the problem, let's try it. :up: Just explain to me how this works. :doh:

Yes, it can. As I said, by using the output of my spreadsheet we can create an AO/UV2 obj file which is the exact copy of the main obj file, exception made for the values of texture coordinates.

Before I upload it, I need to be sure that it has enough rows for processing your model. Can you tell me the number of f and vt lines in the obj files you are processing? You can easily count them by pasting their content in an empty Excel document and by using the data filtering options or the COUNTIF function.

Quote:

Originally Posted by Jeff-Groves (Post 2788547)

I'll have to refresh my memory on how the TMAPs actually work since it's been so long since it had to be done by hex editing. (Pre 2007!)
BUT! Only faces and texture coords are saved to the TMAP sections if I am recalling correctly.

Thank you for your inputs Jeff!

Does that mean that faces and texture coordinates can be in a different order in the secondary map channel than in the primary one?

Quote:

Originally Posted by Jeff-Groves (Post 2788547)

I actually still have a program from 'back in the day' that exports and imports the different TMAP channels. So one could change or add just a TMAP and not have to re-import the complete obj like S3D does.

https://www.subsim.com/radioroom/pic...ictureid=12276

in future this tool might come in handy :up:

Well that was easy.
I compared the main obj and the AO obj in 010.
Your vertices do not match between the Main Hull and AO Hull.
(I suspected that but as I said. I'll not play a guessing game)
So anything else is going to be All screwed up on import!

What S3D does when importing the Main diffuse object and the AO object is this!

IF number counts are the same for the vertices? It ignores what those are in the AO object and only imports the faces and texture coords!

So basically S3D IS importing garbage you created.
That is the flaw AND the opening in S3D that I am working to fix!

Quote:

Originally Posted by gap (Post 2788563)

Can you tell me the number of f and vt lines in the obj files you are processing?

Hull1.obj :
2018 f
868 vt

Hull1-uv2.obj :
2018 f
2729 vt

It already seems problematic... :D

[QUOTE=Jeff-Groves;2788565]Well that was easy.
I compared the main obj and the AO obj in 010.
Your vertices do not match between the Main Hull and AO Hull.

Ah ! You replied while I was writting this...

Quote:

Originally Posted by Jeff-Groves (Post 2788565)

So basically S3D IS importing garbage you created.

Sorry, you mean : garbage that Wings3D created... :D

Quote:

Originally Posted by Jeff-Groves (Post 2788565)

That is the flaw AND the opening in S3D that I am working to fix!

You are trying to fix garbage created by Wings3D when exporting ?

Well. In the dat file format you CAN HAVE different counts on that information!
But the VERTS MUST BE EXACTLY THE SAME!

I'd suspect Wings re-ordered the Verts placement in the exported obj file thus faces and textures are screwed.

And You are the one that used Wings. It didn't to it by itself!
LOL!

Quote:

Originally Posted by Jeff-Groves (Post 2788580)

I don't create garbage by myself... :rotfl2:

Just playing with you Mate.
:har:

It is strange to many that somethings do not work as expected with programs like S3D or TDW's program for SH5.

Both missed things and based their codes on false assumptions.
Sometimes that was a Ego thing on their parts to NOT listen to suggestions.
Sometimes it was a lack of understanding EXACTLY how the import target is constructed!

S3D used ALOT of information from the Grey Wolves in it's construction.
TDW's program aslo credits a Grey Wolf as soon as you open it!
(Guess WHO privateer is!)

Quote:

Originally Posted by Jeff-Groves (Post 2788586)

Just playing with you Mate.
:har:

:up:

Quote:

Originally Posted by Jeff-Groves (Post 2788586)

Guess WHO privateer is!

YOU

:Kaleun_Smile:

https://www.youtube.com/watch?v=EEFsJ-43pnA

I may be the LAST Wizard left here at SubSim.
:haha:

I look at the files in a way that others do not.
With nothing but 010 Hex editor? I opened and changed the Bunker so one can walk all around!

I'm probably the ONLY one left active that can change things and create a better future for other Modders.
I am in debt to the ones that taught me.

Quote:

Originally Posted by Jeff-Groves (Post 2788565)

Well that was easy.
I compared the main obj and the AO obj in 010.
Your vertices do not match between the Main Hull and AO Hull.

If you are talking about my puffer model vertices in the main model do match vertices in the AO model, confirmed by WimMerge comparison and by the fact that my spreadsheet (which supposes an exact match in vertex order) generates a correct output :yep:

I think you meant texture vertices. Didn't you?

Quote:

Originally Posted by Jeff-Groves (Post 2788565)

IF number counts are the same for the vertices? It ignores what those are in the AO object and only imports the faces and texture coords!

So basically S3D IS importing garbage you created.
That is the flaw AND the opening in S3D that I am working to fix!

in an obj file 3D faces are denoted by the vertices composing them. So in a triangulated model you need three numbers to denote a face, e.g. 3 31 15 (this is: vertices # 3, 31 and 15 in the list of vertices are connected to compose a face).

This is for non-UV-mapped objects. For UV-mapped objects you must add three more numbers. Following the previous example: 3/10 31/7 15/25.
This line tells us that vertices #3, #31 and #15 in the vertex list are connected to compose a face in the 3D space, and that vertices #10, #7 and #25 in the texture vertex list compose a face in the UV space, where vt #10 corresponds to v #3, vt #7 to v #31 and vt #25 to v #15.

The errors I and Mister_M have been experiencing lead me to think - but this is only my guessing - that face definitions are written in dat/gr2 files only once, that secondary texture coordinates are stored in the same order as main UV coordinates, and that neither S3d nor GR2E read all the information stored in secondary obj file's face definitions in order to sort texture coordinates appropriately.

Quote:

Originally Posted by Mister_M (Post 2788578)

Hull1.obj :
2018 f
868 vt

Hull1-uv2.obj :
2018 f
2729 vt

It already seems problematic... :D

Oh my :doh:

I was expecting some unbalance in vt count between the two files, but such a huge unbalance was really unexpected.

Did you mess with secondary UV map? I mean: cutting edges which were originally joined or vice-versa?

And since you are at it, can you check that vertices (v lines) are sorted in the same order in the UV2 obj file as in the main obj file? That's a quick comparison in WinMerge :)

Quote:

Originally Posted by gap (Post 2788604)

Oh my :doh:

I was expecting some unbalance in vt count between the two files, but such a huge unbalance was really unexpected.

If its my tramp steamer MisterM was writing about, this might also be my "fault" for using "lightmap pack" function for smaller faces to avoid problem I had where small and long faces like ropes and railings were omitted in AO baking. Lightmap pack uv-maps faces as separate rectangles in 2d space so there would be many more edge cuts in ao uvmap than in diffuse uv map.

Quote:

Originally Posted by gap (Post 2788604)

Oh my :doh:

I was expecting some unbalance in vt count between the two files, but such a huge unbalance was really unexpected.

:/\\!!

Quote:

Originally Posted by gap (Post 2788604)

Did you mess with secondary UV map? I mean: cutting edges which were originally joined or vice-versa?

Quote:

Originally Posted by gap (Post 2788604)

And since you are at it, can you check that vertices (v lines) are sorted in the same order in the UV2 obj file as in the main obj file? That's a quick comparison in WinMerge :)

I guess I have to copy/paste all the lines into 2 .txt files ?
EDIT : So, order is different, a long block (nearly the first 1/3) has been put at the end...

Quote:

Originally Posted by kapuhy (Post 2788606)

If its my tramp steamer MisterM was writing about

Yes

I guess I'm gonna need some crayons to explain things.
:har:

We ARE talking about dat file import of the files Mister_M sent me.
I have not seen the ORIGINAL FILES! Just what I got today.

In a DAT file? number of texture coords and faces does NOT matter!
Unlike SH5 where it DOES matter! (Thus the addition of false info to balance a strict import)

In Dat files, like GR2 files, ALL verts in ALL files MUST MATCH!

The files I have show that the COUNT of verts is fine but the placement does not match between the main and AO obj files.
Thus the Faces are off! Hell! For S3D and TDW's Tool? You could zero every Vert in the AO and as long as counts match? It will import!
Because they DO NOT read and save the verts in the AO obj!

Send me the ORIGINAL files.

Quote:

Originally Posted by Jeff-Groves (Post 2788618)

Send me the ORIGINAL files.

Well, Kapuhy sent me the obj files of his nice tramp steamer. If Kapuhy doesn't mind, I can send you his D/L link.

If not imported/exported via Wings3D, it seems that there isn't any glitch with AO-map in game.