SUBSIM: The Web's #1 resource for all submarine & naval simulations since 1997 |
10-18-11, 10:14 AM | #106 |
Black Magic
|
wow this Undine has exposed a lot of problems with my code. Things that I thought were at certain places can change based on where the pointers start. Correcting this all now and should have new version available soon
|
10-18-11, 10:18 AM | #107 | |||
Seasoned Skipper
Join Date: Aug 2010
Location: 49°44´N 129°40´E
Posts: 665
Downloads: 124
Uploads: 7
|
It's Nelson, from this post;
Quote:
Quote:
Then why GR2EditorViewer trying to find something on this path? This is noted in the Log. Quote:
__________________
Speed squadron is the speed of the slowest ship ... but only so long as on the trail of the squadron did not sit submarines ... |
|||
10-18-11, 10:35 AM | #108 |
Seasoned Skipper
Join Date: Aug 2010
Location: 49°44´N 129°40´E
Posts: 665
Downloads: 124
Uploads: 7
|
Sounds nice!
__________________
Speed squadron is the speed of the slowest ship ... but only so long as on the trail of the squadron did not sit submarines ... |
10-18-11, 10:56 AM | #109 | |
Black Magic
|
Quote:
That's more than likely from the embedded strings. There are pointers that point to this data. So what that was telling you is that the pointer located at 0x1D20 (actual file position of 0x1EE8) points to an embedded string. The pointers are confusing at first but when it clicks it all makes perfect sense. Another strange thing from the Undine: 3D Studio MAX.DocumentContents.Headers.0.1.2.3.4.5.6.7.8.9.1 0.11.12.13.14.15.Elements.Value. Notice something? There's only one null character per string and it's not aligned as per the stated alignment of the section (section 0). Interesting So now I have to figure out what says ignore alignment when reading embedded strings...... |
|
10-18-11, 11:24 AM | #110 |
Black Magic
|
okay that part is fixed about the checking to see if alignment needs to be ignored or not on embedded strings. Now I've come across another 'something new' that I need to look into (from Undine):
[4808] FromArtToolName=3D Studio MAX 0x5C (0x46E8) embedded string from 0xC378 (0x10A04) [4808] ArtToolMajorRevision=10 0x60 (0x46EC) [4808] ArtToolMinorRevision=0 0x64 (0x46F0) [4808] ArtToolPointerSize=64 0x68 (0x46F4) [4808] UnitsPerMeter=0.100000 0x6C (0x46F8) [4808] Origin=0.000000 0.000000 0.000000 0x70 (0x46FC) [4808] RightVector=1.000000 0.000000 0.000000 0x7C (0x4708) [4808] UpVector=0.000000 1.000000 0.000000 0x88 (0x4714) [4808] BackVector=0.000000 0.000000 -1.000000 0x94 (0x4720) It's that ArtToolPointerSize=64. Always seen that as 0x0 |
10-18-11, 12:35 PM | #111 |
Black Magic
|
and another something new for Undine: in granny viewer in ArtToolInfo:ExtendedData there are entries for Selection sets and layers. I'm not looking for those currently and thus is causing problems. Time to add this.
|
10-18-11, 03:40 PM | #112 |
Stowaway
Posts: n/a
Downloads:
Uploads:
|
Nice work so far Mate!
Sorry I couldn't help you with the C# stuff. It's Greek to me. C is what I work in mostly, without the Net stuff. So it hurts my head to try to follow along in a useable way. I did send a link to a compressed GR2 file so you could see the compressed and non-compressed settings in the Section indexes. I'll go through some old notes and see if I have some more useable information for you. I'm sure there's some shoved in a drawer somewhere. I'm a fantastic note taker, but a terrible note organizer! I often find notes and wonder WTF they are for! I've been working on the Fast Finder for a different Game. So most of my time has been spent there. Thier problem is that all files are stored in a cache area as numbers. Kind of like a Manifest number, and no extensions! There are jpg's, png's, gr2's, xml's, and an unknown type slammed into this cache area! Fast Finder allows one to drag the whole cache into it and then checks the files known headers to determine what is what. It then allows a double click to open with a known application, saveing the list so one can open it and work as if just scanned, extremely fast file copying if one decides to copy the files to somewhere else, automatic renameing of the files if a list is provided with the number matched to a known 'Actual in Game Name of file', and more. It also uses a plug-in system so the main application can be expanded in nearly anyway one wishes. So you can see why I'm not around much right now. Regards!! |
10-18-11, 06:34 PM | #113 |
Black Magic
|
Maybe for people like Obelix you might find this interesting:
Like I posted above there are two new menu entries in Granny Viewer for the Undine: SelectionSets and Layers. I have no clue what their purpose is but regardless of purpose I need to be reading these from the file. So let's dig into this..... Let's take a look at the first dozen or so pointers and see what's going on. Remember that part of dumb luck that I spoke about some posts back? It deals with how I decoded the pointers. I looked for patterns in them and noticed that every one whose second word (4 bytes) is a 0x6 usually denoted a section, change, or something different. So I set those as one type of pointer. Pointers that referenced embedded strings were type str and those pointers who in turn referenced those pointers were type str_reference. I also defined a type as pointer_data and pointer but I'd have to go back and look in my code to see what determines each. Point is I made a wild guess and it worked. Dumb luck to the extreme. You have the early versions of the source code that shows how I denoted what type of pointer they are. The really funny thing is that I had to include a hack to get some of the pointers to work correctly. And this hack worked also Looking at the debug output of the first dozen or so pointers shows: [1572] Reading file header data starting at 0x0 [1572] Magic string=)ÞlÀº¤S+%õ·¥öfâî 0x10 [1572] Start of file data file offset (from file beginning)=0x1C8 0x14 where the file list header starts [1572] [1572] Reading section 0 (arttoolinfo) data starting at 0x68 [1572] Unknown=0x0 0x6C always 0x0 [1572] Section file offset=0x468C 0x70 where this section starts in the file [1572] Compressed size=61184 0x74 compressed size and uncompressed sizes should equal for SH5 GR2s [1572] Uncompressed size=61184 0x78 compressed size and uncompressed sizes should equal for SH5 GR2s [1572] Alignment=0x4 0x7C what the alignment boundary is in the file [1572] Start of data fileoffset=0xBD9C 0x80 start of data [1572] Start of data fileoffset2=0xBD9C 0x84 start of data [1572] Pointers file offset=0x1C8 0x88 where the pointers for this section are located [1572] Number of pointers=1467 0x8C [1572] End of pointers file offset=0x468C 0x90 [1572] Unknown2=0x0 0x94 [1572] Current position in file is 0x94 [1572] [1572] Reading section 1 (vertex_data) data starting at 0x94 [1572] Unknown=0x0 0x98 always 0x0 [1572] Section file offset=0x1358C 0x9C where this section starts in the file [1572] Compressed size=1368640 0xA0 compressed size and uncompressed sizes should equal for SH5 GR2s [1572] Uncompressed size=1368640 0xA4 compressed size and uncompressed sizes should equal for SH5 GR2s [1572] Alignment=0x20 0xA8 what the alignment boundary is in the file [1572] Start of data fileoffset=0x14E240 0xAC start of data [1572] Start of data fileoffset2=0x14E240 0xB0 start of data [1572] Pointers file offset=0x1358C 0xB4 where the pointers for this section are located [1572] Number of pointers=0 0xB8 [1572] End of pointers file offset=0x1358C 0xBC [1572] Unknown2=0x0 0xC0 (snip) [1572] Current position in file is 0xC0 [1572] [1572] File info offset at 0x8 (0x4694) [1572] Number of textures=5 0xC (0x4698) [1572] Number of materials=11 0x14 (0x46A0) [1572] Number of skeletons=1 0x1C (0x46A8) [1572] Number of vertex datas=20 0x24 (0x46B0) [1572] Number of tritopologies=20 0x2C (0x46B8) [1572] Number of meshes=20 0x34 (0x46C0) [1572] Number of models=1 0x3C (0x46C8) [1572] Number of trackgroups=0 0x44 (0x46D0) [1572] Number of animations=0 0x4C (0x46D8) [1572] Number of unknown=0 0x54 (0x46E0) [1572] Current position in file is 0x46E4 [1572] Pointer 0 is type str with offsetinsectiondata of 0x5C (0x46E8) and unknown of 0x0 and offsetinsectionvalue of 0xC378 (0x10A04) [1572] Pointer 0 updates the pointer to ArtToolInfo:FromArtTooFromName with pointer to embedded string for it So what do we know? From file header the first section starts at 0x1C8 in the file. Section 0 tells us that the section file offset is at 0x468C and start of pointers is at 0x1C8. So pointers come before data in this file (first thing that tripped me up - all the other GR2 files I've read had data before pointers). There are 1467 pointers for this section and the end of them resides at offset 0x468C. End of pointers equals start of section offset - Check. Let's double check this: file header said first section starts at 0x1C8. Section 0 Pointers start at 0x1C8. Check. There are 1467 pointers that contain 3 words (4 bytes) so 1467 * 3 * 4 = 0x44C4. Start of pointers 0x1C8 + size of pointers 0x44C4 = 0x468C. Section 0 said section file offset is at 0x468C. Check. Section 0 says size of this section is 0xEF00. Section 1 says it starts at 0x1358C. Start of section 1 0x1358C - start of section 0 0x1C8 = 0x133C4. This difference 0x133C4 - size of pointers 0x44C4 = 0xEF00. Check. So with the basic checks out of the way we can proceed. Ha!! Just had a revelation while typing this! I couldn't figure out what to do with the: [1572] Start of data fileoffset=0xBD9C 0x80 start of data [1572] Start of data fileoffset2=0xBD9C 0x84 start of data Those numbers (0xBD9C) were voodoo to me. But it just hit me after telling you all about the above how to use it! Funny how things work like that. If I take the section start offset 0x468C and add 0xBD9C to it I should be at start of embedded strings (0x10428). Now why I don't have to add the start of section (0x1C8) to this is just a quirk of the GR2 file I guess.....let's take a look: 00010428 41 72 74 54 6F 6F 6C 49 6E 66 6F 00 46 72 6F 6D ArtToolInfo.From 00010438 41 72 74 54 6F 6F 6C 4E 61 6D 65 00 41 72 74 54 ArtToolName.ArtT 00010448 6F 6F 6C 4D 61 6A 6F 72 52 65 76 69 73 69 6F 6E oolMajorRevision 00010458 00 41 72 74 54 6F 6F 6C 4D 69 6E 6F 72 52 65 76 .ArtToolMinorRev 00010468 69 73 69 6F 6E 00 ision. Yep, confirmed. Cool. Now have to change my code in the way I was retrieving these embedded strings......Thinking back the section start offset is a physical location in the file not an offset. I didn't know if the 0xBD9C was an offset or physical address. Now I know it's an offset. So everything makes sense now as to why I didn't have to add 0x1C8 to the result. Now I have to go and confirm this with other GR2 files....Confirmed. Works like a charm! [1572] Pointer 0 is type str with offsetinsectiondata of 0x5C (0x46E8) and unknown of 0x0 and offsetinsectionvalue of 0xC378 (0x10A04) [1572] Pointer 0 updates the pointer to ArtToolInfo:FromArtTooFromName with pointer to embedded string for it So how do I know this? Let's look at the pointers in a hex editor. Open up the undine GR2 with hex editor. Now section 0 said pointers start at 0x1C8. So let's navigate to 0x1C8. 000001C8 5C 00 00 00 00 00 00 00 78 C3 00 00 \.......x... the first word (0x5C000000) is what I call offsetinsectiondata. The second word (0x00000000) is what I call unknown. The third word (0x78C30000) is what I call offsetinsectionvalue. You have to remember that these are OFFSETS not physical locations. To get physical location you have to add the section's file offset to it. Those not familiar with reading the Intel hex format will think that the 78C30000 is just that. Incorrect. Intel hex format is little-endian style so you have to read from right to left. The value is actually 0x0000C378 if you wanted to type it into windows calc to get decimal value (50040d). Great, so what does this mean? 0x5C could mean anything right? Yes it could BUT let's set the hex editor at 0xC378 + section 0 section file offset (0x468C). You should have hex editor at 0x10A04. Here's what it shows: 00010A04 33 44 20 53 74 75 64 69 6F 20 4D 41 58 00 44 6F 3D Studio MAX.Do 00010A14 63 75 6D 65 6E 74 43 6F 6E 74 65 6E 74 73 00 48 cumentContents.H 00010A24 65 61 64 65 72 73 00 30 eaders.0 Those not familiar with C or assembly won't recognize the significance of the 00 bytes. Those 00 bytes are null termination characters. They signify the end of a string in C and assembly. If we put this all together then pointer 0 is type of string pointer (str) that points to the embedded string 3D Studio MAX. So some will say big deal, it points to a string. This IS a big deal. It tells us much more than that!!! Fire up RAD Game tools GrannyViewer and open up the undine GR2 file in it. Click the file list tab at the bottom. In the box for undine right click and select View In Detail. What do we see? We see some entries for ArtToolInfo, ExporterInfo, etc. Once again some will say so what? Let's take a deeper look at this. Look at ArtToolInfo once again. Notice the struct * to the left of ArtToolInfo? Those not familiar with C++ won't recognize the significance of the * symbol. This means a pointer to a struct. What this tells me is that if I click the (click to view sub-structure) of ArtToolInfo it's going to tell me the layout for that struct. Let's click it. Hot damn, what do you know We have a perfect definition of this structure. So where does this pointer 0 play into this? Well remember it pointed to an embedded string. The first entry in this new screen in granny is a pointer to a character array (C++ syntax for a string) and what do you know it says "3D Studio MAX". Kinda strange how that is the same value as to what pointer 0 points at isn't it? I'd be willing to bet then that if we set the hex editor at pointer 0's offsetinsectiondata + section 0 section file offset we will find this struct in the file. Let's see. Set hex editor to 0x5C + section 0 section file offset 0x468C. This makes 0x46E8. Let's see what we find: 000046E8 00 00 00 00 0A 00 00 00 00 00 00 00 40 00 00 00 ............@... 000046F8 CD CC CC 3D 00 00 00 00 00 00 00 00 00 00 00 00 ...=............ 00004708 00 00 80 3F 00 00 00 00 00 00 00 00 00 00 00 00 ...?............ 00004718 00 00 80 3F 00 00 00 00 00 00 00 00 00 00 00 00 ...?............ 00004728 00 00 80 BF .... Jackpot! The first 4 bytes are 0x0 because they are where the pointer to the embedded string "3D Studio MAX" resides. In the programming world pointers are usually 32 bits (4 bytes - 8 bits to a byte). GrannyViewer says next entry should be an int32 (4 bytes) and it describes the ArtToolMajorRevision. Well 0x0A000000 is 10 decimal. GrannyViewer says ArtToolMajorRevision is 10. Voila. That is why I say the GrannyViewer is an SDK of sorts. It tells you everything you need to know to read the GR2 file along with the pointer data. It defines the structures (for the most part, sometimes you have to stumble you way 4-8 bytes here or there to find a item) and within those structures it tells you the data type for each structure member. If you skip 0x28 bytes from the last 'entry' for ArtToolInfo you will be at ArtToolInfo:ExtendedDataocumentContents:Headers in GrannyViewer. It's this skipping of 0x28 - 0x10 (these 0x10 are known) bytes that is new to Undine that I have to figure out how to read. These 0x18 bytes are the SectionSets (2) and Layers(3) entries in GrannyViewer ArtToolInfo:ExtendedData: 0000472C 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 ................ 0000473C 00 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ 0000474C 97 00 00 00 00 00 00 00 ........ Now I have a feeling that there is only 0x10 new bytes defined because: 0x4 bytes number of SectionSets (02000000), 0x4 bytes for pointer to those SectionSets in file (00000000), 0x4 bytes for number of Layers (03000000), and 0x4 bytes for pointer to those Layers in file (00000000). Grand total is 0x10 new bytes....I'm guessing here but am usually correct. This also means an assumption I made about where ArtToolInfo:ExtendedDataocumentContents resides in the file is incorrect and I have to re-evaluate this. The reading of the file header and sections I couldn't figure out because GrannyViewer didn't 'show' anything about them. It wasn't until Privateer released his first template for 010 editor that they were revealed. That was the missing piece of the puzzle I needed to put everything together More to come...... Last edited by TheDarkWraith; 10-18-11 at 11:40 PM. |
10-18-11, 06:42 PM | #114 |
Stowaway
Posts: n/a
Downloads:
Uploads:
|
Very well played Mate!
|
10-18-11, 08:00 PM | #115 |
Black Magic
|
another new finding:
If pointers come before data in the section then the next entry after Number of pointers in the Section data is End of pointers file offset. If pointers come after data in the section then that next entry is the file size in bytes. Just noticed this when I tried my fixes for Undine on the lifeboat (my app's error checking barked at me and said danger will robinson!) Updated my section class to display correct debug output for it too: lifeboat: [4984] GR2 file path is C:\Games\Ubisoft\Silent Hunter 5\data\Sea\CMD_small_boat\LifeBoat.GR2 [4984] [4984] Current position in file is 0x0 [4984] [4984] Reading file header data starting at 0x0 [4984] Magic string=)ÞlÀº¤S+%õ·¥öfâî 0x10 [4984] Start of file data file offset (from file beginning)=0x1C8 0x14 where the file list header starts [4984] Unknown1=0x0 0x18 [4984] Unknown2=0x0 0x1C [4984] Unknown3=0x0 0x20 [4984] Current position in file is 0x20 [4984] [4984] Reading info header data starting at 0x20 [4984] File format revision=0x7 0x24 [4984] File size in bytes=238632 0x28 [4984] CRC value=0xA682FD63 0x2C [4984] Bytes from offset to start of sections=0x48 0x30 bytes from start of this to first section [4984] Number of sections=8 0x34 [4984] Root object type (?)=0x6 0x38 [4984] Root object offset (?)=0x3540 0x3C [4984] Unknown1=0x0 0x40 [4984] Unknown2=0x0 0x44 [4984] Tag=0x80000032 0x48 [4984] Current position in file is 0x68 [4984] [4984] Reading section 0 (arttoolinfo) data starting at 0x68 [4984] Unknown=0x0 0x6C always 0x0 [4984] Section file offset=0x1C8 0x70 where this section starts in the file [4984] Compressed size=13952 0x74 compressed size and uncompressed sizes should equal for SH5 GR2s [4984] Uncompressed size=13952 0x78 compressed size and uncompressed sizes should equal for SH5 GR2s [4984] Alignment=0x4 0x7C what the alignment boundary is in the file [4984] Start of data fileoffset=0x177C 0x80 start of data [4984] Start of data fileoffset2=0x177C 0x84 start of data [4984] Pointers file offset=0x37C68 0x88 where the pointers for this section are located [4984] Number of pointers=229 0x8C [4984] End of pointers file offset=0x3A428 0x90 where the pointers for the file end [4984] Unknown2=0x0 0x94 [4984] Current position in file is 0x94 [4984] [4984] Reading section 1 (vertex_data) data starting at 0x94 [4984] Unknown=0x0 0x98 always 0x0 [4984] Section file offset=0x3848 0x9C where this section starts in the file [4984] Compressed size=182400 0xA0 compressed size and uncompressed sizes should equal for SH5 GR2s [4984] Uncompressed size=182400 0xA4 compressed size and uncompressed sizes should equal for SH5 GR2s [4984] Alignment=0x4 0xA8 what the alignment boundary is in the file [4984] Start of data fileoffset=0x2C880 0xAC start of data [4984] Start of data fileoffset2=0x2C880 0xB0 start of data [4984] Pointers file offset=0x38724 0xB4 where the pointers for this section are located [4984] Number of pointers=0 0xB8 [4984] End of pointers file offset=0x3A428 0xBC where the pointers for the file end [4984] Unknown2=0x0 0xC0 [4984] Current position in file is 0xC0 (snip) Undine: [4612] GR2 file path is C:\Games\Ubisoft\Silent Hunter 5\data\Submarine\NSS_Undine\NSS_Undine.GR2 [4612] [4612] Current position in file is 0x0 [4612] [4612] Reading file header data starting at 0x0 [4612] Magic string=)ÞlÀº¤S+%õ·¥öfâî 0x10 [4612] Start of file data file offset (from file beginning)=0x1C8 0x14 where the file list header starts [4612] Unknown1=0x0 0x18 [4612] Unknown2=0x0 0x1C [4612] Unknown3=0x0 0x20 [4612] Current position in file is 0x20 [4612] [4612] Reading info header data starting at 0x20 [4612] File format revision=0x7 0x24 [4612] File size in bytes=1720964 0x28 [4612] CRC value=0xC52FD13B 0x2C [4612] Bytes from offset to start of sections=0x48 0x30 bytes from start of this to first section [4612] Number of sections=8 0x34 [4612] Root object type (?)=0x6 0x38 [4612] Root object offset (?)=0x0 0x3C [4612] Unknown1=0x0 0x40 [4612] Unknown2=0x0 0x44 [4612] Tag=0x80000032 0x48 [4612] Current position in file is 0x68 [4612] [4612] Reading section 0 (arttoolinfo) data starting at 0x68 [4612] Unknown=0x0 0x6C always 0x0 [4612] Section file offset=0x468C 0x70 where this section starts in the file [4612] Compressed size=61184 0x74 compressed size and uncompressed sizes should equal for SH5 GR2s [4612] Uncompressed size=61184 0x78 compressed size and uncompressed sizes should equal for SH5 GR2s [4612] Alignment=0x4 0x7C what the alignment boundary is in the file [4612] Start of data fileoffset=0xBD9C 0x80 start of data [4612] Start of data fileoffset2=0xBD9C 0x84 start of data [4612] Pointers file offset=0x1C8 0x88 where the pointers for this section are located [4612] Number of pointers=1467 0x8C [4612] End of pointers file offset=0x468C 0x90 where the pointers for this section end [4612] Unknown2=0x0 0x94 [4612] Current position in file is 0x94 [4612] [4612] Reading section 1 (vertex_data) data starting at 0x94 [4612] Unknown=0x0 0x98 always 0x0 [4612] Section file offset=0x1358C 0x9C where this section starts in the file [4612] Compressed size=1368640 0xA0 compressed size and uncompressed sizes should equal for SH5 GR2s [4612] Uncompressed size=1368640 0xA4 compressed size and uncompressed sizes should equal for SH5 GR2s [4612] Alignment=0x20 0xA8 what the alignment boundary is in the file [4612] Start of data fileoffset=0x14E240 0xAC start of data [4612] Start of data fileoffset2=0x14E240 0xB0 start of data [4612] Pointers file offset=0x1358C 0xB4 where the pointers for this section are located [4612] Number of pointers=0 0xB8 [4612] End of pointers file offset=0x1358C 0xBC where the pointers for the file end [4612] Unknown2=0x0 0xC0 [4612] Current position in file is 0xC0 (snip) Last edited by TheDarkWraith; 10-18-11 at 08:29 PM. |
10-18-11, 08:32 PM | #116 |
Stowaway
Posts: n/a
Downloads:
Uploads:
|
Oh, I forgot to thank you for the opening Credit also.
|
10-18-11, 08:39 PM | #117 |
Black Magic
|
|
10-18-11, 09:20 PM | #118 |
Stowaway
Posts: n/a
Downloads:
Uploads:
|
There is alot of those WTF moments in the GR2 files.
I don't see them as dynamic. They follow rules. Those rules are changed IF a file is modified AFTER being created. But at that point they follow the RULES. What I mean is>>>>>>>>>>> A file is exported from say 3DS Max. It has our base meshes and such. Now I want to ADD meshes. I know there is code to do so in the Software that comes with the Granny system. But once you do that? Add stuff or correct stuff? I'm not a 100% clear on what it messes with. I do know it changes many pointers. |
10-18-11, 11:34 PM | #119 |
Black Magic
|
The beauty of it is the pointers. Figure out the algorithm and it'll be no problem at all. It'll take time to do but anything can be cracked with enough time The more people working on it the faster it's cracked.
The website GR2Decode or something like that is someone I'm going to reach out to and anyone else that has been working on GR2 files. See if I can't stir up some enthusiasm. The QuArK people are another good one to reach out to. Last edited by TheDarkWraith; 10-19-11 at 12:44 AM. |
10-19-11, 03:33 AM | #120 |
Black Magic
|
just found a big bug in my code that prevented it from reading all of a section's pointers. I was pulling my hair out trying to figure out why some pointers weren't pointing to these new entries for Undine in GrannyViewer. They had to be there. When I went and compared the pointers I had to what was contained in the file they didn't match. That led me to look at the code and find the bug. Well more work to do tomorrow now This will probably shed light on many things I've been looking for in the pointers.
|
|
|