XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Wed Apr 25, 2018 3:33 pm

All times are UTC + 1 hour




Post new topic Reply to topic  [ 11 posts ] 
Author Message
 Post subject: Analyzing and Extracting a Game Model
PostPosted: Wed Mar 28, 2018 4:05 am 
Offline
mega-veteran
mega-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 203
Has thanked: 14 times
Have thanks: 79 times

Attention: This topic is subject to the Manual Researching Section under the tutorial An Imitable Workflow for Reverse Engineering A Game Model.

Part III. Analyzing and Extracting a Game Model

Now we're about to officially start our journal of exploring a game archive.
First you'll need a proper hex editor. My recommendation would be HexEdit. And that's what I'm going to introduce in this tutorial.

This is how its UI looks like:

Image

A few fields that you need to pay attention to:
The Offset Field: It records every address of the loaded file by columns and rows. By default the number of columns is set to 16 (0x10 in hex), but you can always expand them if you want to.
Hex View: The true nature of a file, which is consisted of an array of bytes. Each byte is represented by a two-digit hex number.
Text View: It converts each byte into human readable texts according to the ASCII table.
Toolbars: Here you can find some very useful functions. Choose what you need by navigating to View>Toobars. The most frequently used functions are the Calculator, the Bookmarks, and the Properties.

You can make the ads go away by registering


_________________
An Imitable Workflow for Reverse Engineering a Game Model

"No investigation means no right to speak."
Remember to thank people when you get helped. That's the minimum respect to their efforts.


Last edited by Bigchillghost on Wed Mar 28, 2018 4:41 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Analyzing and Extracting a Game Model
PostPosted: Wed Mar 28, 2018 4:10 am 
Offline
mega-veteran
mega-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 203
Has thanked: 14 times
Have thanks: 79 times
Now it comes to the practice part. I'll take Marvel Avengers: Battle for Earth on Xbox 360 as an example. We're not going to start from the original
game archives, but the extracted .nif files that contain the models.
Download the example files here.

The file we're going to work with is skrull_skin_m.nif. Let's open it in HexEdit. There're some strings at the front part of the file.
We're not going to reverse the entire file structure since we only need the models. So let's focus on the geometry data.

Usually it would be easier to locate the geo data by looking for the face buffer. As face indices are stored as an array of intergers, they usually look like a regular
character table in the text view instead of some random bytes.

Scroll down and we'll find the indices buffer approximately at offset 0x6760:
Image

Let's keep that in mind, or we can set a bookmark with HexEdit here and return later. Now continue to scroll down. Then we find the structure
changed around 0xE780.
Image

These compact data should be the vertices chunk. Note that it doesn't have to come after the face buffer. Sometimes it could be at the front.
But in this case, since the data before the face indices doesn't look compact, basically we can draw the conclusion.

If you've read @shakotay2's tutorial on Hex2Obj (Extracting simple models) you'll know that there're two most common structure of vertices data: "sequential" and "blocked".

But I would prefer to understand them as "normal" and "nested".

"sequential"/"normal"
It defines an array for each different element separately, and then places the arrays in file one after another:
Code:
vertices coordinates
{
   v1 x y z
   v2 x y z
   v3 x y z
   ...   
}
vertex normals
{
   vn1 x y z
   vn2 x y z
   vn3 x y z
   ...   
}
vertex UVs
{
   vt1 u v
   vt2 u v
   vt3 u v
   ...   
}

"blocked"/"nested"
It first defines a structure for a single vertex element, then defines an array of that structure. So that the structure is nested in this vertices array:
Code:
Vertices Array
{
   {
      v1 x y z; vn1 x y z; vt1 u v
   }
   {
      v2 x y z; vn2 x y z; vt2 u v
   }
   {
      v3 x y z; vn3 x y z; vt3 u v
   }
   ...
}

_________________
An Imitable Workflow for Reverse Engineering a Game Model

"No investigation means no right to speak."
Remember to thank people when you get helped. That's the minimum respect to their efforts.


Top
 Profile  
 
 Post subject: Re: Analyzing and Extracting a Game Model
PostPosted: Wed Mar 28, 2018 4:14 am 
Offline
mega-veteran
mega-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 203
Has thanked: 14 times
Have thanks: 79 times
But for either case we'll need to find the aligned size anyway.

Image

The sequence bytes 00 10 catch our eyes at the first glance. And there're always 0x30 bytes between the former and its next one. You can also do that
by measuring the size between each '<' but it won't be easy in this case. So let's expand the number of columns to 48, which is 0x30 in hex, and we
got this in the text view:

Image

As you can see the bytes are aligned into a vertical pattern. But to find the start of it we need to look at the binary values in the hex view.
Now it's time to explain you how endianness works.

Endianness refers to the order in which the bytes of a multi-byte word are stored in memory. So the 4-byte value 0x11223344 is stored as 44 33 22 11
for little-endian, and 11 22 33 44 for big-endian.

Usually the Windows and Android file system uses little endianness, while console file systems, like Wii, PS3 and Xbox 360, are using big endianness.

As I've mentioned before the game we're working with is for Xbox 360, so the file is in big-endian.

_________________
An Imitable Workflow for Reverse Engineering a Game Model

"No investigation means no right to speak."
Remember to thank people when you get helped. That's the minimum respect to their efforts.


Top
 Profile  
 
 Post subject: Re: Analyzing and Extracting a Game Model
PostPosted: Wed Mar 28, 2018 4:18 am 
Offline
mega-veteran
mega-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 203
Has thanked: 14 times
Have thanks: 79 times
Now we switch to the hex view:

Image

We can easily identify that the pattern before the highlighted float is 00 04 02 34 instead of the expected 00 00 3C 00.
So now we know the start offset of the vertices is 0xE7C3. The end offset should be at the same column. We scroll down and find its end at 0x358B3:

Image

You can see that the byte after 0x358B3 is not 0x2C but 0x01, which makes clear our earlier claim. As HexEdit shown, the total vertex chunk size
is 0x270F0 ( = 0x358B3 - 0xE7C3). We devide it by 0x30, and we get the vertices count 0xD05/3333.

Now let's do a little more research on the vertex structure. Obviously it won't be the "normal" type as the vertex size 48 has already exceeded
the maximum amount of bytes needed to store the 3 coordinates x, y and z. So it's "nested" for sure.

And we noticed that the sequence 3C 00 seems to be used for distinguishing different elements. We encounter the first 3C 00 after skipping 10 bytes from 0xE7C3.

10 bytes are not enough to hold 3 floats(= 12 bytes), so the data type could be short or half-float, both of which use 2 bytes. But there're still 4 bytes left, and natually
we will assume that to be another two shorts or half-floats for UVs.

Then right after the first 3C 00, there're 6 bytes following by another 3C 00, and this pattern repeats for 3 times. Assuming they are all two-byte values,
they could be the normal, binormal and tangent vectors. That sounds quite reasonable.

_________________
An Imitable Workflow for Reverse Engineering a Game Model

"No investigation means no right to speak."
Remember to thank people when you get helped. That's the minimum respect to their efforts.


Top
 Profile  
 
 Post subject: Re: Analyzing and Extracting a Game Model
PostPosted: Wed Mar 28, 2018 4:21 am 
Offline
mega-veteran
mega-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 203
Has thanked: 14 times
Have thanks: 79 times
Now let's open this file in Hex2Obj. Press litE to change the endianness to big-endian, seq to switch to "nested" mode("block"), noPtC to
switch to display as point cloud, and fill in the start offset and count of the vertices.

Image

For the data type of the vertices, let's try short type first.
After clicking the mesh button we got the point cloud shown as in the left
figure.

Image

Seems that is the UVs as we can see they' ll overlap if their 3rd coordinates are flatten. So let's change the offset to 0xE7C7 to
skip these two values. The point cloud is as shown in the middle figure.

It doesn't look correct though. Perhaps we should try with half-float now. Then we get the correct result as shown in the right figure.

So we can say that the data type of the vertices is half-float.

_________________
An Imitable Workflow for Reverse Engineering a Game Model

"No investigation means no right to speak."
Remember to thank people when you get helped. That's the minimum respect to their efforts.


Last edited by Bigchillghost on Mon Apr 02, 2018 9:31 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Analyzing and Extracting a Game Model
PostPosted: Wed Mar 28, 2018 4:25 am 
Offline
mega-veteran
mega-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 203
Has thanked: 14 times
Have thanks: 79 times
After that let's return to the face indices block, which is at 0x6760 if you don't remember.
Since all game models are using triangles, which means every face is represented by 3 indices. We know that our vertices count is 3333 < 65535, so the indices
should be using short intergers.

Usually the indices are stored in an increasing order so for most of the cases the first face will be 0, 1, 2.
Base on these info, plus we've already known the endianness, it's not hard to locate its start: 0x6764.

Image

The sequence 00 01 02 15 seems more like some signature bytes(indeed it is), and it'll break the regularity if we take it into acount as part of the buffer.
Besides, we can always correct that if our intuition failed.

To get the indices count, we'll need to find where it ends. There's an easy way to do that.
We scroll down to where the pattern changed, then set the number of columns to 3 x 2(the size of the indices data type), which is 6.
And the last index should end at the same column as the first one.

Image

So it ends at 0xE786. It's impossible to be 0xE78C as 0xF000 has far exceeded the vertices count 0xD05. We measure its size, divide it by 2, and get
the indices count: 0x8022 / 2 = 0x4011 (16401). Now let's get the indices loaded:

Image

Looks OK, except for some extra faces. Note that our current vertices count is 2686 out of 3333, which indicates that there're still 647 vertices left
not displayed, and that their start offset for the face indices is different from the one we're using now:

Image

_________________
An Imitable Workflow for Reverse Engineering a Game Model

"No investigation means no right to speak."
Remember to thank people when you get helped. That's the minimum respect to their efforts.


Last edited by Bigchillghost on Tue Apr 03, 2018 8:33 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Analyzing and Extracting a Game Model
PostPosted: Wed Mar 28, 2018 4:27 am 
Offline
mega-veteran
mega-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 203
Has thanked: 14 times
Have thanks: 79 times
You can grope around for the proper indices count of submesh1 by decreasing the amount and then check the result. But I'm not going to do it that way.

We notice that the vertices count of submesh1 is 2686 (0xA7E), which is larger than the remain 647 (0x287) of submesh2.
So the max index of submesh1 no doubts equals its vertices count. Now we're going to find the correspond indices counts for each submeshes according
to the vertices counts we have.

We don't know whether the file will use long or short type to store the counts, so we just search backward from the start of the vertices for the value as short, which is 0A 7E.
And we actually found it right ahead of the vertices buffer:

Image

And we can figure out the structure of this vertices header at once:
Code:
...             Face Buffer
01               LOD ID?
00 02 70 F0      Vertices Size
00 00 00 02      Submesh Count
00 00 0A 7E      First Submesh's Vertices Count
00 00 0A 7E      Current Used Vertices Sum
00 00 02 87      Second Submesh's Vertices Count
00 00 00 07      Vertices Element IDs Count (the following 7 4-byte sequences are the IDs). There'll be the same amount of elements in a vertex structure.

_________________
An Imitable Workflow for Reverse Engineering a Game Model

"No investigation means no right to speak."
Remember to thank people when you get helped. That's the minimum respect to their efforts.


Top
 Profile  
 
 Post subject: Re: Analyzing and Extracting a Game Model
PostPosted: Wed Mar 28, 2018 4:29 am 
Offline
mega-veteran
mega-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 203
Has thanked: 14 times
Have thanks: 79 times
We didn't find any info about the face buffer here but we find some similar structure before the indices buffer:

Image
Code:

00 00 80 22      Face Buffer Size
00 00 00 02      Submesh Count
00 00 34 0B      First Submesh's Indices Count
00 00 34 0B      Current Used Indices Sum
00 00 0C 06      Second Submesh's Indices Count
00 00 00 01      Indices Element IDs Count (the following sequence 00 01 02 15 is the ID)

So now we got both submesh's indices counts: 0x340B (13323) for submesh1, 0xC06 (3078) for submesh2. We sum them up and get 16401, which is exactly the total
indices count we filled in Hex2Obj. Let's validate it:

Image

Looks we're right.

_________________
An Imitable Workflow for Reverse Engineering a Game Model

"No investigation means no right to speak."
Remember to thank people when you get helped. That's the minimum respect to their efforts.


Last edited by Bigchillghost on Mon Apr 02, 2018 9:49 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Analyzing and Extracting a Game Model
PostPosted: Wed Mar 28, 2018 4:31 am 
Offline
mega-veteran
mega-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 203
Has thanked: 14 times
Have thanks: 79 times
Now it's time to handle the UVs.
As you may recall, the UVs are ahead of the vertex coordinates, so we can't use its position relative to the coordinates to access to them. Instead, we use
the obsolute offset of the UVs. The UVB size is the same as the vertices as they're under the same structure. We fill in the blanks with the right info, select
HF_UV from the data type rollout, then press the UVs button:

Image

Now let's save this model as an obj file by navigating to File>SaveAs mesh.

_________________
An Imitable Workflow for Reverse Engineering a Game Model

"No investigation means no right to speak."
Remember to thank people when you get helped. That's the minimum respect to their efforts.


Top
 Profile  
 
 Post subject: Re: Analyzing and Extracting a Game Model
PostPosted: Wed Mar 28, 2018 4:32 am 
Offline
mega-veteran
mega-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 203
Has thanked: 14 times
Have thanks: 79 times
We can even test with submesh2 if we want to. Just need several steps of calculations to obtain the start offsets:

Vertices: 0xE7C7 + 2686 * 48 = 0x2DF67
UVs: 0x2DF67 - 0x4 = 0x2DF63
FI: 0x6764 + 13323 * 2 = 0xCF7A

With the assistance of the calculator of HexEdit we can get the results in a few seconds.

We've already got the counts for submesh2: 3078 indices and 647 verts. So let's see how it goes:

Image

Everything play as we expected. So far we can already build the model with all these infomation we have, but there're still some remaining bytes in
that vertex structure. What are these data? You can actually find the answer from the strings at the front part of the file, which I told you not to pay
much attention to:

Image

But don't get it wrong. The only reason I skipped these strings is for teaching purpose. When you're actually researching a file format, you should make full
use of these info. It'll definitely save you a lot of time.

_________________
An Imitable Workflow for Reverse Engineering a Game Model

"No investigation means no right to speak."
Remember to thank people when you get helped. That's the minimum respect to their efforts.


Last edited by Bigchillghost on Mon Apr 02, 2018 9:56 am, edited 2 times in total.

Top
 Profile  
 
 Post subject: Re: Analyzing and Extracting a Game Model
PostPosted: Wed Mar 28, 2018 4:34 am 
Offline
mega-veteran
mega-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 203
Has thanked: 14 times
Have thanks: 79 times
Up to now, I've shown you how do analyze a model format, and to use Hex2Obj to validate your guess. I've also left you with another nif file (iron_skin_m.nif).
See if you can extract the model within on your own.

So, I guess that's all for this part.


_________________
An Imitable Workflow for Reverse Engineering a Game Model

"No investigation means no right to speak."
Remember to thank people when you get helped. That's the minimum respect to their efforts.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 11 posts ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group