XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Wed Feb 22, 2017 2:21 pm

All times are UTC + 1 hour




Post new topic Reply to topic  [ 14 posts ] 

Do you think I should make more tutorials?
yes 100%  100%  [ 16 ]
no (its not understandable) 0%  0%  [ 0 ]
no (its uncomfortable to view) 0%  0%  [ 0 ]
no (for other reasons) 0%  0%  [ 0 ]
Total votes : 16
Author Message
 Post subject: Video tutorials on model formats reversing
PostPosted: Fri Jan 06, 2017 9:14 am 
Offline
double-veteran
double-veteran

Joined: Tue Mar 24, 2015 8:12 pm
Posts: 1011
Has thanked: 12 times
Have thanks: 393 times

Video tutorials on model formats reversing.

Far Cry Primal (2016)
https://www.youtube.com/watch?v=-1b3bM7ih6Y
This is an example of easy modern formats. Only 10 minutes to reverse the static geometry.

Planetside (2003) model format full reversing. Shows how to split and identify different parts of model data inside files: geometry, faces, UVs, normals, submeshes, materials, bones. Basic knowledge of hex data is required: how to read "little endian" integers and floats in hex.

This is an old game, and many format parts are different from modern standards, so this one is complex and takes more than an hour in 4 parts. Sorry for sound quality in first part, that was my first tutorial. Other parts have much better sound.

Part 1
https://www.youtube.com/watch?v=LHWOp4YeeoQ
Part 2
https://www.youtube.com/watch?v=WtIY05ZdBvI
Part 3
https://www.youtube.com/watch?v=znHaG38URp0
Part 4
https://www.youtube.com/watch?v=S4Fcdj4I12Q

You can make the ads go away by registering



Last edited by daemon1 on Sat Jan 14, 2017 3:19 pm, edited 2 times in total.

Top
 Profile  
 
 Post subject: Re: Video tutorial on full model format reversing (Planetsid
PostPosted: Wed Jan 11, 2017 4:57 pm 
Offline
double-veteran
double-veteran

Joined: Tue Mar 24, 2015 8:12 pm
Posts: 1011
Has thanked: 12 times
Have thanks: 393 times
Can I please have some feedback on this?

Its a bit hard to make "puzzle solving" video when I already solved it, there may be mistakes. Also I know my English is far from perfect. I made a poll here. Maybe this Planetside was too complex. That's why I did it, it has so many interesting parts. But maybe show something easy?


Top
 Profile  
 
 Post subject: Re: Video tutorial on full model format reversing (Planetsid
PostPosted: Thu Jan 12, 2017 6:38 pm 
Offline
M-M-M-Monster veteran
M-M-M-Monster veteran
User avatar

Joined: Fri Apr 20, 2012 9:24 am
Posts: 1911
Has thanked: 325 times
Have thanks: 893 times
I'm more a reader than a watcher so I can't tell too much.
But 'yes', half an hour for part one is a little bit too long, I guess.

If the example were easier the video(s) would be shorter.

_________________
Extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-j2l.zip

"We are Microsoft. You will be assimilated. Resistance is Futile."


Top
 Profile  
 
 Post subject: Re: Video tutorial on full model format reversing (Planetsid
PostPosted: Sat Jan 14, 2017 3:21 pm 
Offline
double-veteran
double-veteran

Joined: Tue Mar 24, 2015 8:12 pm
Posts: 1011
Has thanked: 12 times
Have thanks: 393 times
shakotay2 wrote:
But 'yes', half an hour for part one is a little bit too long, I guess.
If the example were easier the video(s) would be shorter.


Thanks for the comment! ;)

Yes, this was complex and interesting format, which actually encouraged me to finally record this. And yes, easy example is a good idea, so today I'm adding the 10 minutes easy tutorial.

p.s. Added Far Cry Primal tutorial to the first post.


Top
 Profile  
 
 Post subject: Re: Video tutorial on full model format reversing (Planetsid
PostPosted: Sat Jan 14, 2017 4:36 pm 
Offline
advanced
User avatar

Joined: Wed May 16, 2012 6:36 am
Posts: 75
Location: India
Has thanked: 8 times
Have thanks: 0 time
daemon1 wrote:
shakotay2 wrote:
But 'yes', half an hour for part one is a little bit too long, I guess.
If the example were easier the video(s) would be shorter.


Thanks for the comment! ;)

Yes, this was complex and interesting format, which actually encouraged me to finally record this. And yes, easy example is a good idea, so today I'm adding the 10 minutes easy tutorial.

p.s. Added Far Cry Primal tutorial to the first post.


Thanks man, this should help lots.


Top
 Profile  
 
 Post subject: Re: Video tutorials on model formats reversing
PostPosted: Sat Jan 14, 2017 4:39 pm 
Offline
advanced
User avatar

Joined: Wed May 16, 2012 6:36 am
Posts: 75
Location: India
Has thanked: 8 times
Have thanks: 0 time
And yes you should do more tutorials, but one thing, please use a modern hex editor and not a DOS based one.


Top
 Profile  
 
 Post subject: Re: Video tutorials on model formats reversing
PostPosted: Sat Jan 14, 2017 5:56 pm 
Offline
advanced

Joined: Sat Mar 18, 2006 5:29 pm
Posts: 42
Has thanked: 10 times
Have thanks: 1 time
Awesome! For myself, personally, I'm thinking a tutorial on how to go about identifying compression/data format of game files, making use of QuickBMS scripting and/or specifically using 010 Editor for hex editing and its scripting could help me get more insights. I specifically mention 010 Editor, because I'd like to see how creating a 010 Editor script/Binary Template such as viewtopic.php?p=94871#p94871 could help in a workflow tutorial video. I'm suspecting it eases lots of manual translation/interpretation work.
I'm thinking an explanation video of your Far Cry Primal mammothcarcass_scavenged_01 using the 010 Editor (using its Inspector window to quickly see a type value, and/or possibly defining your own types to show there) might result in a shorter video. ;)

Oh, and if you ever need to do a hex comparison between files or something, an app called "Beyond Compare" may be useful for visualizing binary/hex differences.

Your English is understandable, so that's good! However, as someone who's just interested in modding games rather than merely extracting resources from them, I'm probably not really the intended audience; I'm no animator & I'm still getting my head wrapped around terms like LOD, (sub)meshes, materials vs textures, faces, geometry, UVs, normals?, bones. Perhaps a conceptual structure of a model format would help, something like (this is probably wrong:) a game object would typically include a model file (containing one or more LODs, each containing one or more materials, one or more vertices, materials), textures (sometimes defined as .dss files, having 4 distinct alpha layers, each of which also means something), animations, sounds, physics data. And then only focus on the 'models' part. But that could just be me requiring further studying. ;)

I assume you're doing some off-video background programming to read the binary data from the model into a file to import in 3D Studio Max or whatever it is you have there in the video? Can QuickBMS or a series of standard tools be used for that, rather than programming?


Top
 Profile  
 
 Post subject: Re: Video tutorials on model formats reversing
PostPosted: Sun Jan 15, 2017 11:18 am 
Offline
double-veteran
double-veteran

Joined: Tue Mar 24, 2015 8:12 pm
Posts: 1011
Has thanked: 12 times
Have thanks: 393 times
eri619 wrote:
And yes you should do more tutorials, but one thing, please use a modern hex editor and not a DOS based one.

I'm not familiar with modern hex editors, if I'll try using them, tutorials will be slower and worse.

Whats the problem with this one? Colors, or anything else?


Top
 Profile  
 
 Post subject: Re: Video tutorials on model formats reversing
PostPosted: Sun Jan 15, 2017 11:35 am 
Offline
double-veteran
double-veteran

Joined: Tue Mar 24, 2015 8:12 pm
Posts: 1011
Has thanked: 12 times
Have thanks: 393 times
Yes I know 010 editor can be useful for beginners. But it will not ease anything for me. I don't see how it can make video shorter. Of course, if I make a template, and just show it, it will be short. But the purpose of this tutorial is to show how you can analyse the data, not just show the already reversed files.

As for basics, a hex/data/term tutorial can also be done, but first I wanted to check how can I do that in general. Just a first example for a test. So maybe later.

bouvrie wrote:
I assume you're doing some off-video background programming to read the binary data from the model into a file to import in 3D Studio Max or whatever it is you have there in the video? Can QuickBMS or a series of standard tools be used for that, rather than programming?

Yes, I'm not showing the programming part. And writing a QuickBMS script IS actually a programming. Writing a standalone tool is not much different from BMS scripting. Moreover, I'd say "normal" programming is even easier. With all the free tools of today.


Top
 Profile  
 
 Post subject: Re: Video tutorials on model formats reversing
PostPosted: Mon Jan 16, 2017 1:18 am 
Offline
advanced

Joined: Sat Mar 18, 2006 5:29 pm
Posts: 42
Has thanked: 10 times
Have thanks: 1 time
daemon1 wrote:
Yes I know 010 editor can be useful for beginners. But it will not ease anything for me. I don't see how it can make video shorter.
I don't know 010 editor much, I just assumed the Inspector would be handy, as you can skip the 'windows calculator' bit (all values the hex can translate to are in the Inspector window). And I thought perhaps a 010 Editor Binary Template would allow for a more insightful way of analyzing similar files in the future. It's kind of a step between analyzing hex and creating a BMS script, right? I figured making a Binary Template would be somewhat like taking 'notes', only in a more structured way; once the Template covers the file, you know you defined/researched it completely. At some point, the observations must be put in an algorithm or file definition, if that process were part of the tutorial you'd have documented the whole workflow. :)
*edit* Ah, something like you did on the Planetside .uber video @ 8m40s. ;)

Basically in the video, you go from geometry to face data. You scroll down, and after a while you get to the "face data". All I see as a n00b is that the hex pattern changes a bit. I wouldn't have a clue what the fingerprint/pattern of face data is, or how to distinguish 32-/64 bits floats and (Unsigned)integers from just looking at their hex representation, let alone deal with data that is encrypted/compressed in some way... I just figured that if there's another (more graphical) hex editor, perhaps they can highlight/mark distinct patterns when it encounters them. That'd make looking through the file easier to understand. But I really appreciate the videos in HIEW already, I just need to do some more manual work and practice to get better at it! :)


Top
 Profile  
 
 Post subject: Re: Video tutorials on model formats reversing
PostPosted: Mon Jan 16, 2017 7:02 pm 
Offline
double-veteran
double-veteran

Joined: Tue Mar 24, 2015 8:12 pm
Posts: 1011
Has thanked: 12 times
Have thanks: 393 times
It's good that you described your impression. It will be useful to me to improve tutorials.

No I can't skip calculator part, inspector will not help in this case. I'm not representing hex as other types, I'm making calculations there.

bouvrie wrote:
It's kind of a step between analyzing hex and creating a BMS script, right?

Yes, it's kind of step between, but this step is not needed. If the file is simple, you can write a tool without any templates. If the file is complex, it will take time to make a template, and then you have to rewrite it all again into the tool, so in any case you will only lose time. So as I said before, templates are only needed if you're a beginner, or if you have bad memory ;)

I'm making notes and pictures (on a paper), and this is more convenient for me than templates. The picture you see on the Planetside .uber video is what I had in my memory when reversing. This was only needed to explain it to the viewers, you can't make such a thing in 010 editor.

bouvrie wrote:
I just figured that if there's another (more graphical) hex editor, perhaps they can highlight/mark distinct patterns when it encounters them

No, there's no such a thing. No editor can tell faces from vertices, you have to do it all yourself.

bouvrie wrote:
All I see as a n00b is that the hex pattern changes a bit

That's right, I also only see this. I even said that in a video, I don't know where vertices end and where faces start. The only way is make an assumption, check it. If its wrong, make another one, and check again. With time and experience, you will usually be able to make it right from the first try. I'm trying to explain how do to that.


Top
 Profile  
 
 Post subject: Re: Video tutorials on model formats reversing
PostPosted: Thu Jan 19, 2017 4:28 pm 
Offline
advanced

Joined: Sat Mar 18, 2006 5:29 pm
Posts: 42
Has thanked: 10 times
Have thanks: 1 time
Ah ok, I was expecting too much advanced tooling for reverse engineering. I figured since there are file rippers that can (deep) scan unknown files for headers of possible known resources contained therein (i.e. wav/voc/MP3/png/etc) and extract them, having such a pattern searching/highlighting feature embedded in a hexeditor interface could speed up the process of identifying the data format. That's what I was hoping 010 Editor could do, guess I was wrong.

Personally, I also kinda used pen and paper (do screenshots annotated in Paint count?:P) for making sense of the Far Cry 4 Map Format. Part of the file (header) is easily identifyable (ASCII strings, PNG file), after which the actual map data comes in a segment with the "FC3M" header. I think that part is compressed as I can't make anything from it without using the Dunia tools to extract it to dat/fat. :)


Top
 Profile  
 
 Post subject: Re: Video tutorials on model formats reversing
PostPosted: Thu Jan 19, 2017 4:39 pm 
Offline
double-veteran
double-veteran

Joined: Tue Mar 24, 2015 8:12 pm
Posts: 1011
Has thanked: 12 times
Have thanks: 393 times
bouvrie wrote:
headers of possible known resources contained therein (i.e. wav/voc/MP3/png/etc)


Yes, these are easy to detect automatically. They can be easily seen in hex as well. But you need some practice to read any custom data. Do you think it will be useful if I make basic hex/int/float reading tutorial now? There's a comprehensive text tutorial here on xentax, but do you think visual example will be easier to understand?


Top
 Profile  
 
 Post subject: Re: Video tutorials on model formats reversing
PostPosted: Fri Jan 20, 2017 1:59 pm 
Offline
advanced

Joined: Sat Mar 18, 2006 5:29 pm
Posts: 42
Has thanked: 10 times
Have thanks: 1 time
daemon1 wrote:
Do you think it will be useful if I make basic hex/int/float reading tutorial now? There's a comprehensive text tutorial here on xentax, but do you think visual example will be easier to understand?
Well, I haven't even looked at textual tutorials for this yet, so your time could be better spent! ;)

A visual interface could just help researchers reverse data formats, IF they (an Inspector Window of sorts) would be able to intelligently scan and 'guess' some data based on known patterns, and try to display it as such a picture/texture, play it as such a sound/movie, etc. That way you don't have to manually re-invent the wheel all over again.

And while there is no such thing, HIEW does the job just fine. Although it isn't sexy and could scare off people who may want to start modding, but aren't all that accustomed to console windows. (:



Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 14 posts ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group