XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Sat Mar 25, 2017 7:04 am

All times are UTC + 1 hour


Forum rules


Please click here to view the forum rules



Post new topic Reply to topic  [ 78 posts ]  Go to page 1, 2, 3, 4, 5, 6  Next
Author Message
 Post subject: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Thu Feb 02, 2017 6:55 pm 
Offline
n00b

Joined: Fri Jul 08, 2011 10:36 am
Posts: 14
Has thanked: 0 time
Have thanks: 19 times

UPDATE : I updated the viewer to also load the skeleton and there is an option to export the model as iqe. Read below for updated "How to use?".

Most of the credit goes to daemon1. He found out all of the data in the pak files and explained it really well with videos. This program is like an implementation of his findings.

Hello everyone,

I was looking at the last of us pak files, trying to find some sort of index table for vertex/face positions but I noticed that it is also possible to get this information by searching for patterns in the file, without actually using an index table.

So I made a program to find and view meshes from the pak files (ps4 version only). You can also export them as obj. I used Unity to be able to render the meshes (not very experienced with low level graphic libraries), that is why the program is a bit large.

First some screenshots to get attention :D

Ellie :
Image

David:
Image

UPDATED

How to Use?
It is actually pretty straight forward. You go to File->Open and select the model file (ex. "ellie-tstrm.pak") in the file selection dialog that pops up. After that another dialog pops up asking you to select the skeleton file this time. If the model has a separate skeleton file, you select that (ex. "ellie-skel-tstrm.pak"). Some models don't have separate skeleton files (like infected), instead the skeleton is inside the same file. In that case you just click "Cancel" for the second dialog, or you can just select the model file again. It will work either way.

The program will go through the file, finding meshes/skeleton and add them to the scene.

You can see the meshes that are currently in the scene, in a list box on the left. If you select a mesh from the list, it will get highlighted in the scene. Bones will also appear as white spheres but it is not possible to move/rotate the bones. They are just there to show you that the skeleton is loaded properly.

Normally the meshes should load fine. If for some reason one of the meshes appear distorted you can just disable it by checking the "Disable Mesh" box. Disabled meshes won't get exported.

Controls
Right-click and move mouse : Orbit around model
Ctrl + Mouse Wheel Scroll : Zoom
Mouse Wheel Click and move mouse : Pan
W Key : Wireframe shader

Exporting
There are two export options :

Obj : This will export all meshes as separate obj files. Of course there will be no skinning data and skeleton.
IQE : This will export the model as a single iqe file. IQE isn't really popular but it is a format that is not complex and that supports skinning, so I went with it.
There is a blender importer script for iqe : https://raw.githubusercontent.com/ccxvi ... _import.py From blender you can pretty much export it to any format you want.

I exported the model as fbx and imported it in Maya. It came out fine :
Image

Download the program : https://www.mediafire.com/?edt4a6k6mqd64tn

I tried loading some files other than the characters, like props, weapons, animals etc. and they all seemed to work fine with minor problems. Still there might be issues with some of the files, because I didn't tested every one of them. If that is the case, let me know.

If you are also interested in getting the textures, here is a tool that I posted in another thread : viewtopic.php?f=10&t=15801

Also huge thanks to luxox18. I used the data that he found out manually in the pak files as reference values/offsets while developing the program.

You can make the ads go away by registering



Last edited by akderebur on Thu Mar 23, 2017 4:32 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Mon Feb 13, 2017 3:18 pm 
Offline
advanced

Joined: Sat Oct 23, 2010 3:50 pm
Posts: 49
Has thanked: 2 times
Have thanks: 0 time
wow ,this is realy awsome!!


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Thu Feb 16, 2017 2:05 am 
Offline
advanced

Joined: Tue Nov 06, 2012 7:57 am
Posts: 65
Has thanked: 7 times
Have thanks: 1 time
Will this work on uncharted 4?


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Thu Feb 16, 2017 2:19 am 
Offline
n00b

Joined: Fri Jul 08, 2011 10:36 am
Posts: 14
Has thanked: 0 time
Have thanks: 19 times
Jaydenthetank wrote:
Will this work on uncharted 4?

I don't know, it might work if the file structure is identical with the last of us. I don't have the uncharted files, so didn't have the chance to test them.


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Thu Feb 16, 2017 6:53 am 
Offline
advanced

Joined: Tue Nov 06, 2012 7:57 am
Posts: 65
Has thanked: 7 times
Have thanks: 1 time
Just curious because I don't think anyone has ripped ps4 models before


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Thu Feb 16, 2017 11:21 am 
Offline
double-veteran
double-veteran

Joined: Mon Jul 13, 2009 6:16 pm
Posts: 653
Has thanked: 64 times
Have thanks: 103 times
Jaydenthetank wrote:
Just curious because I don't think anyone has ripped ps4 models before


https://www.youtube.com/watch?v=-Y1bigdq-6o&t

_________________
Image


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Sun Feb 19, 2017 12:10 am 
Offline
mega-veteran
mega-veteran
User avatar

Joined: Sun Jul 06, 2014 6:30 am
Posts: 310
Has thanked: 51 times
Have thanks: 195 times
akderebur wrote:
Jaydenthetank wrote:
Will this work on uncharted 4?

I don't know, it might work if the file structure is identical with the last of us. I don't have the uncharted files, so didn't have the chance to test them.


Uncharted 4 is something I started looking into. But the whole game is just one single pak file, so I suspect it's not similar at all. EDIT: I found out it's because PS4 games are encrypted, and still nobody knows how to unencrypt them (which begs the question how you unencrypted this game?)

I am curious if you can release the source code or futher explain the findings, as I'm sure it would help with my own investigations, the more we can pool information, the farther we can go.


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Sun Feb 19, 2017 2:43 am 
Offline
n00b

Joined: Fri Jul 08, 2011 10:36 am
Posts: 14
Has thanked: 0 time
Have thanks: 19 times
volfin wrote:
Uncharted 4 is something I started looking into. But the whole game is just one single pak file, so I suspect it's not similar at all. EDIT: I found out it's because PS4 games are encrypted, and still nobody knows how to unencrypt them (which begs the question how you unencrypted this game?)

I did not, luxox is the guy you are looking for :D

He released some character files in this thread : viewtopic.php?f=10&t=15801


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Sun Feb 19, 2017 6:55 am 
Offline
ultra-veteran
ultra-veteran
User avatar

Joined: Tue Jan 04, 2005 9:47 pm
Posts: 585
Has thanked: 43 times
Have thanks: 112 times
[out]


Last edited by Wobble on Sat Mar 11, 2017 11:17 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Sun Feb 19, 2017 8:01 am 
Offline
M-M-M-Monster veteran
M-M-M-Monster veteran
User avatar

Joined: Fri Apr 20, 2012 9:24 am
Posts: 1953
Has thanked: 342 times
Have thanks: 949 times
[well, I've been told, my guess is wrong for PS4, does apply for PS3 models only]

Guess, it's also a matter of Sony's edge compression:
viewtopic.php?f=16&t=12070&p=99284#p99284

Only tool I know (so far) that can handle it is Noesis:
rapi.rpgCommitTriangles(edgeDecomp, noesis.RPGEODATA_USHORT, indexCount, noesis.RPGEO_TRIANGLE, 1)

But detailed informations are kept secret.
That's sad because more infos could help with Gran Turismo 6 Models, too.

I tend to ignore edge decompression, because it's so boring. I remember fiddling around with Sony's edge SDK (or something like that, not sure).
Just a waste of time and life... :D

_________________
Extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-jm8.zip

"We are Microsoft. You will be assimilated. Resistance is Futile."


Last edited by shakotay2 on Mon Feb 20, 2017 7:28 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Sun Feb 19, 2017 12:32 pm 
Offline
double-veteran
double-veteran

Joined: Tue Mar 24, 2015 8:12 pm
Posts: 1060
Has thanked: 14 times
Have thanks: 424 times
akderebur wrote:
I did not, luxox is the guy you are looking for


luxox will not be able to help you. Hes just the guy who delivers decrypted files from the scene. He can't decrypt them.


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Sun Feb 19, 2017 1:49 pm 
Offline
n00b

Joined: Fri Jul 08, 2011 10:36 am
Posts: 14
Has thanked: 0 time
Have thanks: 19 times
Wobble wrote:
Viewers and characters are nice, but as a coder, we would like to see some discussion on formats, algorithms, or code.

There is no real algorithm that can bring you further, it is just like pattern matching program designed specifically for The Last of Us pak files. Let me explain how the program works and you will see what I mean.

The way I did it was I basically opened bunch of character files in the hex editor and tried to find out recurring patterns in each file. Here is an example for getting vertex/index count and index offset.

There are 4 patterns for this (in hex) :
  • 03 00 00 00 00 00 00 00 00 00 00 00
  • 04 00 00 00 00 00 00 00 00 00 00 00
  • 05 00 00 00 00 00 00 00 00 00 00 00
  • 06 00 00 00 00 00 00 00 00 00 00 00

Also after 44 bytes from the start of the above patterns, there should be this pattern : FFFFFFFF.

So lets say the file contains 10 meshes. The program starts and searches for these patterns in the file and when there is a match it adds the offset to a list. When it is done there are like 3000 (I am making up numbers for this example) records in the list. It then goes through the list, for each offset it advances 44 bytes and checks if there is FFFFFFFF. If it finds out that pattern, it adds that offset (end of FFFFFFFF) – 56 bytes to a new list. The reason for offset - 56 bytes is because the data we are looking for actually starts there, again something that I found out looking at the hex editor. In the end that list has 10 records with correct offsets.

Now we can read the actual data. The program goes through each offset in the new list :
  • binary reader’s position is set to the current offset
  • first integer read = vertex count
  • second integer read = index count
  • advance 28 bytes
  • integer read = index offset

I think at this point you can see that the program is a product of binary search and pattern matching and that it doesn’t use a proper way to find data. It is like a bruteforcer for finding data in tlou files only. I find it hard to believe that anyone can develop it further to let’s say work with uncharted 4 files. If the file structure is identical, it will work. If not you would need new patterns and it would be a whole new program in that case.

The program as it is contains more code for graphics and ui programming than the actual code for reading the pak file and it is a bit messy :D. If you want I can strip the Unity stuff and share the part related to pak files.

shakotay2 wrote:
Guess, it's also a matter of Sony's edge compression

Mesh data (index, vertex, uv) is not compressed in the ps4 files. Only the ps3 version has edge compression.


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Mon Feb 20, 2017 12:37 am 
Offline
ultra-veteran
ultra-veteran
User avatar

Joined: Tue Jan 04, 2005 9:47 pm
Posts: 585
Has thanked: 43 times
Have thanks: 112 times
[out]


Last edited by Wobble on Sat Mar 11, 2017 11:17 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Mon Feb 20, 2017 2:22 am 
Offline
mega-veteran
mega-veteran

Joined: Fri Jul 29, 2011 9:18 pm
Posts: 165
Has thanked: 51 times
Have thanks: 36 times
ps4 developers are not using the edge compression anymore. Instead are using swizzled textures and models with same structure as PC with little endian data. (nothing compressed)

in this game specifically the vertex data and the uv data are separated and all faces start with 00 00 01 00 02.

not all games can be decrypted, for the moment is possible in games with firmware 1.76 (2013 - 2014 games)

oh, and about how to obtain decrypted games.. I just have a contact in the ps4 scene


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Mon Feb 20, 2017 11:14 am 
Offline
ultra-veteran
ultra-veteran
User avatar

Joined: Tue Jan 04, 2005 9:47 pm
Posts: 585
Has thanked: 43 times
Have thanks: 112 times
[out]



Last edited by Wobble on Sat Mar 11, 2017 11:17 am, edited 1 time in total.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 78 posts ]  Go to page 1, 2, 3, 4, 5, 6  Next

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group