XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Sat Jul 21, 2018 9:31 am

All times are UTC + 1 hour


Forum rules


Please click here to view the forum rules



Post new topic Reply to topic  [ 104 posts ]  Go to page 1, 2, 3, 4, 5 ... 7  Next
Author Message
 Post subject: The Last of Us Model Viewer/Extractor (PS4)
PostPosted: Thu Feb 02, 2017 6:55 pm 
Offline
mega-veteran
mega-veteran

Joined: Fri Jul 08, 2011 10:36 am
Posts: 194
Has thanked: 26 times
Have thanks: 148 times

UPDATE
- Added support for loading maps
- Fixed some problems with the characters
- Fixed "Remake UVs"

Hello everyone,

I was looking at The Last Of Us pak files, trying to find some sort of index table for vertex/face positions but I noticed that it is also possible to get this information by searching for patterns in the file, without actually using an index table.

So I made a program to find and view meshes from the pak files (ps4 version only). You can also export them as obj/iqe. I used Unity to be able to render the meshes (not very experienced with low level graphic libraries), that is why the program is a bit large.

First some screenshots to get attention :D

Ellie :
Image

David:
Image

UPDATED

How to Use?
- File -> Open to load the pak file
- In the first file select, select the pak file that has the model data (ex. "ellie-tstrm.pak")
- In the second file select, select the pak file that has the skeleton data (ex. "ellie-skel-tstrm.pak")
- Some models don't have separate skeleton files (like infected), instead the skeleton is inside the same file. In this case select the first pak file again.
- If you don't want the skeleton, or if you are loading a map, click "Cancel" for the second file select

The program will go through the file, finding meshes/skeleton and add them to the scene.

You can see the meshes that are currently in the scene, in a list box on the left. If you select a mesh from the list, it will get highlighted in the scene. Bones will also appear as white spheres but it is not possible to move/rotate the bones. They are just there to show you that the skeleton is loaded properly.

Normally the meshes should load fine. If for some reason one of the meshes appear distorted you can just disable it by checking the "Disable Mesh" box. Disabled meshes won't get exported.

Preview Controls
Right-click and move mouse : Orbit around model
Ctrl + Mouse Wheel Scroll : Zoom
Mouse Wheel Click and move mouse : Pan
W Key : Wireframe shader

Exporting
There are two export options:

Obj : This will export all meshes as separate obj files. Of course there will be no skinning data and skeleton.
IQE : This will export the model as a single iqe file. IQE isn't really popular but it is a format that is not complex and that supports skinning, so I went with it.
There is a blender importer script for iqe : https://raw.githubusercontent.com/ccxvi ... _import.py From blender you can pretty much export it to any format you want.

Download: http://www.mediafire.com/file/0b9i8nat4 ... r_Upd2.rar

Example model imported into Maya:
Image

I tried loading some files other than the characters, like props, weapons, animals etc. and they all seemed to work fine with minor problems. Still there might be issues with some of the files, because I didn't tested every one of them. If that is the case, let me know.

If you are also interested in getting the textures, check out my texture tool : viewtopic.php?f=18&t=17800

Credits
daemon1 - for figuring out all of the model data and explaining it really well with videos
luxox18 - for finding out data manually, so that I can use them as a reference

You can make the ads go away by registering



Last edited by akderebur on Sun May 13, 2018 2:33 pm, edited 6 times in total.

Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Mon Feb 13, 2017 3:18 pm 
Offline
advanced

Joined: Sat Oct 23, 2010 3:50 pm
Posts: 57
Has thanked: 2 times
Have thanks: 2 times
wow ,this is realy awsome!!


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Thu Feb 16, 2017 2:05 am 
Offline
advanced

Joined: Tue Nov 06, 2012 7:57 am
Posts: 76
Has thanked: 8 times
Have thanks: 1 time
Will this work on uncharted 4?


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Thu Feb 16, 2017 2:19 am 
Offline
mega-veteran
mega-veteran

Joined: Fri Jul 08, 2011 10:36 am
Posts: 194
Has thanked: 26 times
Have thanks: 148 times
Jaydenthetank wrote:
Will this work on uncharted 4?

I don't know, it might work if the file structure is identical with the last of us. I don't have the uncharted files, so didn't have the chance to test them.


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Thu Feb 16, 2017 6:53 am 
Offline
advanced

Joined: Tue Nov 06, 2012 7:57 am
Posts: 76
Has thanked: 8 times
Have thanks: 1 time
Just curious because I don't think anyone has ripped ps4 models before


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Thu Feb 16, 2017 11:21 am 
Offline
double-veteran
double-veteran

Joined: Mon Jul 13, 2009 6:16 pm
Posts: 690
Has thanked: 70 times
Have thanks: 114 times
Jaydenthetank wrote:
Just curious because I don't think anyone has ripped ps4 models before


https://www.youtube.com/watch?v=-Y1bigdq-6o&t

_________________
Image


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Sun Feb 19, 2017 12:10 am 
Offline
ultra-veteran
ultra-veteran
User avatar

Joined: Sun Jul 06, 2014 6:30 am
Posts: 388
Has thanked: 86 times
Have thanks: 254 times
akderebur wrote:
Jaydenthetank wrote:
Will this work on uncharted 4?

I don't know, it might work if the file structure is identical with the last of us. I don't have the uncharted files, so didn't have the chance to test them.


Uncharted 4 is something I started looking into. But the whole game is just one single pak file, so I suspect it's not similar at all. EDIT: I found out it's because PS4 games are encrypted, and still nobody knows how to unencrypt them (which begs the question how you unencrypted this game?)

I am curious if you can release the source code or futher explain the findings, as I'm sure it would help with my own investigations, the more we can pool information, the farther we can go.


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Sun Feb 19, 2017 2:43 am 
Offline
mega-veteran
mega-veteran

Joined: Fri Jul 08, 2011 10:36 am
Posts: 194
Has thanked: 26 times
Have thanks: 148 times
volfin wrote:
Uncharted 4 is something I started looking into. But the whole game is just one single pak file, so I suspect it's not similar at all. EDIT: I found out it's because PS4 games are encrypted, and still nobody knows how to unencrypt them (which begs the question how you unencrypted this game?)

I did not, luxox is the guy you are looking for :D

He released some character files in this thread : viewtopic.php?f=10&t=15801


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Sun Feb 19, 2017 6:55 am 
Offline
ultra-veteran
ultra-veteran
User avatar

Joined: Tue Jan 04, 2005 9:47 pm
Posts: 585
Has thanked: 43 times
Have thanks: 112 times
[out]


Last edited by Wobble on Sat Mar 11, 2017 11:17 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Sun Feb 19, 2017 8:01 am 
Offline
M-M-M-Monster veteran
M-M-M-Monster veteran
User avatar

Joined: Fri Apr 20, 2012 9:24 am
Posts: 2313
Location: Nexus, searching for Jim Kirk
Has thanked: 524 times
Have thanks: 1199 times
[well, I've been told, my guess is wrong for PS4, does apply for PS3 models only]

Guess, it's also a matter of Sony's edge compression:
viewtopic.php?f=16&t=12070&p=99284#p99284

Only tool I know (so far) that can handle it is Noesis:
rapi.rpgCommitTriangles(edgeDecomp, noesis.RPGEODATA_USHORT, indexCount, noesis.RPGEO_TRIANGLE, 1)

But detailed informations are kept secret.
That's sad because more infos could help with Gran Turismo 6 Models, too.

I tend to ignore edge decompression, because it's so boring. I remember fiddling around with Sony's edge SDK (or something like that, not sure).
Just a waste of time and life... :D

_________________
"you can't always get things served on a silver tray"
extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-jm9.zip

"We are Microsoft. You will be assimilated. Resistance is Futile."


Last edited by shakotay2 on Mon Feb 20, 2017 7:28 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Sun Feb 19, 2017 12:32 pm 
Offline
M-M-M-Monster veteran
M-M-M-Monster veteran

Joined: Tue Mar 24, 2015 8:12 pm
Posts: 1567
Has thanked: 29 times
Have thanks: 994 times
akderebur wrote:
I did not, luxox is the guy you are looking for


luxox will not be able to help you. Hes just the guy who delivers decrypted files from the scene. He can't decrypt them.


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Sun Feb 19, 2017 1:49 pm 
Offline
mega-veteran
mega-veteran

Joined: Fri Jul 08, 2011 10:36 am
Posts: 194
Has thanked: 26 times
Have thanks: 148 times
Wobble wrote:
Viewers and characters are nice, but as a coder, we would like to see some discussion on formats, algorithms, or code.

There is no real algorithm that can bring you further, it is just like pattern matching program designed specifically for The Last of Us pak files. Let me explain how the program works and you will see what I mean.

The way I did it was I basically opened bunch of character files in the hex editor and tried to find out recurring patterns in each file. Here is an example for getting vertex/index count and index offset.

There are 4 patterns for this (in hex) :
  • 03 00 00 00 00 00 00 00 00 00 00 00
  • 04 00 00 00 00 00 00 00 00 00 00 00
  • 05 00 00 00 00 00 00 00 00 00 00 00
  • 06 00 00 00 00 00 00 00 00 00 00 00

Also after 44 bytes from the start of the above patterns, there should be this pattern : FFFFFFFF.

So lets say the file contains 10 meshes. The program starts and searches for these patterns in the file and when there is a match it adds the offset to a list. When it is done there are like 3000 (I am making up numbers for this example) records in the list. It then goes through the list, for each offset it advances 44 bytes and checks if there is FFFFFFFF. If it finds out that pattern, it adds that offset (end of FFFFFFFF) – 56 bytes to a new list. The reason for offset - 56 bytes is because the data we are looking for actually starts there, again something that I found out looking at the hex editor. In the end that list has 10 records with correct offsets.

Now we can read the actual data. The program goes through each offset in the new list :
  • binary reader’s position is set to the current offset
  • first integer read = vertex count
  • second integer read = index count
  • advance 28 bytes
  • integer read = index offset

I think at this point you can see that the program is a product of binary search and pattern matching and that it doesn’t use a proper way to find data. It is like a bruteforcer for finding data in tlou files only. I find it hard to believe that anyone can develop it further to let’s say work with uncharted 4 files. If the file structure is identical, it will work. If not you would need new patterns and it would be a whole new program in that case.

The program as it is contains more code for graphics and ui programming than the actual code for reading the pak file and it is a bit messy :D. If you want I can strip the Unity stuff and share the part related to pak files.

shakotay2 wrote:
Guess, it's also a matter of Sony's edge compression

Mesh data (index, vertex, uv) is not compressed in the ps4 files. Only the ps3 version has edge compression.


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Mon Feb 20, 2017 12:37 am 
Offline
ultra-veteran
ultra-veteran
User avatar

Joined: Tue Jan 04, 2005 9:47 pm
Posts: 585
Has thanked: 43 times
Have thanks: 112 times
[out]


Last edited by Wobble on Sat Mar 11, 2017 11:17 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Mon Feb 20, 2017 2:22 am 
Offline
mega-veteran
mega-veteran

Joined: Fri Jul 29, 2011 9:18 pm
Posts: 176
Has thanked: 54 times
Have thanks: 41 times
ps4 developers are not using the edge compression anymore. Instead are using swizzled textures and models with same structure as PC with little endian data. (nothing compressed)

in this game specifically the vertex data and the uv data are separated and all faces start with 00 00 01 00 02.

not all games can be decrypted, for the moment is possible in games with firmware 1.76 (2013 - 2014 games)

oh, and about how to obtain decrypted games.. I just have a contact in the ps4 scene


Top
 Profile  
 
 Post subject: Re: Last of Us Model Viewer/Extractor (PS4)
PostPosted: Mon Feb 20, 2017 11:14 am 
Offline
ultra-veteran
ultra-veteran
User avatar

Joined: Tue Jan 04, 2005 9:47 pm
Posts: 585
Has thanked: 43 times
Have thanks: 112 times
[out]



Last edited by Wobble on Sat Mar 11, 2017 11:17 am, edited 1 time in total.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 104 posts ]  Go to page 1, 2, 3, 4, 5 ... 7  Next

All times are UTC + 1 hour


Who is online

Users browsing this forum: dracosfire83 and 7 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group