Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archive

The Original Forum. Game archives, full of resources. How to open them? Get help here.
emoose
beginner
Posts: 26
Joined: Thu Sep 10, 2015 9:06 pm
Has thanked: 3 times
Been thanked: 24 times

Re: Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archiv

Post by emoose »

Alright, figured out how those LUAs are crypted, after spending hours trying to unpack GZ just so I could attach a debugger to it >.<

A CityHash (using the old GZ algo) is made from the filename of the LUA file, including file extension. The NOT of this hash is used as the XOR key for the file, the key also changes as the file is decrypted.

Some code I wrote up quick which should explain it better:

Code: Select all

public byte[] DecryptFPKLUAFile(string filePath)
{
    var filename = Path.GetFileName(filePath).ToLower();
    var hash = HashFileNameLegacy(filename, false);
    var key = BitConverter.GetBytes(~hash);

    var fileData = File.ReadAllBytes(filePath);
    var decData = new byte[fileData.Length - 1];

    for(int i = 0; i < fileData.Length - 1; i++)
    {
        key[i & 7] ^= fileData[i + 1];
        decData[i] = key[i & 7];
    }

    return decData;
}
The last byte of the file should be 0x00 if it decrypted properly :)

Going to look into adding this into GzsTool later, AFAIK the reverse isn't needed since the game should run fine with decrypted LUAs.
A lot of these LUAs also mention other FPK filepaths too, so we might have a big update to the dictionary soon :)

EDIT: Added to my fork, when extracting from an FPK it'll automatically try decrypting any files that look encrypted. You can download it from https://github.com/emoose/GzsTool/relea ... Decryption

EDIT2: Just pushed a quick fix for some files not decrypting properly, grab it from https://github.com/emoose/GzsTool/relea ... Decryption

EDIT3: Another quick fix (hopefully the last!) for QARs not extracting properly: https://github.com/emoose/GzsTool/relea ... Decryption
Last edited by emoose on Thu Oct 01, 2015 4:58 am, edited 1 time in total.
JRavens
beginner
Posts: 20
Joined: Mon Dec 22, 2014 10:17 pm
Has thanked: 10 times
Been thanked: 3 times

Re: Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archiv

Post by JRavens »

emoose wrote:Alright, figured out how those LUAs are crypted, after spending hours trying to unpack GZ just so I could attach a debugger to it >.<

A CityHash (using the old GZ algo) is made from the filename of the LUA file, including file extension. The NOT of this hash is used as the XOR key for the file, the key also changes as the file is decrypted.

Some code I wrote up quick which should explain it better:

Code: Select all

public byte[] DecryptFPKLUAFile(string filePath)
{
    var filename = Path.GetFileName(filePath).ToLower();
    var hash = HashFileNameLegacy(filename, false);
    var key = BitConverter.GetBytes(~hash);

    var fileData = File.ReadAllBytes(filePath);
    var decData = new byte[fileData.Length - 1];

    for(int i = 0; i < fileData.Length - 1; i++)
    {
        key[i & 7] ^= fileData[i + 1];
        decData[i] = key[i & 7];
    }

    return decData;
}
The last byte of the file should be 0x00 if it decrypted properly :)

Going to look into adding this into GzsTool later, AFAIK the reverse isn't needed since the game should run fine with decrypted LUAs.
A lot of these LUAs also mention other FPK filepaths too, so we might have a big update to the dictionary soon :)

EDIT: Added to my fork, when extracting from an FPK it'll automatically try decrypting any files that look encrypted. You can download it from https://github.com/emoose/GzsTool/relea ... Decryption

EDIT2: Just pushed a quick fix for some files not decrypting properly, grab it from https://github.com/emoose/GzsTool/relea ... Decryption
emoose you are a scholar and a gentleman! Thank you mate (again) :)
HeartlessSeph
advanced
Posts: 72
Joined: Tue Sep 15, 2015 2:13 am
Has thanked: 2 times
Been thanked: 6 times

Re: Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archiv

Post by HeartlessSeph »

emoose wrote:Alright, figured out how those LUAs are crypted, after spending hours trying to unpack GZ just so I could attach a debugger to it >.<

A CityHash (using the old GZ algo) is made from the filename of the LUA file, including file extension. The NOT of this hash is used as the XOR key for the file, the key also changes as the file is decrypted.

Some code I wrote up quick which should explain it better:

Code: Select all

public byte[] DecryptFPKLUAFile(string filePath)
{
    var filename = Path.GetFileName(filePath).ToLower();
    var hash = HashFileNameLegacy(filename, false);
    var key = BitConverter.GetBytes(~hash);

    var fileData = File.ReadAllBytes(filePath);
    var decData = new byte[fileData.Length - 1];

    for(int i = 0; i < fileData.Length - 1; i++)
    {
        key[i & 7] ^= fileData[i + 1];
        decData[i] = key[i & 7];
    }

    return decData;
}
The last byte of the file should be 0x00 if it decrypted properly :)

Going to look into adding this into GzsTool later, AFAIK the reverse isn't needed since the game should run fine with decrypted LUAs.
A lot of these LUAs also mention other FPK filepaths too, so we might have a big update to the dictionary soon :)

EDIT: Added to my fork, when extracting from an FPK it'll automatically try decrypting any files that look encrypted. You can download it from https://github.com/emoose/GzsTool/relea ... Decryption

EDIT2: Just pushed a quick fix for some files not decrypting properly, grab it from https://github.com/emoose/GzsTool/relea ... Decryption
You and cra0kalo are making me far too hyped 8D
JRavens
beginner
Posts: 20
Joined: Mon Dec 22, 2014 10:17 pm
Has thanked: 10 times
Been thanked: 3 times

Re: Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archiv

Post by JRavens »

emoose wrote:EDIT2: Just pushed a quick fix for some files not decrypting properly, grab it from https://github.com/emoose/GzsTool/relea ... Decryption
Hmmm... I extracted chunk0.dat with no problem, but any fpk or fpkd from it I try does not extract at all and just generates an empty .xml:

EDIT: Nvm the problem was on my end. Either something to do with file paths being too long or rights management on that drive's files or something... not sure, but when I moved the files to another drive to work on them the problem went away. Apologies.
emoose
beginner
Posts: 26
Joined: Thu Sep 10, 2015 9:06 pm
Has thanked: 3 times
Been thanked: 24 times

Re: Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archiv

Post by emoose »

JRavens wrote:
emoose wrote:EDIT2: Just pushed a quick fix for some files not decrypting properly, grab it from https://github.com/emoose/GzsTool/relea ... Decryption
Hmmm... I extracted chunk0.dat with no problem, but any fpk or fpkd from it I try does not extract at all and just generates an empty .xml:

EDIT: Nvm the problem was on my end. Either something to do with file paths being too long or rights management on that drive's files or something... not sure, but when I moved the files to another drive to work on them the problem went away. Apologies.
Actually there was a problem with my code too, it wasn't decompressing files inside PC QARs/chunks as I left a small bit out, thanks for letting me know since I probably wouldn't have noticed otherwise.

Fixed version can be grabbed from https://github.com/emoose/GzsTool/relea ... Decryption :)
User avatar
Higus
advanced
Posts: 78
Joined: Sat Jun 30, 2012 5:35 am
Has thanked: 6 times
Been thanked: 5 times

Re: Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archiv

Post by Higus »

emoose wrote:Alright, figured out how those LUAs are crypted, after spending hours trying to unpack GZ just so I could attach a debugger to it >.<

A CityHash (using the old GZ algo) is made from the filename of the LUA file, including file extension. The NOT of this hash is used as the XOR key for the file, the key also changes as the file is decrypted.

Some code I wrote up quick which should explain it better:

Code: Select all

public byte[] DecryptFPKLUAFile(string filePath)
{
    var filename = Path.GetFileName(filePath).ToLower();
    var hash = HashFileNameLegacy(filename, false);
    var key = BitConverter.GetBytes(~hash);

    var fileData = File.ReadAllBytes(filePath);
    var decData = new byte[fileData.Length - 1];

    for(int i = 0; i < fileData.Length - 1; i++)
    {
        key[i & 7] ^= fileData[i + 1];
        decData[i] = key[i & 7];
    }

    return decData;
}
The last byte of the file should be 0x00 if it decrypted properly :)

Going to look into adding this into GzsTool later, AFAIK the reverse isn't needed since the game should run fine with decrypted LUAs.
A lot of these LUAs also mention other FPK filepaths too, so we might have a big update to the dictionary soon :)
This is great! Now we just need to find out where the damn Camera lua files are located... Anybody found anything? I've been searching my ass off for these but haven't found a trace of their existence, at least not the way they're supposed to be named/placed as in Ground Zeroes...
Tex
veteran
Posts: 89
Joined: Sat Aug 20, 2011 4:51 am
Has thanked: 21 times
Been thanked: 11 times

Re: Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archiv

Post by Tex »

emoose wrote:Alright, figured out how those LUAs are crypted,
Great work.

emoose, have you generated any of the fpk names for the dictionary using the methods used in TppPacklist.lua:MakeDefaultMissionAreaPackPath and AddFOBLayoutPack? Couldn't be sure since you have enough of the fpks that I didn't check if any were missing. If not I could post a slightly more readable breakdown for you to start from.

Also has anyone had success in using GzsTool to package a qar that the game actually accepts?
Been building my mod with MGSV_QAR_Tool fine after my initial lack of success with gzstool back when the patch with 01.dat came out.
Decided to give gzstool another go now. It builds the 01.dat fine, same size as MGSV_QAR_Tools, but loading the game there's no evidence of the game using the files.
Last edited by Gh0stBlade on Sat Sep 24, 2016 4:32 pm, edited 1 time in total.
Reason: Double post merge.
atvaark
advanced
Posts: 52
Joined: Fri Aug 28, 2015 3:19 pm
Been thanked: 47 times

Re: Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archiv

Post by atvaark »

I was on vacation for the last few weeks and haven't much time to look at the issues.
I'll take a look at what's different in the QAR tool and I will also merge lua decryption while I'm at it.
Topher
n00b
Posts: 12
Joined: Thu Oct 01, 2015 1:08 pm
Has thanked: 4 times
Been thanked: 9 times

Re: Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archiv

Post by Topher »

emoose wrote:Alright, figured out how those LUAs are crypted, ...
You are a genius. Atvaark and yourself are the kind of people that inspire me to continue learning to code.

You've both done a great service to the community, and we thank you.

Edit: Might I suggest we get a wiki going with all of the inner workings of the fox data structure? I have learned a lot from looking at the code for GzsTool but I think it would be good to create some reference documentation to go alongside future development :)
User avatar
Higus
advanced
Posts: 78
Joined: Sat Jun 30, 2012 5:35 am
Has thanked: 6 times
Been thanked: 5 times

Re: Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archiv

Post by Higus »

atvaark wrote:I was on vacation for the last few weeks and haven't much time to look at the issues.
I'll take a look at what's different in the QAR tool and I will also merge lua decryption while I'm at it.
I've two questions I'd love answered;
(FtexTool) 1. Could you tell me how I can get proper 1024 or 2048 resolution textures to convert to 3 ftexs instead of 5 ftexs?
(GzsTool) 2. Can we get a way to repack new files in pftxs without having their hashes?

Being able to add textures to a pftxs is possible after finding their hashes from extracting texture qars, but converting 1024 or 2048 res textures with FtexTool will output into 5 ftexs, when only three originally exist that can be included with their hashes from the texture qar (so we can only get in custom textures at 512x512 from 3 out of 5 ftexs)

If either questions have a solution, then they cancel the other one out as an issue.
atvaark
advanced
Posts: 52
Joined: Fri Aug 28, 2015 3:19 pm
Been thanked: 47 times

Re: Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archiv

Post by atvaark »

Higus wrote:
atvaark wrote:I was on vacation for the last few weeks and haven't much time to look at the issues.
I'll take a look at what's different in the QAR tool and I will also merge lua decryption while I'm at it.
I've two questions I'd love answered;
(FtexTool) 1. Could you tell me how I can get proper 1024 or 2048 resolution textures to convert to 3 ftexs instead of 5 ftexs?
(GzsTool) 2. Can we get a way to repack new files in pftxs without having their hashes?

Being able to add textures to a pftxs is possible after finding their hashes from extracting texture qars, but converting 1024 or 2048 res textures with FtexTool will output into 5 ftexs, when only three originally exist that can be included with their hashes from the texture qar (so we can only get in custom textures at 512x512 from 3 out of 5 ftexs)

If either questions have a solution, then they cancel the other one out as an issue.
I haven't figured out how the ftexs file count relates to the width/height/area/mip-maps/etc yet. All I have is a huge spreadsheet with the above information that I created from all the textures.

Adding files to archives is something I'd like to add in the future. Adding files to QAR archives is almost done. There are some still issues with creating hashes from new files. When I'm done with that I'll implement adding files to fpk and pftxs files.
User avatar
Higus
advanced
Posts: 78
Joined: Sat Jun 30, 2012 5:35 am
Has thanked: 6 times
Been thanked: 5 times

Re: Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archiv

Post by Higus »

atvaark wrote:
Higus wrote:
atvaark wrote:I was on vacation for the last few weeks and haven't much time to look at the issues.
I'll take a look at what's different in the QAR tool and I will also merge lua decryption while I'm at it.
I've two questions I'd love answered;
(FtexTool) 1. Could you tell me how I can get proper 1024 or 2048 resolution textures to convert to 3 ftexs instead of 5 ftexs?
(GzsTool) 2. Can we get a way to repack new files in pftxs without having their hashes?

Being able to add textures to a pftxs is possible after finding their hashes from extracting texture qars, but converting 1024 or 2048 res textures with FtexTool will output into 5 ftexs, when only three originally exist that can be included with their hashes from the texture qar (so we can only get in custom textures at 512x512 from 3 out of 5 ftexs)

If either questions have a solution, then they cancel the other one out as an issue.
I haven't figured out how the ftexs file count relates to the width/height/area/mip-maps/etc yet. All I have is a huge spreadsheet with the above information that I created from all the textures.

Adding files to archives is something I'd like to add in the future. Adding files to QAR archives is almost done. There are some still issues with creating hashes from new files. When I'm done with that I'll implement adding files to fpk and pftxs files.
Sweet, thanks for the prompt reply.

I'd like to add though that I have found a nice workaround for anybody interested; If you save your new dds with NO mipmaps, then you may convert it to have just one ftexs and the new ftex+ftexs will ignore any of the other original ftexs from the texture qars, so that your new textures may be loaded at max resolution within the convenience of a pftxs (no need to go hash-digging for adding extra ftexs files!)
Image
HeartlessSeph
advanced
Posts: 72
Joined: Tue Sep 15, 2015 2:13 am
Has thanked: 2 times
Been thanked: 6 times

Re: Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archiv

Post by HeartlessSeph »

Higus wrote: Sweet, thanks for the prompt reply.

I'd like to add though that I have found a nice workaround for anybody interested; If you save your new dds with NO mipmaps, then you may convert it to have just one ftexs and the new ftex+ftexs will ignore any of the other original ftexs from the texture qars, so that your new textures may be loaded at max resolution within the convenience of a pftxs (no need to go hash-digging for adding extra ftexs files!)
Image
Wow, that's an awesome yet simple fix. There's been no downside to this has there?
Tex
veteran
Posts: 89
Joined: Sat Aug 20, 2011 4:51 am
Has thanked: 21 times
Been thanked: 11 times

Re: Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archiv

Post by Tex »

Topher wrote: Edit: Might I suggest we get a wiki going with all of the inner workings of the fox data structure? I have learned a lot from looking at the code for GzsTool but I think it would be good to create some reference documentation to go alongside future development :)
Been thinking about a MGSV modding wiki in general for a while, forums aren't the best repository for information. Nexus automatically hosts a wiki for each game it hosts files for, so there's that if no one has a better host?
Sergeanur
advanced
Posts: 53
Joined: Fri Dec 30, 2011 12:26 pm
Been thanked: 37 times

Re: Metal Gear Solid 5 Ground Zeroes/Phantom Pain g0s archiv

Post by Sergeanur »

Minor updates:
MGSV QAR Tool v1.3.2
MGSV FPK Tool v1.2
* A bug which caused drag n' drop do nothing should be fixed.
* LUA decryption/encryption added to FPK Tool.
Use -n option to leave data unencrypted and work with older inf format (not tested!!!)
Post Reply