XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Tue Oct 17, 2017 4:19 pm

All times are UTC + 1 hour


Forum rules


Please click here to view the forum rules



Post new topic Reply to topic  [ 118 posts ]  Go to page 1, 2, 3, 4, 5 ... 8  Next
Author Message
 Post subject: Nier Automata
PostPosted: Fri Mar 17, 2017 8:38 pm 
Offline
n00b

Joined: Thu Dec 23, 2010 9:49 am
Posts: 19
Has thanked: 4 times
Have thanks: 13 times

Platinum is using CriWare's "cpk" archive format again.

Can be easily opened with https://github.com/wmltogether/CriPakTools

Contains the usual dat and dtt and so forth just like some of their other games though some things might be updated.

I was able to extract textures but some are somewhat "corrupted"(or are just aux data like directional lightmaps)

Image


Sample file: https://transfer.sh/13foZS/um2140.dat

There's some other file formats contained that could be mapped out. Some of the CPKs contain a lot of wd_ files(world data. Much of it procedurally generated using Simplygon with not much else going on.

data012.cpk contains a lot of quest and scripting data and the licenses involve a lot of ruby so there might be a huge repo of ruby scripts somewhere among the assets.

the .z files found within the .dat files are predictably zlib. An example of a decompressed "q171_hap.z" file found within "q171.dat" which is within "data012.cpk"(so many nested levels of files...)
http://pastebin.com/raw/0rJZ4CKH

pak files are pretty much containers for multiple zip streams.

Found all around the archives are what looks like "compiled" ruby code. Constants and some signs of syntax are exposed here and there. Probably pre-lexicographed ruby code to avoid storing pure plaintext.
Image
Very likely it's just pre-compiled mruby VM bytecode output from mrbc.
https://raw.githubusercontent.com/mruby ... y/opcode.h

Bin files found in quest data such as 'q171_12f2963a_scp.bin' is for sure output from mrbc. (the ones with the `RITE0003` header)
https://github.com/mruby/mruby/blob/08a ... dump.h#L53

Some of the middleware that Nier uses:
https://github.com/mruby/mruby
https://github.com/antimon2/enumerable_ ... /README.md
https://www.musl-libc.org/
http://rake.rubyforge.org/
https://github.com/yhara/enumerable-lazy
https://www.simplygon.com/
http://www.geomerics.com/enlighten/

Made a quick extractor for the .dat file files.
https://gist.github.com/Wunkolo/213aa61 ... ebb8ab89c2

You can make the ads go away by registering


_________________
Aka Wunkolo


Top
 Profile  
 
 Post subject: Re: Nier Automata
PostPosted: Sat Mar 18, 2017 10:34 pm 
Offline
n00b

Joined: Thu Dec 23, 2010 9:49 am
Posts: 19
Has thanked: 4 times
Have thanks: 13 times
Model format ".wmb" has been changed. Magic is "WMB3" which is weird because some of their older games used "WMB4". Format uses a deeply nested table format similar to WMB4 somewhat. Seems to include a full map of uniform data(passed to the shader) and parameters and such. Material parameters are all floats where applicable and zero otherwise(such as for texture inputs). Header is 0x90 bytes. Writing a reader for it now while I map out all the table types and I'll use something like fbxsdk to write an exporter most likely.

Some sample wmb files
https://transfer.sh/gSpXc/sample.zip

_________________
Aka Wunkolo


Top
 Profile  
 
 Post subject: Re: Nier Automata
PostPosted: Sat Mar 18, 2017 11:38 pm 
Offline
veteran

Joined: Sat Apr 19, 2014 8:02 am
Posts: 87
Has thanked: 42 times
Have thanks: 10 times
Glad to see folks working on this game. Have you seen (or already taking part in) the thread on the facepunch forums for Nier? They've been discussing and working on the format a bit. If you haven't seen it, here's the link

https://facepunch.com/showthread.php?t=1557100

Looking forward to progress :D


Top
 Profile  
 
 Post subject: Re: Nier Automata
PostPosted: Sun Mar 19, 2017 12:06 am 
Offline
advanced

Joined: Tue Oct 28, 2014 7:51 am
Posts: 53
Has thanked: 3 times
Have thanks: 4 times
So as far as my comprehension goes dtt format is basically DDS embedded with some sort of overhead?
I'm looking into it cause some of the textures could really use some high definition love.


Top
 Profile  
 
 Post subject: Re: Nier Automata
PostPosted: Sun Mar 19, 2017 12:25 am 
Offline
n00b

Joined: Thu Dec 23, 2010 9:49 am
Posts: 19
Has thanked: 4 times
Have thanks: 13 times
ssringo wrote:
Glad to see folks working on this game. Have you seen (or already taking part in) the thread on the facepunch forums for Nier? They've been discussing and working on the format a bit. If you haven't seen it, here's the link

https://facepunch.com/showthread.php?t=1557100

Looking forward to progress :D


The WMB format is entirely different from anything any tool out there uses. It's a new format specifically for the new rendering engine they have set up and it's entirely different from the most recent "WMB" iteration seen here(http://pastebin.com/CEHriQBv) which is what most tools out there use and are totally incompatible with the latest format. Here's a snippet from the reader I am working on right now. Lots of metaprogramming elsewhere as this is just a small snippet but this is very likely to be what Platinum games internally does as well. Commented with some of my research notes. I pretty much mapped out the Materials/Uniform parameters(values that get passed to the shader) among other things. 2B's internal name is pl0000 even in the ruby bytecode.

Code:
template< typename EntryType = void >
struct TableReference
{
   typedef EntryType ElementType;
   std::uint32_t Offset;
   std::uint32_t Count;
};

struct Header
{
   std::uint32_t Magic; // WMB3
   std::uint32_t Unknown4; // Version?

   std::uint32_t Unknown8; // 0

   std::uint16_t UnknownC; // 0x0, 0xA
   std::uint16_t UnknownE; // 0xffff

   Vector3 UnknownVector10;
   Vector3 UnknownVector1C;

   TableReference<UnknownTable28Entry> UnknownTable28; // Table of 0x60 byte structures

   TableReference<std::uint8_t> UnknownTable30; // Table of 1 byte structures

   TableReference<UnknownTable38Entry> UnknownTable38; // Table of 0x30 byte structures

   TableReference<void> UnknownTable40; // Table of 0x20 byte structures

   TableReference<void> LODTable; // Table of 0x1C byte structures

   TableReference<void> UnknownTable50; // Often Zero

   TableReference<std::uint32_t> UnknownTable58;

   TableReference<
      TableReference<std::uint16_t>
   > UnknownTable60; // Table of... Table References to tables of uint16_t

   TableReference<MaterialParamTableEntry> MaterialParamTable; // Table of 0x30 byte structures

   TableReference<UnknownTable70Entry> UnknownTable70; // Table of 0x30 byte structures

   TableReference<void> UnknownTable78; // Table of {uint32_t Zero, uint32_t Unknown } structures
};



That person is using Ninjaripper which basically middle-mans between the game and directx driver so it won't have a lot of the auxiliary information to get things like rigging and texture names and so forth.


SkacikPL wrote:
So as far as my comprehension goes dtt format is basically DDS embedded with some sort of overhead?
I'm looking into it cause some of the textures could really use some high definition love.


DTT files are DAT files intended to store texture(.wtp) and model(.wmb) data among others(bxm,mot,sop,etc). .wtp files are regular DirectX textures(.dds) and can simply be renamed to a .dds extension.

Image

_________________
Aka Wunkolo


Top
 Profile  
 
 Post subject: Re: Nier Automata
PostPosted: Sun Mar 19, 2017 2:52 am 
Offline
n00b

Joined: Thu Dec 23, 2010 9:49 am
Posts: 19
Has thanked: 4 times
Have thanks: 13 times
Current structure so far:

Code:
#pragma once
#include <cstdint>
#include <cstddef>

namespace WMB
{
#pragma pack(push, 1)

template<size_t Dimension>
struct Vector
{
   std::float_t f32[Dimension];
};

using Vector4 = Vector<4>;
using Vector3 = Vector<3>;
using Vector2 = Vector<2>;

template< typename EntryType >
struct TableReference
{
   typedef EntryType ElementType; // For reading functions
   std::uint32_t Offset;
   std::uint32_t Count;
};

struct StringReference
{
   std::uint32_t Offset; // Offset for null-terminated string
};

struct UnknownTable28Entry
{
   struct
   {
      // Probably Very wrong
      Vector4 UnknownVector0;
      Vector2 UnknownVector10;
      Vector2 UnknownVector18;
   } Elements[3];
};

struct GeoStreamTableEntry
{
   struct UnknownTable0Entry
   {
      Vector3 UnknownVector0; // Position?
      Vector2 UnknownVectorC; // UV?

      std::uint32_t Unknown18; // Zero
      std::uint32_t Unknown1C;
   };

   TableReference<UnknownTable0Entry> UnknownTable0; // Possible vertex stream

   TableReference<void> UnknownTable8; // Tends to be 0

   std::uint32_t VertexStride10; // Stride for UnknownTable0?
   std::uint32_t VertexStride14; // Stride

   TableReference<void> UnknownTable18; // Tends to be 0

   std::uint32_t Unknown20; // Vertex Count?
   std::uint32_t Unknown24;
   TableReference<void> IndexTable28; // Possible indice array
};

struct MaterialParamTableEntry
{
   std::uint32_t Unknown0;
   std::uint16_t Unknown4;
   std::uint16_t Unknown6;

   StringReference ParamMeshName; // Null Terminated String (ex: PL_0000_Eye)
   StringReference ParamMeshClass; // Null Terminated String (ex: Eye00_XXXXX)
   StringReference ParamsName; // Null Terminated String (ex: Default)

   std::uint32_t Unknown14; // Usually 1

   struct ParamTableEntry
   {
      StringReference UniformName; // Null terminated (ex: g_AlbedoMap)
      std::float_t Unknown4; // Uniform Value(float, 0 if not applicable such as for a texture)
   };

   TableReference<ParamTableEntry> ParamTable;

   struct UnknownTable20Entry
   {
      std::uint32_t Unknown0; // (ex: 0, 2)
      std::uint32_t Unknown4; // (ex: 2DA470h, FFFFFFFFh)
   };

   TableReference<UnknownTable20Entry> UnknownTable20;

   TableReference<ParamTableEntry> VertexShaderParamTable;

   // Same structure as "ParamTableEntry" above
   // (ex: Binormal0, Color0, Normal, Position, Tangent, TexCoord0,
   //      g_AlbedoColor_X, g_ReflectionIntensity, g_Metallic, etc )
};

struct MeshGroupTableEntry
{
   StringReference GroupName; // (ex: Body)

   Vector3 UnknownVector4;
   Vector3 UnknownVector10;

   // Possibly face indexes
   TableReference<std::uint16_t> UnknownTable1C; // Table uint16_t
   TableReference<std::uint16_t> UnknownTable24; // Table of uint16_t
};

struct UnknownTable78Entry
{
   std::uint32_t Unknown0; // 0x0
   std::uint32_t UnknownIndex4;
};

///////////////////////////////////////////////////////////////////////////////

struct Header
{
   std::uint32_t Magic; // WMB3
   std::uint32_t Unknown4; // Version?

   std::uint32_t Unknown8; // 0

   std::uint16_t UnknownC; // 0x0, 0xA
   std::int16_t UnknownE; // 0xffff

   Vector3 UnknownVector10;
   Vector3 UnknownVector1C;

   TableReference<UnknownTable28Entry> BoneTable28; // Table of 0x60 byte structures

   TableReference<std::int8_t> UnknownTable30; // Table of 1 byte structures

   TableReference<GeoStreamTableEntry> GeoStreamTable38; // Table of 0x30 byte structures

   TableReference<void> UnknownTable40; // Table of 0x20 byte structures

   // Possibly LOD related
   TableReference<void> UnknownTable48; // Table of 0x1C byte structures

   TableReference<void> UnknownTable50; // Zero

   TableReference<std::uint32_t> UnknownTable58; // Table of uint_32

   TableReference<
      TableReference<std::uint16_t>
   > UnknownTable60; // Table of... Table References to tables of uint16_t

   TableReference<MaterialParamTableEntry> MaterialParamTable; // Table of 0x30 byte structures

   // Named/Grouped chunks of geometry
   TableReference<MeshGroupTableEntry> MeshGroupTable; // Table of 0x30 byte structures

   TableReference<UnknownTable78Entry> UnknownTable78; // Table of { uint32_t Zero, uint32_t Unknown } structures

   TableReference<void> UnknownTable80; // Zero
   TableReference<void> UnknownTable88; // Zero
};

#pragma pack(pop)
}

_________________
Aka Wunkolo


Top
 Profile  
 
 Post subject: Re: Nier Automata
PostPosted: Sun Mar 19, 2017 4:42 pm 
Offline
beginner
User avatar

Joined: Wed Jul 22, 2015 10:08 pm
Posts: 23
Has thanked: 5 times
Have thanks: 3 times
Good work...
Anyway this https://gist.github.com/Wunkolo/213aa61 ... ebb8ab89c2
Have any code to recreate the .dat? i'm not a c++ dev so i can't understand the format... only the header struct...


Top
 Profile  
 
 Post subject: Re: Nier Automata
PostPosted: Sun Mar 19, 2017 7:58 pm 
Offline
n00b

Joined: Thu Dec 23, 2010 9:49 am
Posts: 19
Has thanked: 4 times
Have thanks: 13 times
marcussacana wrote:
Good work...
Anyway this https://gist.github.com/Wunkolo/213aa61 ... ebb8ab89c2
Have any code to recreate the .dat? i'm not a c++ dev so i can't understand the format... only the header struct...


Added a markdown file to the gist link that describes the format a bit that should help. A "union" is basically a struct in which all the data overlaps each other so "ExtentionTableEntry" is just four bytes long. The offsets seem file-relative but you could also just load the entire file into memory and just use those same offsets as indexes in a byte array.

I haven't had the need to write an injector since you would also have to write a cpk injector which is something no one has really wanted to take on.

_________________
Aka Wunkolo


Top
 Profile  
 
 Post subject: Re: Nier Automata
PostPosted: Sun Mar 19, 2017 9:22 pm 
Offline
ultra-n00b

Joined: Sun Oct 12, 2014 11:05 pm
Posts: 1
Has thanked: 0 time
Have thanks: 0 time
Do you happen to know in which cpk the music is stored?


Top
 Profile  
 
 Post subject: Re: Nier Automata
PostPosted: Mon Mar 20, 2017 2:43 am 
Offline
beginner
User avatar

Joined: Wed Jul 22, 2015 10:08 pm
Posts: 23
Has thanked: 5 times
Have thanks: 3 times
Okay, i extracted the data009.cpk, extracted the "subtitle\subtitle1270_us.dat" (contains new game cutscene subtitle)
After this, i translated and recreated the .dat without problem... and i patch the .cpk... but
WTF the text don't change.... i confirm if the old text contains in the new data009.cpk but yes, not found... i have sure the tools works and the file has been edited, but the game don't change anything... probabbly have the file in another packget/folder, anyone found the correct file?


Top
 Profile  
 
 Post subject: Re: Nier Automata
PostPosted: Mon Mar 20, 2017 3:59 am 
Offline
beginner
User avatar

Joined: Wed Jul 22, 2015 10:08 pm
Posts: 23
Has thanked: 5 times
Have thanks: 3 times
Okay, after analyze more i see, this subtitle files from data009.cpk it's from the demo cutscene when you don't move the mouse in main menu...
looks i need found the correct file... :/


Top
 Profile  
 
 Post subject: Re: Nier Automata
PostPosted: Mon Mar 20, 2017 5:00 am 
Offline
beginner
User avatar

Joined: Wed Jul 22, 2015 10:08 pm
Posts: 23
Has thanked: 5 times
Have thanks: 3 times
Okay, the text is in data002.cpk\ph1\p100.dat\p100_fa238e7c_scp.bin... this RITE0003 format... don't have any tool to edit he?


Top
 Profile  
 
 Post subject: Re: Nier Automata
PostPosted: Mon Mar 20, 2017 5:16 am 
Offline
n00b

Joined: Thu Dec 23, 2010 9:49 am
Posts: 19
Has thanked: 4 times
Have thanks: 13 times
marcussacana wrote:
Okay, the text is in data002.cpk\ph1\p100.dat\p100_fa238e7c_scp.bin... this RITE0003 format... don't have any tool to edit he?


"RITE0003" files are pre-compiled ruby code. Nier uses an internal virtual machine that runs Ruby code using a library called mRuby which is able to compile ruby source code into a binary form so the ruby source code:

Code:
puts 'Hey'

would compile to this after running it through mrbc
Code:
    0000000: 5249 5445 3030 3033 78a7 0000 005d 4d41  RITE0003x....]MA
    0000010: 545a 3030 3030 4952 4550 0000 003f 3030  TZ0000IREP...?00
    0000020: 3030 0000 0037 0001 0004 0000 0000 0004  00...7..........
    0000030: 0080 0006 0100 003d 0080 00a0 0000 004a  .......=.......J
    0000040: 0000 0001 0000 0348 6579 0000 0001 0004  .......Hey......
    0000050: 7075 7473 0045 4e44 0000 0000 08         puts.END.....


which is just mruby VM bytecode

unless you write a decompiler or disassembler for the bytecode then you probably won't be editing it easily.

_________________
Aka Wunkolo


Top
 Profile  
 
 Post subject: Re: Nier Automata
PostPosted: Mon Mar 20, 2017 5:50 am 
Offline
beginner
User avatar

Joined: Wed Jul 22, 2015 10:08 pm
Posts: 23
Has thanked: 5 times
Have thanks: 3 times
yeah... so, i found this git https://github.com/xxuejie/mruby-bytecode
and this https://github.com/yamanekko/mrb_parser
this can help...? (i don't know how to use ruby... mrb_parse looks more easy to use)
learn about the ruby VM to translate a game... shit... I wait the lucky to see anyone develops a tool at least to edit the text

and to anyone like continue this research:
My Stuff with Stable CPK/DAT Unpacker/Repacker - https://www.mediafire.com/?op28t19b91ot4j7


Top
 Profile  
 
 Post subject: Re: Nier Automata
PostPosted: Tue Mar 21, 2017 3:41 am 
Offline
ultra-n00b

Joined: Tue Mar 21, 2017 3:06 am
Posts: 1
Has thanked: 3 times
Have thanks: 1 time
KazukiMutou wrote:
Do you happen to know in which cpk the music is stored?


From what I could tell and my limited knowledge, the music isn't stored in the .cpk archives.
/data/sound/stream is where you can find the .wem and .wsp files, I used ww2ogg.exe and revorb.exe with a .bat script to convert.

I'm completely new to this really and am just messing about at this point to see what works, don't crucify me lol.
I wasn't sure about the .wsp files at first, even after an extended period of google searching so I just modified the .bat to convert .wsp instead of .wem and to my surprise, it worked. however, these .wsp files/archives are at least 30MB large and are being converted down to 2-5MB audio files so I'm not sure if these .wsp files contain more than one audio file in them? I'm hoping someone more experienced can have a bit of a look to check it out.

For those wanting to know what I did precisely,

1. Download the latest ww2ogg release from the Github repo (024)
2. Download revorb.exe
3. Unzip ww2ogg and drag "ww2ogg.exe" and "packed_codebooks_aoTuV_603.bin" to a new folder. Put "Revorb.exe" into the same folder
4. Create a plain text file in that folder, rename it to convert.bat (whatever suits you, as long as it's .bat)
5. Edit the .bat and use the following for .wsp files (change the first line with (*.wsp) to (*.wem) for the .wem files)
Code:
for %%f in (*.wsp) do ww2ogg.exe %%f --pcb packed_codebooks_aoTuV_603.bin
pause
for %%f in (*.ogg) do revorb.exe %%f
pause

6. Copy the .wsp and .wem files into that folder and run the batch, press a key after it finishes the first time so it starts the revorb section of the batch and then once again to close and now you can listen to the files.

Only one file "5731267.wem" has glitch like sounds but I'm guessing that it is quite literally part of the game, just remember to turn your speaker/headphone volume down as it's quite loud. Obviously this is far from the full collection of audio files but it's a start I guess from a total noob at this stuff.

"BGM_*" - Background/music tracks, the juicy stuff, obviously. However I'm not finding any of the more noticeable tracks that I've heard so far in the game.
"core_*_" - Various small sounds for interaction with the world etc.
"p100_0_" - Sounds of metal/wood objects being shifted
"p100_2_, p200_0_ & p300_0_" - Wind/Aircraft sounds
"plffff_1_" - Louder aircraft sound
"r150_0_" - Earth rumbling sound?



Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 118 posts ]  Go to page 1, 2, 3, 4, 5 ... 8  Next

All times are UTC + 1 hour


Who is online

Users browsing this forum: nongnoobjung, wlscks7 and 8 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group