Nier Automata

[wunk]

Platinum is using CriWare's "cpk" archive format again.

Can be easily opened with https://github.com/wmltogether/CriPakTools

Contains the usual dat and dtt and so forth just like some of their other games though some things might be updated.

I was able to extract textures but some are somewhat "corrupted"(or are just aux data like directional lightmaps)




Sample file: https://transfer.sh/13foZS/um2140.dat

There's some other file formats contained that could be mapped out. Some of the CPKs contain a lot of wd_ files(world data. Much of it procedurally generated using Simplygon with not much else going on.

data012.cpk contains a lot of quest and scripting data and the licenses involve a lot of ruby so there might be a huge repo of ruby scripts somewhere among the assets.

the .z files found within the .dat files are predictably zlib. An example of a decompressed "q171_hap.z" file found within "q171.dat" which is within "data012.cpk"(so many nested levels of files...)
http://pastebin.com/raw/0rJZ4CKH

pak files are pretty much containers for multiple zip streams.

Found all around the archives are what looks like "compiled" ruby code. Constants and some signs of syntax are exposed here and there. Probably pre-lexicographed ruby code to avoid storing pure plaintext.

Very likely it's just pre-compiled mruby VM bytecode output from mrbc.
https://raw.githubusercontent.com/mruby ... y/opcode.h

Bin files found in quest data such as 'q171_12f2963a_scp.bin' is for sure output from mrbc. (the ones with the `RITE0003` header)
https://github.com/mruby/mruby/blob/08a ... dump.h#L53

Some of the middleware that Nier uses:
https://github.com/mruby/mruby
https://github.com/antimon2/enumerable_ ... /README.md
https://www.musl-libc.org/
http://rake.rubyforge.org/
https://github.com/yhara/enumerable-lazy
https://www.simplygon.com/
http://www.geomerics.com/enlighten/

Made a quick extractor for the .dat file files.
https://gist.github.com/Wunkolo/213aa61 ... ebb8ab89c2

[wunk]

Model format ".wmb" has been changed. Magic is "WMB3" which is weird because some of their older games used "WMB4". Format uses a deeply nested table format similar to WMB4 somewhat. Seems to include a full map of uniform data(passed to the shader) and parameters and such. Material parameters are all floats where applicable and zero otherwise(such as for texture inputs). Header is 0x90 bytes. Writing a reader for it now while I map out all the table types and I'll use something like fbxsdk to write an exporter most likely.

Some sample wmb files
https://transfer.sh/gSpXc/sample.zip

[ssringo]

Glad to see folks working on this game. Have you seen (or already taking part in) the thread on the facepunch forums for Nier? They've been discussing and working on the format a bit. If you haven't seen it, here's the link

https://facepunch.com/showthread.php?t=1557100

Looking forward to progress

[SkacikPL]

So as far as my comprehension goes dtt format is basically DDS embedded with some sort of overhead?
I'm looking into it cause some of the textures could really use some high definition love.

[wunk]

Glad to see folks working on this game. Have you seen (or already taking part in) the thread on the facepunch forums for Nier? They've been discussing and working on the format a bit. If you haven't seen it, here's the link

https://facepunch.com/showthread.php?t=1557100

Looking forward to progress

The WMB format is entirely different from anything any tool out there uses. It's a new format specifically for the new rendering engine they have set up and it's entirely different from the most recent "WMB" iteration seen here(http://pastebin.com/CEHriQBv) which is what most tools out there use and are totally incompatible with the latest format. Here's a snippet from the reader I am working on right now. Lots of metaprogramming elsewhere as this is just a small snippet but this is very likely to be what Platinum games internally does as well. Commented with some of my research notes. I pretty much mapped out the Materials/Uniform parameters(values that get passed to the shader) among other things. 2B's internal name is pl0000 even in the ruby bytecode.

Code: Select all

template< typename EntryType = void >
struct TableReference
{
	typedef EntryType ElementType;
	std::uint32_t Offset;
	std::uint32_t Count;
};

struct Header
{
	std::uint32_t Magic; // WMB3
	std::uint32_t Unknown4; // Version?

	std::uint32_t Unknown8; // 0

	std::uint16_t UnknownC; // 0x0, 0xA
	std::uint16_t UnknownE; // 0xffff

	Vector3 UnknownVector10;
	Vector3 UnknownVector1C;

	TableReference<UnknownTable28Entry> UnknownTable28; // Table of 0x60 byte structures

	TableReference<std::uint8_t> UnknownTable30; // Table of 1 byte structures

	TableReference<UnknownTable38Entry> UnknownTable38; // Table of 0x30 byte structures

	TableReference<void> UnknownTable40; // Table of 0x20 byte structures

	TableReference<void> LODTable; // Table of 0x1C byte structures

	TableReference<void> UnknownTable50; // Often Zero

	TableReference<std::uint32_t> UnknownTable58;

	TableReference<
		TableReference<std::uint16_t>
	> UnknownTable60; // Table of... Table References to tables of uint16_t

	TableReference<MaterialParamTableEntry> MaterialParamTable; // Table of 0x30 byte structures

	TableReference<UnknownTable70Entry> UnknownTable70; // Table of 0x30 byte structures

	TableReference<void> UnknownTable78; // Table of {uint32_t Zero, uint32_t Unknown } structures
};

That person is using Ninjaripper which basically middle-mans between the game and directx driver so it won't have a lot of the auxiliary information to get things like rigging and texture names and so forth.

So as far as my comprehension goes dtt format is basically DDS embedded with some sort of overhead?
I'm looking into it cause some of the textures could really use some high definition love.

DTT files are DAT files intended to store texture(.wtp) and model(.wmb) data among others(bxm,mot,sop,etc). .wtp files are regular DirectX textures(.dds) and can simply be renamed to a .dds extension.

[wunk]

Current structure so far:

Code: Select all

#pragma once
#include <cstdint>
#include <cstddef>

namespace WMB
{
#pragma pack(push, 1)

template<size_t Dimension>
struct Vector
{
	std::float_t f32[Dimension];
};

using Vector4 = Vector<4>;
using Vector3 = Vector<3>;
using Vector2 = Vector<2>;

template< typename EntryType >
struct TableReference
{
	typedef EntryType ElementType; // For reading functions
	std::uint32_t Offset;
	std::uint32_t Count;
};

struct StringReference
{
	std::uint32_t Offset; // Offset for null-terminated string
};

struct UnknownTable28Entry
{
	struct
	{
		// Probably Very wrong
		Vector4 UnknownVector0;
		Vector2 UnknownVector10;
		Vector2 UnknownVector18;
	} Elements[3];
};

struct GeoStreamTableEntry
{
	struct UnknownTable0Entry
	{
		Vector3 UnknownVector0; // Position?
		Vector2 UnknownVectorC; // UV?

		std::uint32_t Unknown18; // Zero
		std::uint32_t Unknown1C;
	};

	TableReference<UnknownTable0Entry> UnknownTable0; // Possible vertex stream

	TableReference<void> UnknownTable8; // Tends to be 0

	std::uint32_t VertexStride10; // Stride for UnknownTable0?
	std::uint32_t VertexStride14; // Stride

	TableReference<void> UnknownTable18; // Tends to be 0

	std::uint32_t Unknown20; // Vertex Count?
	std::uint32_t Unknown24;
	TableReference<void> IndexTable28; // Possible indice array
};

struct MaterialParamTableEntry
{
	std::uint32_t Unknown0;
	std::uint16_t Unknown4;
	std::uint16_t Unknown6;

	StringReference ParamMeshName; // Null Terminated String (ex: PL_0000_Eye)
	StringReference ParamMeshClass; // Null Terminated String (ex: Eye00_XXXXX)
	StringReference ParamsName; // Null Terminated String (ex: Default)

	std::uint32_t Unknown14; // Usually 1

	struct ParamTableEntry
	{
		StringReference UniformName; // Null terminated (ex: g_AlbedoMap)
		std::float_t Unknown4; // Uniform Value(float, 0 if not applicable such as for a texture)
	};

	TableReference<ParamTableEntry> ParamTable;

	struct UnknownTable20Entry
	{
		std::uint32_t Unknown0; // (ex: 0, 2)
		std::uint32_t Unknown4; // (ex: 2DA470h, FFFFFFFFh)
	};

	TableReference<UnknownTable20Entry> UnknownTable20;

	TableReference<ParamTableEntry> VertexShaderParamTable;

	// Same structure as "ParamTableEntry" above
	// (ex: Binormal0, Color0, Normal, Position, Tangent, TexCoord0,
	//      g_AlbedoColor_X, g_ReflectionIntensity, g_Metallic, etc )
};

struct MeshGroupTableEntry
{
	StringReference GroupName; // (ex: Body)

	Vector3 UnknownVector4;
	Vector3 UnknownVector10;

	// Possibly face indexes
	TableReference<std::uint16_t> UnknownTable1C; // Table uint16_t
	TableReference<std::uint16_t> UnknownTable24; // Table of uint16_t
};

struct UnknownTable78Entry
{
	std::uint32_t Unknown0; // 0x0
	std::uint32_t UnknownIndex4;
};

///////////////////////////////////////////////////////////////////////////////

struct Header
{
	std::uint32_t Magic; // WMB3
	std::uint32_t Unknown4; // Version?

	std::uint32_t Unknown8; // 0

	std::uint16_t UnknownC; // 0x0, 0xA
	std::int16_t UnknownE; // 0xffff

	Vector3 UnknownVector10;
	Vector3 UnknownVector1C;

	TableReference<UnknownTable28Entry> BoneTable28; // Table of 0x60 byte structures

	TableReference<std::int8_t> UnknownTable30; // Table of 1 byte structures

	TableReference<GeoStreamTableEntry> GeoStreamTable38; // Table of 0x30 byte structures

	TableReference<void> UnknownTable40; // Table of 0x20 byte structures

	// Possibly LOD related
	TableReference<void> UnknownTable48; // Table of 0x1C byte structures

	TableReference<void> UnknownTable50; // Zero

	TableReference<std::uint32_t> UnknownTable58; // Table of uint_32

	TableReference<
		TableReference<std::uint16_t>
	> UnknownTable60; // Table of... Table References to tables of uint16_t

	TableReference<MaterialParamTableEntry> MaterialParamTable; // Table of 0x30 byte structures

	// Named/Grouped chunks of geometry
	TableReference<MeshGroupTableEntry> MeshGroupTable; // Table of 0x30 byte structures

	TableReference<UnknownTable78Entry> UnknownTable78; // Table of { uint32_t Zero, uint32_t Unknown } structures

	TableReference<void> UnknownTable80; // Zero
	TableReference<void> UnknownTable88; // Zero
};

#pragma pack(pop)
}

[marcussacana]

Good work...
Anyway this https://gist.github.com/Wunkolo/213aa61 ... ebb8ab89c2
Have any code to recreate the .dat? i'm not a c++ dev so i can't understand the format... only the header struct...

[wunk]

Good work...
Anyway this https://gist.github.com/Wunkolo/213aa61 ... ebb8ab89c2
Have any code to recreate the .dat? i'm not a c++ dev so i can't understand the format... only the header struct...

Added a markdown file to the gist link that describes the format a bit that should help. A "union" is basically a struct in which all the data overlaps each other so "ExtentionTableEntry" is just four bytes long. The offsets seem file-relative but you could also just load the entire file into memory and just use those same offsets as indexes in a byte array.

I haven't had the need to write an injector since you would also have to write a cpk injector which is something no one has really wanted to take on.

[KazukiMutou]

Do you happen to know in which cpk the music is stored?

[marcussacana]

Okay, i extracted the data009.cpk, extracted the "subtitle\subtitle1270_us.dat" (contains new game cutscene subtitle)
After this, i translated and recreated the .dat without problem... and i patch the .cpk... but
WTF the text don't change.... i confirm if the old text contains in the new data009.cpk but yes, not found... i have sure the tools works and the file has been edited, but the game don't change anything... probabbly have the file in another packget/folder, anyone found the correct file?

[marcussacana]

Okay, after analyze more i see, this subtitle files from data009.cpk it's from the demo cutscene when you don't move the mouse in main menu...
looks i need found the correct file... :/

[marcussacana]

Okay, the text is in data002.cpk\ph1\p100.dat\p100_fa238e7c_scp.bin... this RITE0003 format... don't have any tool to edit he?

[wunk]

Okay, the text is in data002.cpk\ph1\p100.dat\p100_fa238e7c_scp.bin... this RITE0003 format... don't have any tool to edit he?

"RITE0003" files are pre-compiled ruby code. Nier uses an internal virtual machine that runs Ruby code using a library called mRuby which is able to compile ruby source code into a binary form so the ruby source code:

Code: Select all

puts 'Hey'

would compile to this after running it through mrbc

Code: Select all

    0000000: 5249 5445 3030 3033 78a7 0000 005d 4d41  RITE0003x....]MA
    0000010: 545a 3030 3030 4952 4550 0000 003f 3030  TZ0000IREP...?00
    0000020: 3030 0000 0037 0001 0004 0000 0000 0004  00...7..........
    0000030: 0080 0006 0100 003d 0080 00a0 0000 004a  .......=.......J
    0000040: 0000 0001 0000 0348 6579 0000 0001 0004  .......Hey......
    0000050: 7075 7473 0045 4e44 0000 0000 08         puts.END.....

which is just mruby VM bytecode

unless you write a decompiler or disassembler for the bytecode then you probably won't be editing it easily.

[marcussacana]

yeah... so, i found this git https://github.com/xxuejie/mruby-bytecode
and this https://github.com/yamanekko/mrb_parser
this can help...? (i don't know how to use ruby... mrb_parse looks more easy to use)
learn about the ruby VM to translate a game... shit... I wait the lucky to see anyone develops a tool at least to edit the text

and to anyone like continue this research:
My Stuff with Stable CPK/DAT Unpacker/Repacker - https://www.mediafire.com/?op28t19b91ot4j7

[SirZephy]

Do you happen to know in which cpk the music is stored?

From what I could tell and my limited knowledge, the music isn't stored in the .cpk archives.
/data/sound/stream is where you can find the .wem and .wsp files, I used ww2ogg.exe and revorb.exe with a .bat script to convert.

I'm completely new to this really and am just messing about at this point to see what works, don't crucify me lol.
I wasn't sure about the .wsp files at first, even after an extended period of google searching so I just modified the .bat to convert .wsp instead of .wem and to my surprise, it worked. however, these .wsp files/archives are at least 30MB large and are being converted down to 2-5MB audio files so I'm not sure if these .wsp files contain more than one audio file in them? I'm hoping someone more experienced can have a bit of a look to check it out.

For those wanting to know what I did precisely,

1. Download the latest ww2ogg release from the Github repo (024)
2. Download revorb.exe
3. Unzip ww2ogg and drag "ww2ogg.exe" and "packed_codebooks_aoTuV_603.bin" to a new folder. Put "Revorb.exe" into the same folder
4. Create a plain text file in that folder, rename it to convert.bat (whatever suits you, as long as it's .bat)
5. Edit the .bat and use the following for .wsp files (change the first line with (*.wsp) to (*.wem) for the .wem files)

Code: Select all

for %%f in (*.wsp) do ww2ogg.exe %%f --pcb packed_codebooks_aoTuV_603.bin
pause
for %%f in (*.ogg) do revorb.exe %%f
pause

6. Copy the .wsp and .wem files into that folder and run the batch, press a key after it finishes the first time so it starts the revorb section of the batch and then once again to close and now you can listen to the files.

Only one file "5731267.wem" has glitch like sounds but I'm guessing that it is quite literally part of the game, just remember to turn your speaker/headphone volume down as it's quite loud. Obviously this is far from the full collection of audio files but it's a start I guess from a total noob at this stuff.

"BGM_*" - Background/music tracks, the juicy stuff, obviously. However I'm not finding any of the more noticeable tracks that I've heard so far in the game.
"core_*_" - Various small sounds for interaction with the world etc.
"p100_0_" - Sounds of metal/wood objects being shifted
"p100_2_, p200_0_ & p300_0_" - Wind/Aircraft sounds
"plffff_1_" - Louder aircraft sound
"r150_0_" - Earth rumbling sound?