Help with The Godfather II .str files

[venixuc]

The "dac49de4.str" file contains text displayed in this game. I was able to extract its content with QuickBMS (using dead_space_3.bms). However, the text I would like to edit is compressed inside NAM_1.dat. When extracting .str content with Game Extractor, NAM_1 file is exported as kapR file and the text is decompressed, it's possible to edit it with hex editor as long as the original file size is preserved.
But, in order to reimport the modified file into .str, it has to be compressed.
So, the question is: How to convert kapR file to NAM_1.dat?
I've attached the .rar archive which contains:
- dac49de4.str - the original archive
- NAM_1.dat, NAM_2.bal, NAM_3 dat, NAME - files extracted with QuickBMS
- kapR - the decompressed NAM_1.dat.
The files of interest for this are NAM_1.dat and kapR. Also, is there any other way to edit kapR other than using hex editor that would allow changing its size (something like .gxt editor for GTA games)? The Game Extractor exports it without extension.
Any help is appreciated.

[ikskoks]

So, the question is: How to convert kapR file to NAM_1.dat?

This NAM_1 file is compressed with Refpack algorithm.
To pack this, you would need to compress it again with Refpack
Here is some info https://www.google.com/search?client=fi ... compressor

But maybe it will be better to just use quickbms to parse the file,
here is file format for STR http://www.watto.org/specs.html?specs=Archive_STR_3SLO

Also, is there any other way to edit kapR other than using hex editor that would allow changing its size (something like .gxt editor for GTA games)?

It is possible only by reverse engineering whole file format and creating working custom editor manually. That's a lot of work.

[venixuc]

Thanks for your reply, ikskoks.
I inspected the kapR file further and found out that it can be opened with notepad. It's even possible to view and edit the text I need (I'm not trying to translate the game, I want to edit npc names - more precisely mobster names). But, when I compare the non-edited and edited files in HxD, I see this:
https://ibb.co/VgZd0Gw (non-edited)
https://ibb.co/NnzPQL1 (edited)
Several "00" values are changed to "20". Can this corrupt the file?
If the file doesn't get corrupted while doing this, it would be easy to change its size. But then a new problem emerges; QuickBMS refuses to reimport the larger file into .str.
Also, the npc names are grouped - here you can see a fragment of Corleone mobster names:
https://ibb.co/NpGJrnW
The same thing goes with the mobster names from another families. If I wanted to keep the file size intact and use HxD for editing, would I need to keep the original name length or just the sequence length of the name group?
For example, mobster name "Big Greenie" Grinberg has 23 characters, including the spacing and quotation marks. The next name in the sequence is Micheal Galli, with 14 characters (spacing included). And let's say that the entire sequence of Corleone mobster names has e.g. 600 characters.
Would it be possible to give Big Greenie a new, 20-characters long name and 17-characters long name to Galli? Something like this:
https://ibb.co/KyNHv9K
This wouldn't "shift" the rest of the file or change its size, but would the new names show up correctly in game?

And one more thing:
I compiled the refpack-brute-force.c but it doesn't open the file; when I type refpack-brute-force INFILE kapR or refpack-brute-force INFILE NAM_1.dat, it says that no such file/directory exists.

[ikskoks]

and found out that it can be opened with notepad.

Several "00" values are changed to "20". Can this corrupt the file?

Ok, so never use notepad for editing binary files. It can corrupt file for many ways.
It's always better to use hex editor or custom tool.

QuickBMS refuses to reimport the larger file into .str.

That's the case I was talking earlier. It's easier to use quickbms because of it's
reimport function. But of course edited file must be the same size as original to make it work.
Only way to change this is to write a custom tool and as I said - it can be a lot of
reverse engineering and programming work.

Also, the npc names are grouped

would I need to keep the original name length or just the sequence length of the name group?

but would the new names show up correctly in game?

There is a way to change this. When games use text blocks like this one, they can have pointer tables
before these blocks. And here you fortunately have such pointer table starting at offset 36308.

Here is the structure:

Code: Select all

num_of_strings *
{
  2 bytes (uint16) - string pointer
  2 bytes (uint16) - ID  // e.g. "1"
}

Here you can see how it looks for the first strings https://i.imgur.com/poy87uD.png
And here how these pointers looks in the hex editor https://i.imgur.com/5glpvrs.png

So basically you need to edit those pointers to make some strings larger and some smaller.

I compiled the refpack-brute-force.c but it doesn't open the file; when I type refpack-brute-force INFILE kapR or refpack-brute-force INFILE NAM_1.dat, it says that no such file/directory exists.

Maybe you forgot to put paths in quotes or something like that?

[venixuc]

There is a way to change this.

If you mean "change the fact that the names are grouped", I'm not trying to change that; I don't think it would be possible at all due to the game mechanic - there are approx. 100 npc model variations and 200 names for crime families, so one model can have one of 2 possible names (e.g. a specific Corleone mobster can be Nick Williams or Ray Sforza - this is random). This means that these names are also paired in some way, but that's not really important for what I'm trying to do; if I was to edit the file (I'm not doing it yet because I can't get the refpack program working), I would do it block by block - this would require keeping the string length of the entire block and the number of names intact.

But I'd like to know more about string lengths within one block. I mean, if I changed the names "Big Greenie" Grinberg and Michael Galli to Alessandro Esposito and Virgilio Zingaro would the new string lengths be correctly registered by the game (I definitely wouldn't like to launch the game and see that the new names are shown as Alessandro EspositoVir or Alessandro Esposito Vi or Virgilio Zing because old lengths are kept instead). Does editing string lengths within one block (without changing number of strings) require editing pointers as well?

As for the refpack, I really don't know what I'm doing wrong. No matter what I type in - file name or path to it, it always "fails to open input file for reading - no such file or directory". The file is in the same directory as the .exe and the cmd is opened inside that folder. I'm starting to wonder if it's functional at all.
Here is a fragment of the refpack-brute-force.c code describing its usage and the error I'm getting:

Code: Select all

if (argc != 3) {
		printf("Usage: refpack_brute_force INFILE OUTFILE\n");
		return EXIT_SUCCESS;
	}

	fd = fopen(argv[1], "rb");
	if (!fd) {
		fprintf(stderr, "Error: failed to open input file for reading"
			" (%s)\n", strerror(errno));
		return EXIT_FAILURE;
	}

[ikskoks]

If you mean "change the fact that the names are grouped"

I meant only string editing. Change of grouping would require changing game source code.

Does editing string lengths within one block (without changing number of strings) require editing pointers as well?

Yes, exactly. You need to change pointers to manipulate lengths.

As for the refpack, I really don't know what I'm doing wrong.

Maybe just use other compressor. First link in google isn't always the best one xd

This one is working fine for me https://cncguild.net/item-133
https://i.imgur.com/cdcjdGG.png

[venixuc]

This one is working fine for me https://cncguild.net/item-133

Actually that tool is the first one I tried and it works for me too. But after the compression, the file size is 150 kb while it has to be 127 kb. And again, QuickBMS doesn't want to deal with it due to its size. I haven't modified the file before compressing it; I just compressed the kapR file extracted with Game Extractor.
I also tried decompressing NAM_1.dat extracted with QuickBMS, but it says that it isn't compressed (???!!!):
https://ibb.co/TqDy3x7

I also tried the one from Niotso Wiki:

The original RefPack.cpp written by Frank Barchard was released by WCNews: http://download.wcnews.com/files/docume ... efPack.cpp

However this one won't compile at all. Even if it did, I don't think it would work; the page describes that the refpack variation of the file is the one from The Sims Online (2002). Considering the fact that even The Godfather: The Game (2006) tools aren't compatible with The Godfather II (for example, there is a Zenhax topic about editing .str files from the first game - the strtool they used there doesn't recognize .str files from the second game, it says that "the input file is not a .str file"), I can say that this tool is pretty much useless.

Anyways, I got the RefPack Brute Force working. Or so I thought - the compressed file is completely empty.

The Game Extractor was the only tool capable of making NAM_1.dat content readable - not by opening it directly, but by extracting from the .str (it exports the file as kapR). And when it comes to .str files, GE can only export their content - importing anything back corrupts the .str and the game doesn't load.

Another option would be the Gibbed tools:
https://github.com/gibbed?tab=repositories
There is a RefPack tool there too. But yet again, I'm having problems with compiling it:

Code: Select all

error CS5001: Compression.exe does not contain a static `Main' method suitable for an entry point

I used csc.exe from .NET Framework.

[ikskoks]

Maybe try my refpack script
https://github.com/bartlomiejduda/Tools ... script.bms

Only change extension on line 13 to the one you want to have after decompressing.

First decompress DAT file with this script, make some changes in the file and then use reimport mode.
Let me know if it worked.

[venixuc]

Maybe try my refpack script
https://github.com/bartlomiejduda/Tools ... script.bms

This happens when I try to decompress NAM_1.dat:
https://ibb.co/C0S8FSc
While the script can't decompress NAM_1.dat, it successfully compresses kapR file and the size after compression is 127 kb which makes it possible to import it back into .str with QuickBMS. But then the game crashes. The cause is obvious: when the compressed file is opened with hex editor, its content looks quite different than the original. Again, I haven't modified the kapR before the compression.

The refpack algorithm variation is probably different. I can't be 100% sure, but it appears that GE decompresses the file properly. However, standalone NAM_1.dat cannot be opened with GE; it decompresses it when .str archive is opened.

That's why I decided to try the Gibbed RefPack. Its algorithm is the one used in Dead Space 2, and there is a good chance that this one would work, just like dead_space_3.bms script for QuickBMS. But I'm still unable to compile it.

I've attached two NAM_1.dat files; original (extracted with QuickBMS) and modified (compressed kapR).

[ikskoks]

Oh boy, this case is more complicated than I thought, haha.

As for decompression method from Game Extractor, you can refer to the source code
https://sourceforge.net/projects/gameex ... p/download
The files you should look in are "Plugin_STR_3SLO.java" and of course "Exporter_REFPACK.java"

there is a good chance that this one would work, just like dead_space_3.bms script for QuickBMS

I have googled this script and it seems that it also uses "dk2" as a comtype parameter

Code: Select all

elif CHUNK_NAME == "Rpak"
        comtype dk2

so I have no idea what's wrong here...

But I'm still unable to compile it.

I was able to compile it without issues in Visual Studio 2019,
but it seems it is only a DLL file, not a working program.
To make it usable you probably need write some wrapper or at least main function.
I'm not a C# programmer, so I can't help you with that. xd


Edit: Here is compiled DLL in case you need it

Gibbed_Refpack_dll.zip

[venixuc]

I was able to compile it without issues in Visual Studio 2019,
but it seems it is only a DLL file, not a working program.

I compiled it to .dll too with csc.exe (using -target:library command). But the .dll file itself isn't very useful. All of the Gibbed tools are available only as the source code, and if this refpack code isn't meant to be a standalone .exe, it would be extremely difficult to determine which one uses it as an extension.

I haven't found any plugins in GE source code. The "3SLO" (ols3) signature is defined inside dead_space_3.bms for extracting .str file content. When I try decompressing NAM_1.dat using this script, this happens:
https://ibb.co/bH3XCsV
It doesn't recognize the signature.

Now, both QuickBMS and GE break down the .str srchive in several chunks. The QuickBMS extracts 4 chunks: NAM_1.dat, NAM_2.bal, NAM_3.dat and NAME (these files are attached in my first post), while GE extracts 3 chunks - kapR, Unnamed File 000002 and Unnamed File 000003. The 'unnamed files' are equivalent to NAM_2.bal and NAM_3.dat respectively, and GE doesn't decompress them (just like QuickBMS). Also, GE doesn't extract anything equivalent to NAME (this file only contains a header fragment from the .str).

The interesting file here is NAM_1.dat; while QuickBMS extracts it just how it looks like inside the .str archive, GE decompresses it. Its directory also contains a number of bms scripts inside .rar file. But I couldn't find any scripts or plugins responsible for the decompression; they could be hardcoded or inside .dll.

Not everything inside .str archive is compressed; only chunks with "kapR" identificator. The only file with this identificator inside dac49de4.str is NAM_1.dat. That's why GE decompresses it and extracts it as kapR. Finding out how GE decompresses these chunks would be helpful for compressing the extracted data.

I found a bms script which covers 99% of NAM_1.dat when i try decompressing it - skate3.bms. However, it says that it can't read 4 bytes at offset 0001FFBC and it doesn't complete the process. The same message shows up if I change idstring 6F 6C 73 33 (ols3) to B8 FF 01 00 with hex editor inside dead_space_3.bms. I wonder if skate3.bms could be modified to successfully decompress NAM_1.dat.

The last script line before the error is:

Code: Select all

30 get crc long

It looks like this
https://ibb.co/D1JVZYt

This time, I've attached the complete GE .str extraction results and the skate3.bms script.

[venixuc]

I've found "Exporter_REFPACK.java" GE plugin. While I can't really tell what's going on with it by looking at its content, one thing got my attention:

A.K.A. DBPF compression. Refer to http://wiki.niotso.org/RefPack for more info.

It refers to the RefPack.cpp file available there. This means that this file could be useful, if only it could be compiled; there are way too many missing libraries when I try compiling it.

Until the way to compress kapR (or decompress NAM_1.dat) is found, I'm unable to test editing results. This could take ages, and this topic could be long forgotten by that time.

Before that happens, I'd like to ask about string pointers. For example, the very first string in Corleone names block is "Big Greenie" Grinberg. How to find its pointer, and what exactly has to be edited there (value, position or something else)?

[ikskoks]

It refers to the RefPack.cpp file available there. This means that this file could be useful, if only it could be compiled; there are way too many missing libraries when I try compiling it.

You don't really need to compile it as it is already used in quickbms.
Check the file \src\libs\shadowforce\RefPack.cpp in quickbms source.
I have compared both files and there is almost 100% match https://i.imgur.com/rtl1MpZ.png

Test all refpack types from src\comtype.h in the bms script I have send you earlier.

Code: Select all

CT_RefPack
XREFPACK
XREFPACK0
CT_RefPack_COMPRESS
DK2
DK2_COMPRESS

Btw this Gibbed dll also can be useful. There are ways
to use dynamic libraries in C++ projects https://www.google.com/search?client=fi ... 2B+project
(or just write a wrapper in C#)



As for pointers, I have explained it with screenshots earlier
https://i.imgur.com/poy87uD.png
https://i.imgur.com/5glpvrs.png

You only need to change values. For example you have "Specs" word with pointer 7050.
When you change it to 7052, game will read this as "ecs".
But ot will give you of course 2 extra characters to use in the previous string.

[venixuc]

Test all refpack types from src\comtype.h in the bms script I have send you earlier.

Code: Select all

CT_RefPack
XREFPACK
XREFPACK0
CT_RefPack_COMPRESS
DK2
DK2_COMPRESS

Well, xrefpacks don't work at all. This type gives an error "unsupported compression -1 in reimport mode" with last script line:

Code: Select all

clog NAME 0 ZSIZE SIZE

With ct_refpacks and dk2s it's possible to complete the compression of kapR, but the data inside new NAM_1.dat is compressed too much which creates a huge "gap" from offset 0001b3c0 to 0001ffbc; this causes the game to crash.

It's also worth mentioning that changing comtype in the script doesn't make decompression of original NAM_1.dat possible - it always says that the input file is incomplete. There is something in the script itself that compresses the data differently.

https://www.zenhax.com/viewtopic.php?f=4&t=27&start=20
In this Zenhax topic "How to recognize the compression algorithms with your eyes", GHFear wrote:

Just want to add how I spot RefPack/dk2 nowadays.
First off...
0xXX = random value.
RefPack/dk2 compression mostly looks like this "0xXX\0xFB" (where the 0xFB seems to be the tell tale sign of refpack.) and then right after is the Decompressed Size as a 4byte 32 bit integer / long. (to further confirm that it in fact is refpack/dk2.)
So all in all it looks like this "0xXX\0xFB\0x00\0x1B\0x62\0x1C"

Depending on the strength of the compression the strings can still be sorta readable like this: "c:\datatemp\inter\PS2\Neutral\Chapâs\Hogwarts\ZoneãONE_HW_Viaduct_E.nãnce_DD\LefTextur�'àET_BURNâOOK_PAPER.ss€"

If you see this pattern, use comtype dk2

So the comtype here is probably dk2. I've tested every dk2 bms script I could find on QuickBMS site; none of them is able to decompress NAM_1.dat. Only GE is able to do this, but it cannot be used to edit .str files. And, GE is unable to open original NAM_1.dat directly. Could this mean that it doesn't decompress the data the right way?

[ikskoks]

none of them is able to decompress NAM_1.dat.

Delete 4 bytes from the beginning of NAM_1.dat and you will be able to decompress it.