XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Sat Sep 23, 2017 7:44 pm

All times are UTC + 1 hour


Forum rules


Please click here to view the forum rules



Post new topic Reply to topic  [ 19 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Recettear compression
PostPosted: Sun Sep 19, 2010 3:34 pm 
Offline
advanced
User avatar

Joined: Mon Aug 06, 2007 9:51 pm
Posts: 41
Has thanked: 2 times
Have thanks: 4 times














You can make the ads go away by registering

Alright, this one's driving me nuts, and it's "only" compressed text.
The game stores all data that isn't music/sound/models in one large file (which is, for some reason, split into 12 smaller files)
There is a file called lnkdatas.bin which contains the whole folder/file structure, using it I was able to isolate specific files, but I'm having a hard time decompressing them (mostly because, well, I have no clue about compression)

I've attached a file which is called enemylist.txt by the game and I'm assuming it only contains text when decompressed. (I'll be using ? for any bytes that don't translate into alphanumerical characters.)

The first enemy in that list is the Wisp, which seems to be called "wisp1" in the file, alright so far. That enemy drops the items Natural Heater and Salamander Scale and maybe other things.

At offset 0x52, you see the string "Natur?al He??er" which obviously, when decompressed, is Natural Heater. The first thing I don't get is why Natur and al are even split up (by 0x04). The ?? in Heater is probably the compression at work (..replacing two bytes by two other bytes, but there are also cases where three or more bytes are reduced to 2, so whatever)

The really confusing thing is at 0x6A. The string says "S??am?and?? Sc???e" and with a bit of imagination, that's Salamander Scale.
Taking a look at the first ?? (in "S??am"), they are represented by 0x0112 (Big Endian) and should be "al" deconverted. However, the second ?? (in "and??"), which should be "er", is ALSO represented by 0x0112
The only explanation is that the single ? (represented by 0x08) does something funky, but what and how?

My guess is that the first entry would look something like this after decompression:
Code:
wisp1:
1: <some item name?>
2: <some item name?>
3:Natural Heater
4: <some item name?>
5:Salamander Scale


There's absolutely no trace of what the first/second/4th item could be, would that mean there's a large dictionary somewhere?

Another strange thing is that there are sometimes ASCII-character thrown in where there shouldn't be any, like the string "WorTn" that appears at 0xAC, which is "Worn Sword". Any ideas how the game could know that this T must be interpreted as something else?

So yeah, a bit of a lengthy post. If you have any idea what this compression (or even encryption?) algorithm could be or saw something similar somewhere else, do tell.

Edit: Uploaded the proper enemylist.txt


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.



Last edited by Forte on Sun Sep 19, 2010 11:08 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Recettear compression
PostPosted: Sun Sep 19, 2010 5:52 pm 
Offline
mega-veteran
mega-veteran

Joined: Fri Apr 29, 2005 2:39 pm
Posts: 184
Has thanked: 5 times
Have thanks: 22 times
it looks like standrd LZS. I'll do some tests

ok, it's defintely LZS but not standard (ah and your unpacker is bugged :P)


Top
 Profile  
 
 Post subject: Re: Recettear compression
PostPosted: Sun Sep 19, 2010 6:16 pm 
Offline
advanced
User avatar

Joined: Mon Aug 06, 2007 9:51 pm
Posts: 41
Has thanked: 2 times
Have thanks: 4 times
Thanks a lot, at least now I have something to go on.

And what do you mean, bugged? I'm fairly sure the file came out alright (although I didn't write any tools yet and got it out manually)


Top
 Profile  
 
 Post subject: Re: Recettear compression
PostPosted: Sun Sep 19, 2010 6:20 pm 
Offline
mega-veteran
mega-veteran

Joined: Fri Apr 29, 2005 2:39 pm
Posts: 184
Has thanked: 5 times
Have thanks: 22 times
nope, you were missing a byte at the beginning. Anyway I'm writing an unpacker. I'll use it to study the compression better


Top
 Profile  
 
 Post subject: Re: Recettear compression
PostPosted: Sun Sep 19, 2010 6:37 pm 
Offline
advanced
User avatar

Joined: Mon Aug 06, 2007 9:51 pm
Posts: 41
Has thanked: 2 times
Have thanks: 4 times
Err, you're right, actually <_<
The byte at the beginning is 0 and there's one less byte at the end (making the overall size the same).
What does the initial byte say anyway? Even bmps have one before the magic number.


Top
 Profile  
 
 Post subject: Re: Recettear compression
PostPosted: Sun Sep 19, 2010 6:56 pm 
Offline
mega-veteran
mega-veteran

Joined: Fri Apr 29, 2005 2:39 pm
Posts: 184
Has thanked: 5 times
Have thanks: 22 times
every file has it. It's a flag for the compression. I'm studying the algo right now.

If you need it here you are the unpacker without the decompression feature.

http://www.tbhreloaded.it/Vash/Recettar ... .v.0.9.rar

edit, almost done. I need to understand special cases


Top
 Profile  
 
 Post subject: Re: Recettear compression
PostPosted: Sun Sep 19, 2010 10:01 pm 
Offline
advanced
User avatar

Joined: Mon Aug 06, 2007 9:51 pm
Posts: 41
Has thanked: 2 times
Have thanks: 4 times
I think I understand the basics around that, but I'm still not getting anywhere.

There's a control code at 0x62 in the enemylist which is 0xA2 = 1010 0010 in bin.
So I'm reading two bytes as a reference, one literal, one reference, three literals, one reference and one literal again (3 * 2 + 5 = 11 bytes till next control code). The first three work out fine, but then I'm supposed to read 0x04 as a literal, and everything starts to fall apart. There's also 12 bytes till the next control code and even if I ignore the 0x04, it ends up garbled. What the hell.


Top
 Profile  
 
 Post subject: Re: Recettear compression
PostPosted: Mon Sep 20, 2010 12:50 am 
Offline
mega-veteran
mega-veteran

Joined: Fri Apr 29, 2005 2:39 pm
Posts: 184
Has thanked: 5 times
Have thanks: 22 times
I almost finished. This algorythm has many variants and I need to find them all.


Top
 Profile  
 
 Post subject: Re: Recettear compression
PostPosted: Mon Sep 20, 2010 1:08 pm 
Offline
advanced
User avatar

Joined: Mon Aug 06, 2007 9:51 pm
Posts: 41
Has thanked: 2 times
Have thanks: 4 times
Alright, looks like I finally got it working (at least text files, images still don't work, although they weren't really my priority).
Tabs seem to be messed up, but eh.
Again, thanks a lot.


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.



Top
 Profile  
 
 Post subject: Re: Recettear compression
PostPosted: Mon Sep 20, 2010 1:54 pm 
Offline
mega-veteran
mega-veteran

Joined: Fri Apr 29, 2005 2:39 pm
Posts: 184
Has thanked: 5 times
Have thanks: 22 times
I made it..here you are, the unpacker with decompression algorythm included. It's a bit sloppy but it's working.

http://www.tbhreloaded.it/Vash/Recettar ... .v.1.0.rar

I didn't try but the game should read the files from outside. I'm in a hurry now but I'll make some tests whe I'll be back home.

P.S. the algorythm for this compression is the stupidiest EVER, it has no sense, standard LZSS compresses lot more!


Top
 Profile  
 
 Post subject: Re: Recettear compression
PostPosted: Mon Sep 20, 2010 2:14 pm 
Offline
advanced
User avatar

Joined: Mon Aug 06, 2007 9:51 pm
Posts: 41
Has thanked: 2 times
Have thanks: 4 times
Yeah, it does seem like they took the normal LZS algorithm and changed it only for the sake of changing it.

By the way, mind posting your decompression routine?
I only covered 2 special cases (the one I described a few posts ago and one where the alg attempts to read more bytes than there are available) and it works for text files. Other files seem to have other special cases, what are they?


Top
 Profile  
 
 Post subject: Re: Recettear compression
PostPosted: Mon Sep 20, 2010 3:53 pm 
Offline
mega-veteran
mega-veteran

Joined: Fri Apr 29, 2005 2:39 pm
Posts: 184
Has thanked: 5 times
Have thanks: 22 times
I'm out now, I'll post it as soon as possible but I'm warning you...the code is a mess xD

Code:
while (ftell(bin)<pointer[i]+size[i])
{
flag=fgetc(bin); 

for (w=0;w<8;w++)
{
bit[w]=(flag>>(7-w))&0x1;
}   
times=realsize[i]-ftell(ext);
if (times >= 8)
times=8;
for (w=0;w<times;w++)
{
if (bit[w] == 0)
{
if  (ftell(bin)< pointer[i]+size[i])
{
if (ftell(ext)<realsize[i])
{   
byte=fgetc(bin);
fputc(byte,ext);   
}else{
w=times;   
}
}
}

if (bit[w] == 1)
{
if (ftell(ext)<realsize[i])
{     
if  (ftell(bin)< pointer[i]+size[i])
{   
recover=fgetc(bin);
if (recover >= 0x10)
{
   
nib1=recover/0x10;
nib2=recover-(nib1*0x10);
recover=nib1+1;
jump=fgetc(bin);
tmp2=jump;
jump=(nib2*0x100)+tmp2;

if ((nib1==ftell(ext)/0x100 || ftell(ext)/0x100) && (nib1 != nib2))
{
tmp=nib2;
nib2=nib1;
nib1=tmp;
recover=nib1+1;
jump=(nib2*0x100)+tmp2;       
}

}else{
jump=fgetc(bin);
recover=recover+1;   
}
if (recover == 1)
{
tmp=fgetc(bin);
recover=tmp+0x10+1;   
}
if (jump >= recover)
{
fseek(ext,-jump,SEEK_CUR);
fread(buffer,recover,1,ext);
fseek(ext,0x00,SEEK_END);
fwrite(buffer,recover,1,ext);
}else{
fseek(ext,-jump,SEEK_CUR);
fread(buffer,jump,1,ext);
fseek(ext,0x00,SEEK_END);
y=0;
for (z=0;z<recover;z++)
{
 
fputc(buffer[y],ext);
y++;
if (y>=jump)
{y=0;}

} //else

} //for 
}
}else{
w=times;   
}
}
}//for
}


Top
 Profile  
 
 Post subject: Re: Recettear compression
PostPosted: Sat Nov 27, 2010 6:12 pm 
Offline
advanced

Joined: Tue Jun 03, 2008 9:55 pm
Posts: 53
Has thanked: 3 times
Have thanks: 1 time
Hi Vash

i use your unpacker but all files unpacked are 0bytes, do you made another unpacker ?


Top
 Profile  
 
 Post subject: Re: Recettear compression
PostPosted: Sat Nov 27, 2010 6:36 pm 
Offline
mega-veteran
mega-veteran

Joined: Fri Apr 29, 2005 2:39 pm
Posts: 184
Has thanked: 5 times
Have thanks: 22 times
nope. It works as it is. If i recall correctly you need to have lnkdatas.bin in the same directory and unite every dataXXX.bin in one file.
The tool should do that for you but if it doesn't work, use this command:

Code:
copy /b data000.bin + data001.bin + data002.bin + data003.bin + data004.bin + data005.bin + data006.bin + data007.bin + data008.bin + data009.bin + data010.bin + data011.bin data.bin


Top
 Profile  
 
 Post subject: Re: Recettear compression
PostPosted: Sun Nov 28, 2010 11:57 am 
Offline
advanced

Joined: Tue Jun 03, 2008 9:55 pm
Posts: 53
Has thanked: 3 times
Have thanks: 1 time
Ah oki, thx works now...

- Game can start like this ( i want make a french translation of this game ) ?


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 19 posts ]  Go to page 1, 2  Next

All times are UTC + 1 hour


Who is online

Users browsing this forum: Google [Bot] and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group