Files extractors scripting

[aluigi]

a quick question: what are ALL the current available solutions for writing file extractors for simple formats through scripting?

an example of which I'm aware is the mexcom scripting but if I'm not in error it doesn't support things like zlib/deflate decompression and XORing (correct me if I'm wrong) but most important it works only with the registered version of multiex (correct me here too if needed) so it must be not considered.

the idea of creating file extractors on the fly with any text editor "stimulates" a lot my brain but obviously it's useless to reinvent the wheel if already exists a good solution.

[chrrox]

I love your Idea of writing a file extractor that could be customized through some type of xml file or settings dialog.
This would not be easy to complete but would be very worth it in the end.
The only ripper that I know of is jaedernaub but I do not know how to add file formats to it.

[Rheini]

Well you can use MexCom scripts with GameExtractor.
I personally played around with Lua with added binary operations some time ago.

[stevenx]

Need command line extractor looks like ext.exe ,ext.ini.
It is useful when need repack.

[stevenx]

I think not exists a perfact extractor now.
My idea is looks like this:

At first define some struct of file format.
For example

-------------------------------
Type 1
-------------------------------
Head
MagicNumber XXX bytes
FileNumber XXX bytes
For each
FilePathName XXX bytes
00 1 bytes

Body
XXX bytes
-------------------------------
Type 2
-------------------------------
MagicNumber XXX bytes
FileNumber XXX bytes
For each
FilePathName XXX bytes
offset 4 bytes
length 4 bytes

Body
XXX bytes
-------------------------------
Type 3
-------------------------------
FileNumber XXX bytes
For each
FilePathName XXX bytes
offset 4 bytes
length 4 bytes

Body
XXX bytes

and so on

In the ext.ini your just to define the value of every protities of "FilePathName","offse" and so on and the extract TYPE .
The main logic is in the ext.exe.
I wish the tool be made in command line because it can be used in a translation patch to unpacking and repacking.

[aluigi]

"if" I will work on the tool it will be fore sure in double mode: command-line tool if launched from the console (cmd.exe), minimalistic GUI if launched with double-click on the executable.

in the meantime I need to find the best way/format for the script, for example the idea of simply declaring the fields of the file format (so not programming the tool through the script but simply the structure of the file) and then let the tool to do the rest seems something very interesting.

well, for the moment dreaming doesn't cost anything :)

[aluigi]

well I have some good news.
yesterday and today I have worked on the script parser and the results are so good that my beta version already works.

I have made it compatible with the BMS scripting language so people don't need to learn 1000 languages and then it's a good language.
obviously the BMS support has been enhanced so it's possible to use XOR, rot13, zlib/deflate, lzo, lzss plus various new operator in the "math" command (and, or, xor, complement, not, shift and modulus) and some other things like choosing the desired endianness.
then the input BMS script no longer needs the semicolon ';' and the file number which are now optional.

note that the BMS guide available on http://wiki.xentax.com/index.php/BMS doesn't report some additional fields which instead already exists like the GetCT, ComType and ReverseLong or some internals variables like EXTRCNT so I have tried to find and add also these ones.

about the new commands, the following is an example of BMS script for the game Outcry (you can call it outcry.bms or outcry.txt or as you want):

Code: Select all

idstring SBPK
get FILES long

savepos NAMES_OFFSET
for i = 1 to FILES
    get FNAMESZ long
    getdstring  FNAME FNAMESZ
    get OFFSET  long
    get SIZE    long
next i

savepos FILES_OFFSET
goto NAMES_OFFSET
for i = 1 to FILES
    get FNAMESZ long
    getdstring  FNAME FNAMESZ
    get OFFSET  long
    get SIZE    long
    math OFFSET += FILES_OFFSET

    filexor 0xcc
    log FNAME OFFSET SIZE 0 0
    filexor 0
next i

as visible it's files are XORed with the byte 0xcc.

now the link to the beta version:

http://aluigi.org/papers/quickbms.zip

obviously it needs to be tested more because 2 days are not enough for a project like this which is not so basic so anyone is invited to test it and suggest new features.

P.S.: the usage is very simple and the following are some examples:
listing the files in the archive common.spk of outcry:

Code: Select all

quickbms -l outcry.bms c:\common.spk .

extracting the files in c:\folder:

Code: Select all

quickbms outcry.bms c:\common.spk c:\folder

extracing or listing only the dds files:

Code: Select all

quickbms -f "*.dds" outcry.bms c:\common.spk c:\folder

[stevenx]

Wow,it is a real good news!
If repacking is possible, it will be amazing.
I　will think more fearure for it.

[stevenx]

Does it works when the file name table at the end of the file?

[aluigi]

if you know the size of the structure or where is located the particular value you are searching (for example some file formats have a 32bit number in the last 4 bytes which has the offset where are located the other file structures) you could use something like the following:

Code: Select all

get OFFSET asize
math OFFSET -= 4
goto OFFSET
get OFFSET long
goto OFFSET
...

"asize" is not available in the BMS guide but is supported by both multiex and my tool, it contains the total size of the file.
anyway tell me a file format which has the file name table at the end, it's good to make various exercises for testing the BMS language and where is possible to enhance it.

oh and about the "repacking" question, no it will be not supported

*EDIT*: updated version to 0.1.1 and attached 3 examples of BMS files which show different operations or different types for doing the same thing (7x7m.bms and 7x7m_special.bms)

[stevenx]

File Format for test:
(Can you make a script for it?)

game name :
Evil Days of Luckless John

web:
http://www.centauriproduction.com/raide ... casino.php

Version :
Centauri Production Resource File 3.00 (not 3.01)

File Format:
Header
string Header; // "Centauri Production Resource File 3.00\n\n\0"
int FilesCount;
int TOCOffset;

ToC
string FileName;
long FileTime;
int isPacked;
int Offset;
int UnpackedSize;
int PackedSize;
int unk1; // Could be crc or something else.

[stevenx]

Another test file :
Dracula Origin

viewtopic.php?f=10&t=3061&start=0&st=0& ... ula+Origin

Can you test your tool with this file and make a script sample?

[aluigi]

the script for Centauri is attached although I have tested it only with the gooka2 game (the only one for which there is a demo available on the centauri websites).

important note: don't use "int" as size specifier in the examples because in the BMS scripts "int" is used for the 16 bit fields while here you meant a 32 bit one. I tell you because this can create confusion (indeed my tool supports various aliases for these size identifiers like u32, u32, short, u16, 16, u8, 8 and so on).

no luck with the dracula origin format because the structure with the filenames is not clear and I guess it's the compressed block at the end of file. unfortunately at the moment there are no ways (in the BMS format) to parse formats which have the filename table compressed so maybe in future I will find an alternative solution/command if necessary.

[stevenx]

Very thanks.

Some advice.
1. I think will have many updates for this tool,so can you add a -ver option to show the tool's version?

2. If the archive have 2 files ,one is index ,one is data,looks like abc.idx, abc.dat.
Can this tool extract this kind of archive ?
(For example: So Blonde )

[stevenx]

For test :

This game's file name table is at the end of the game.
Under cover: Operation Wintersun