XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Fri Oct 19, 2018 8:57 pm

All times are UTC + 1 hour


Forum rules


Please click here to view the forum rules



Post new topic Reply to topic  [ 139 posts ]  Go to page 1, 2, 3, 4, 5 ... 10  Next
Author Message
 Post subject: Files extractors scripting
PostPosted: Mon Apr 13, 2009 3:49 pm 
Offline
VVIP member
VVIP member
User avatar

Joined: Thu Dec 08, 2005 12:26 pm
Posts: 1925
Location: www.ZENHAX.com
Has thanked: 4 times
Have thanks: 644 times














You can make the ads go away by registering

a quick question: what are ALL the current available solutions for writing file extractors for simple formats through scripting?

an example of which I'm aware is the mexcom scripting but if I'm not in error it doesn't support things like zlib/deflate decompression and XORing (correct me if I'm wrong) but most important it works only with the registered version of multiex (correct me here too if needed) so it must be not considered.

the idea of creating file extractors on the fly with any text editor "stimulates" a lot my brain but obviously it's useless to reinvent the wheel if already exists a good solution.


Top
 Profile  
 
 Post subject: Re: Files extractors scripting
PostPosted: Mon Apr 13, 2009 5:06 pm 
Offline
Moderator
User avatar

Joined: Sun May 18, 2008 3:01 pm
Posts: 2486
Has thanked: 57 times
Have thanks: 1239 times
I love your Idea of writing a file extractor that could be customized through some type of xml file or settings dialog.
This would not be easy to complete but would be very worth it in the end.
The only ripper that I know of is jaedernaub but I do not know how to add file formats to it.


Top
 Profile  
 
 Post subject: Re: Files extractors scripting
PostPosted: Mon Apr 13, 2009 10:00 pm 
Offline
Moderator

Joined: Wed Oct 18, 2006 9:48 pm
Posts: 651
Location: Germany
Has thanked: 19 times
Have thanks: 41 times
Well you can use MexCom scripts with GameExtractor.
I personally played around with Lua with added binary operations some time ago.

_________________
Image

Remember: If you don't want to program a tool yourself, hack another one :wink:
__________
http://www.gameformats.de.vu


Top
 Profile  
 
 Post subject: Re: Files extractors scripting
PostPosted: Tue Apr 14, 2009 1:00 am 
Offline
veteran

Joined: Sat Nov 01, 2008 12:02 pm
Posts: 130
Has thanked: 3 times
Have thanks: 0 time
Need command line extractor looks like ext.exe ,ext.ini.
It is useful when need repack.


Top
 Profile  
 
 Post subject: Re: Files extractors scripting
PostPosted: Tue Apr 14, 2009 4:37 am 
Offline
veteran

Joined: Sat Nov 01, 2008 12:02 pm
Posts: 130
Has thanked: 3 times
Have thanks: 0 time
I think not exists a perfact extractor now.
My idea is looks like this:

At first define some struct of file format.
For example

-------------------------------
Type 1
-------------------------------
Head
MagicNumber XXX bytes
FileNumber XXX bytes
For each
FilePathName XXX bytes
00 1 bytes

Body
XXX bytes
-------------------------------
Type 2
-------------------------------
MagicNumber XXX bytes
FileNumber XXX bytes
For each
FilePathName XXX bytes
offset 4 bytes
length 4 bytes

Body
XXX bytes
-------------------------------
Type 3
-------------------------------
FileNumber XXX bytes
For each
FilePathName XXX bytes
offset 4 bytes
length 4 bytes

Body
XXX bytes

and so on

In the ext.ini your just to define the value of every protities of "FilePathName","offse" and so on and the extract TYPE .
The main logic is in the ext.exe.
I wish the tool be made in command line because it can be used in a translation patch to unpacking and repacking.


Top
 Profile  
 
 Post subject: Re: Files extractors scripting
PostPosted: Tue Apr 14, 2009 11:25 pm 
Offline
VVIP member
VVIP member
User avatar

Joined: Thu Dec 08, 2005 12:26 pm
Posts: 1925
Location: www.ZENHAX.com
Has thanked: 4 times
Have thanks: 644 times
"if" I will work on the tool it will be fore sure in double mode: command-line tool if launched from the console (cmd.exe), minimalistic GUI if launched with double-click on the executable.

in the meantime I need to find the best way/format for the script, for example the idea of simply declaring the fields of the file format (so not programming the tool through the script but simply the structure of the file) and then let the tool to do the rest seems something very interesting.

well, for the moment dreaming doesn't cost anything :)


Top
 Profile  
 
 Post subject: Re: Files extractors scripting
PostPosted: Thu Apr 16, 2009 9:35 pm 
Offline
VVIP member
VVIP member
User avatar

Joined: Thu Dec 08, 2005 12:26 pm
Posts: 1925
Location: www.ZENHAX.com
Has thanked: 4 times
Have thanks: 644 times
well I have some good news.
yesterday and today I have worked on the script parser and the results are so good that my beta version already works.

I have made it compatible with the BMS scripting language so people don't need to learn 1000 languages and then it's a good language.
obviously the BMS support has been enhanced so it's possible to use XOR, rot13, zlib/deflate, lzo, lzss plus various new operator in the "math" command (and, or, xor, complement, not, shift and modulus) and some other things like choosing the desired endianness.
then the input BMS script no longer needs the semicolon ';' and the file number which are now optional.

note that the BMS guide available on http://wiki.xentax.com/index.php/BMS doesn't report some additional fields which instead already exists like the GetCT, ComType and ReverseLong or some internals variables like EXTRCNT so I have tried to find and add also these ones.

about the new commands, the following is an example of BMS script for the game Outcry (you can call it outcry.bms or outcry.txt or as you want):
Code:
idstring SBPK
get FILES long

savepos NAMES_OFFSET
for i = 1 to FILES
    get FNAMESZ long
    getdstring  FNAME FNAMESZ
    get OFFSET  long
    get SIZE    long
next i

savepos FILES_OFFSET
goto NAMES_OFFSET
for i = 1 to FILES
    get FNAMESZ long
    getdstring  FNAME FNAMESZ
    get OFFSET  long
    get SIZE    long
    math OFFSET += FILES_OFFSET

    filexor 0xcc
    log FNAME OFFSET SIZE 0 0
    filexor 0
next i
as visible it's files are XORed with the byte 0xcc.

now the link to the beta version:

http://aluigi.org/papers/quickbms.zip

obviously it needs to be tested more because 2 days are not enough for a project like this which is not so basic so anyone is invited to test it and suggest new features.

P.S.: the usage is very simple and the following are some examples:
listing the files in the archive common.spk of outcry:
Code:
quickbms -l outcry.bms c:\common.spk .

extracting the files in c:\folder:
Code:
quickbms outcry.bms c:\common.spk c:\folder

extracing or listing only the dds files:
Code:
quickbms -f "*.dds" outcry.bms c:\common.spk c:\folder


Top
 Profile  
 
 Post subject: Re: Files extractors scripting
PostPosted: Thu Apr 16, 2009 11:16 pm 
Offline
veteran

Joined: Sat Nov 01, 2008 12:02 pm
Posts: 130
Has thanked: 3 times
Have thanks: 0 time
Wow,it is a real good news!
If repacking is possible, it will be amazing.
I will think more fearure for it.


Top
 Profile  
 
 Post subject: Re: Files extractors scripting
PostPosted: Fri Apr 17, 2009 12:56 am 
Offline
veteran

Joined: Sat Nov 01, 2008 12:02 pm
Posts: 130
Has thanked: 3 times
Have thanks: 0 time
Does it works when the file name table at the end of the file?


Top
 Profile  
 
 Post subject: Re: Files extractors scripting
PostPosted: Fri Apr 17, 2009 1:40 am 
Offline
VVIP member
VVIP member
User avatar

Joined: Thu Dec 08, 2005 12:26 pm
Posts: 1925
Location: www.ZENHAX.com
Has thanked: 4 times
Have thanks: 644 times
if you know the size of the structure or where is located the particular value you are searching (for example some file formats have a 32bit number in the last 4 bytes which has the offset where are located the other file structures) you could use something like the following:
Code:
get OFFSET asize
math OFFSET -= 4
goto OFFSET
get OFFSET long
goto OFFSET
...
"asize" is not available in the BMS guide but is supported by both multiex and my tool, it contains the total size of the file.
anyway tell me a file format which has the file name table at the end, it's good to make various exercises for testing the BMS language and where is possible to enhance it.

oh and about the "repacking" question, no it will be not supported

*EDIT*: updated version to 0.1.1 and attached 3 examples of BMS files which show different operations or different types for doing the same thing (7x7m.bms and 7x7m_special.bms)


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.



Top
 Profile  
 
 Post subject: Re: Files extractors scripting
PostPosted: Fri Apr 17, 2009 4:39 am 
Offline
veteran

Joined: Sat Nov 01, 2008 12:02 pm
Posts: 130
Has thanked: 3 times
Have thanks: 0 time
File Format for test:
(Can you make a script for it?)

game name :
Evil Days of Luckless John

web:
http://www.centauriproduction.com/raide ... casino.php

Version :
Centauri Production Resource File 3.00 (not 3.01)

File Format:
Header
string Header; // "Centauri Production Resource File 3.00\n\n\0"
int FilesCount;
int TOCOffset;

ToC
string FileName;
long FileTime;
int isPacked;
int Offset;
int UnpackedSize;
int PackedSize;
int unk1; // Could be crc or something else.


Top
 Profile  
 
 Post subject: Re: Files extractors scripting
PostPosted: Fri Apr 17, 2009 4:49 am 
Offline
veteran

Joined: Sat Nov 01, 2008 12:02 pm
Posts: 130
Has thanked: 3 times
Have thanks: 0 time
Another test file :
Dracula Origin

viewtopic.php?f=10&t=3061&start=0&st=0&sk=t&sd=a&hilit=Dracula+Origin

Can you test your tool with this file and make a script sample?


Top
 Profile  
 
 Post subject: Re: Files extractors scripting
PostPosted: Fri Apr 17, 2009 2:24 pm 
Offline
VVIP member
VVIP member
User avatar

Joined: Thu Dec 08, 2005 12:26 pm
Posts: 1925
Location: www.ZENHAX.com
Has thanked: 4 times
Have thanks: 644 times
the script for Centauri is attached although I have tested it only with the gooka2 game (the only one for which there is a demo available on the centauri websites).

important note: don't use "int" as size specifier in the examples because in the BMS scripts "int" is used for the 16 bit fields while here you meant a 32 bit one. I tell you because this can create confusion (indeed my tool supports various aliases for these size identifiers like u32, u32, short, u16, 16, u8, 8 and so on).

no luck with the dracula origin format because the structure with the filenames is not clear and I guess it's the compressed block at the end of file. unfortunately at the moment there are no ways (in the BMS format) to parse formats which have the filename table compressed so maybe in future I will find an alternative solution/command if necessary.


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.



Top
 Profile  
 
 Post subject: Re: Files extractors scripting
PostPosted: Fri Apr 17, 2009 4:07 pm 
Offline
veteran

Joined: Sat Nov 01, 2008 12:02 pm
Posts: 130
Has thanked: 3 times
Have thanks: 0 time
Very thanks.

Some advice.
1. I think will have many updates for this tool,so can you add a -ver option to show the tool's version?

2. If the archive have 2 files ,one is index ,one is data,looks like abc.idx, abc.dat.
Can this tool extract this kind of archive ?
(For example: So Blonde )


Last edited by stevenx on Fri Apr 17, 2009 4:11 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Files extractors scripting
PostPosted: Fri Apr 17, 2009 4:08 pm 
Offline
veteran

Joined: Sat Nov 01, 2008 12:02 pm
Posts: 130
Has thanked: 3 times
Have thanks: 0 time
For test :

This game's file name table is at the end of the game.
Under cover: Operation Wintersun


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 139 posts ]  Go to page 1, 2, 3, 4, 5 ... 10  Next

All times are UTC + 1 hour


Who is online

Users browsing this forum: Google [Bot] and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group