READ THE RULES: Click here

Follow us on Facebook: https://www.facebook.com/xentax/ :)

QUICKBMS GUIDE 2 ZLIB

Read or post any tutorial related to file format analysis for modding purposes.
User avatar
chrrox
Moderator
Posts: 2537
Joined: Sun May 18, 2008 3:01 pm
Has thanked: 57 times
Been thanked: 1263 times

QUICKBMS GUIDE 2 ZLIB

Post by chrrox » Tue Jun 09, 2009 4:47 am

Ok so now that you have mastered tutorial 1 I am going to show you how to work with compressed archives.
We will be working with the game Astro Ranger
Here is the full client http://astro.nefficient.co.kr/astrorang ... client.exe

ok so set up our c:\temp directory bye extracting our file BoneObject.hsp
1. to c:\temp
2. create a new text document called astro.bms
3. and place the newest version of quickbms in the folder also.

Ok so open up BoneObject.hsp in your hex editor and lets take a look at it.
Image
good we have some plain text.
you will notice I highlighted the first 4 bytes 20 50 53 48 or " PSH" that is a space followed bye P S H.
hmm that seems familiar that is the file extension only backwards. this is know as the idstring
so up until now you would think to write in bms
get IDSTRING long
there is nothing wrong with that but there is a better command
idstring " PSH"
make sure you include the quotes.
so open your bms string and on the first line type
idstring " PSH"
the reason this command is better is it will tell the program not to run if it does not find that string don't try to extract that file.
"aka noob proofing it"

Ok so now lets look at what we can read I see
Datas\Texture\BoneObject\npc_nagoya_octopus01_body.dds , Datas\Texture\BoneObject\Toon.bmp , Datas\Texture\BoneObject\Toon_a.bmp , Datas\Texture\BoneObject\Toon_zero.bmp
so I will assume there are 4 files in this archive.
well lets look at the next 4 bytes and see what it is 01 00 00 00 hmm that is equal to 00 00 00 01 or 1 and we have more files in this archive than that so we do not know what this represents
so lets write that in bms language
get UNK1 long
this saves those 4 bytes as the variable UNK1.

ok the next 4 bytes are 04 00 00 00 hmm this translates into 00 00 00 04 or 4
hey that is the number of files we counted so lets write that in bms
get FILES long
this saves those 4 bytes as the variable FILES.

the next 4 bytes are 00 00 00 00 well that is equal to zero so for now I will write that in bms
get NULL1 long
this saves those 4 bytes as the variable NULL1

ok now we have reached the first file name Datas\Texture\BoneObject\npc_nagoya_octopus01_body.dds
this is 0x36 bytes long but wait there was no indicator like the last file that told us how long the name is how do we write this?
well lets look for a pattern
Datas\Texture\BoneObject\npc_nagoya_octopus01_body.dds is 0x36
Datas\Texture\BoneObject\Toon.bmp is 0x21
Datas\Texture\BoneObject\Toon_a.bmp is 0x23
Datas\Texture\BoneObject\Toon_zero.bmp is 0x26
hmm I don't see anything that makes that a pattern.
but I do see all the names are followed bye a lot of zero's. how long is the name + the zeros of each file?
Datas\Texture\BoneObject\npc_nagoya_octopus01_body.dds + 0's is 0x80
Datas\Texture\BoneObject\Toon.bmp + 0's is 0x80
Datas\Texture\BoneObject\Toon_a.bmp + 0's is 0x80
Datas\Texture\BoneObject\Toon_zero.bmp + 0's is 0x80
hey they are all the same size when I include the 0's
so in bms I would write this as
getdstring NAME 0x80
this tells it to grab 0x80 bytes and store the text value of it
and as an added feature it will automatically remove trailing 0's :)

ok so now we have 0xC bytes before I see the next file name
which is 3 long values
so lets write those in and we will figure out what they represent later.
get UNK2 long
get UNK3 long
get UNK4 long

ok so now we see the name again
we have our pattern so lets write our script based on what we learned
so it would look like this up until now

Code: Select all

idstring " PSH"
get UNK1 long
get FILES long
get NULL1 long
for i = 0 < FILES
getdstring NAME 0x80
get UNK2 long
get UNK3 long
get UNK4 long
clog NAME OFFSET ZSIZE SIZE
next i
ok this may look complex but it is almost identical to the first tutorial file except we added 1 more variable
ZSIZE this represents the compressed file size while SIZE represents the decompressed file size
and we also changed the log command to clog to represent it is a compressed file.

ok so now we have our loop and the commands to extract our files but we still need to fill in the variables
OFFSET ZSIZE SIZE
so that means our 3 unknown values must represent that but how do we know what order they are in?

Well ill let you in on a cool trick follow the file loop to the end start at Datas\Texture\BoneObject\npc_nagoya_octopus01_body.dds and highlight the whole 0x80 length
then add our 3 unknown variables so that means we are highlighting 0x8C for our length. the first file is from 0x10 - 0x9B
so now do this for the rest of the files and you end up at highlighting 0x1B4 - 0x23F
Image
ok so we reached the end of our loop now what?
well the next 2 bytes are 78 9C and this is an archive extractors best friend when you see this at the start of a file.
78 9C represents the standard zlib compression header :)
ok so this means our first file starts there which is at offset 0x240

well lets go back to our first file in the list and look at those unknown variables.
24 72 00 00 is = 00 00 72 24 = 0x7224
80 00 02 00 is = 00 02 00 80 = 0x20080
40 02 00 00 is = 00 00 02 40 = 0x240
I think we have a winner so the third variable is 0x240 aka the offset
so lets update our script

Code: Select all

idstring " PSH"
get UNK1 long
get FILES long
get NULL1 long
for i = 0 < FILES
getdstring NAME 0x80
get UNK2 long
get UNK3 long
get OFFSET long
clog NAME OFFSET ZSIZE SIZE
next i
now that just leaves ZSIZE and SIZE
well bye process of elimination the decompressed file must be bigger than the compressed file so we compare the 2 variables
24 72 00 00 is = 00 00 72 24 = 0x7224
80 00 02 00 is = 00 02 00 80 = 0x20080
well 0x20080 is definitely bigger so we now know the last 2 variables

Code: Select all

idstring " PSH"
get UNK1 long
get FILES long
get NULL1 long
for i = 0 < FILES
getdstring NAME 0x80
get ZSIZE long
get SIZE long
get OFFSET long
clog NAME OFFSET ZSIZE SIZE
next i
now try our code out on the file
open the command prompt and change to the directory
c:\temp
now type
quickbms.exe -l astro.bms BoneObject.hsp .
yay it listed our files without any errors now lets try extracting them
create a folder called extract
and type
quickbms.exe astro.bms BoneObject.hsp extract
if we look in there we now have folders and in those folders are 4 pictures :)
we did it.

Let me know if you want more pictures or any way I can improve the tutorials.
Last edited by chrrox on Tue Jun 09, 2009 2:33 pm, edited 1 time in total.

Rheini
Moderator
Posts: 652
Joined: Wed Oct 18, 2006 9:48 pm
Location: Germany
Has thanked: 19 times
Been thanked: 40 times
Contact:

Re: QUICKBMS GUIDE 2 ZLIB

Post by Rheini » Tue Jun 09, 2009 12:13 pm

chrrox wrote:Let me know if you want more pictures or any way I can improve the tutorials.
As I already commented, you could polish its look a bit, e.g. a neat little pdf

Rahly
VVIP member
VVIP member
Posts: 412
Joined: Thu Aug 05, 2004 10:17 am
Been thanked: 1 time

Re: QUICKBMS GUIDE 2 ZLIB

Post by Rahly » Tue Jun 09, 2009 1:17 pm

You sure are a pdf junkie :)
"By nature men are alike. Through practice they have become far apart." Confucius (Analect 17:2)

Rheini
Moderator
Posts: 652
Joined: Wed Oct 18, 2006 9:48 pm
Location: Germany
Has thanked: 19 times
Been thanked: 40 times
Contact:

Re: QUICKBMS GUIDE 2 ZLIB

Post by Rheini » Tue Jun 09, 2009 1:25 pm

Not really, I'd prefer djvu (http://djvu.sourceforge.net/), but unfortunately it's not really well supported :wink:

User avatar
aluigi
VVIP member
VVIP member
Posts: 1917
Joined: Thu Dec 08, 2005 12:26 pm
Location: www.ZENHAX.com
Has thanked: 4 times
Been thanked: 644 times
Contact:

Re: QUICKBMS GUIDE 2 ZLIB

Post by aluigi » Tue Jun 09, 2009 2:41 pm

imho it's only needed a bit of usage of phpbb code without useless pdf stuff

pixellegolas
ultra-veteran
ultra-veteran
Posts: 423
Joined: Mon Aug 11, 2008 11:30 pm
Has thanked: 27 times
Been thanked: 15 times

Re: QUICKBMS GUIDE 2 ZLIB

Post by pixellegolas » Tue Jun 09, 2009 4:07 pm

instead of commenting the format or typos i'm gonna say:

good tutorial! I am impressed of how much you have learnt in such a small time!

User avatar
Gocha
veteran
Posts: 109
Joined: Fri Dec 12, 2008 8:16 pm
Location: Batumi, Georgia, GE
Has thanked: 57 times
Been thanked: 8 times
Contact:

Re: QUICKBMS GUIDE 2 ZLIB

Post by Gocha » Tue Jun 09, 2009 5:24 pm

I Love this Tuts 8D

I can't imagine what would happen if you make the tut on the Assassin's forge files - if it will happen i'd be in shock!
My great respect and appreciation for them, who research game files! Special thanks to: aluigi, bacter, DerPlaya, Rick, Turfster, twig, Zench. Sorry if someone is missing in my list, I'll update when I'll notice it again

Rheini
Moderator
Posts: 652
Joined: Wed Oct 18, 2006 9:48 pm
Location: Germany
Has thanked: 19 times
Been thanked: 40 times
Contact:

Re: QUICKBMS GUIDE 2 ZLIB

Post by Rheini » Tue Jun 09, 2009 5:34 pm

I just remembered, I also tried to create a tutorial some time ago, in fact a video tutorial, but I couldn't finish it.
Maybe this is some inspiration or whatever :ninja:

http://ul.to/v1ms63

Edit: This was the original topic: viewtopic.php?f=22&t=2867

User avatar
shekofte
mega-veteran
mega-veteran
Posts: 220
Joined: Sun Jan 18, 2009 1:45 pm
Location: Sagittarius
Has thanked: 240 times
Been thanked: 12 times
Contact:

Re: QUICKBMS GUIDE 2 ZLIB

Post by shekofte » Wed Jun 10, 2009 5:45 pm

excuse me ! Mr chrrox , bugtest
one question remind me ! about works that facilitated by QUICKBMS ?
is it possible to invoke an individual decomperession method ? like inflate !
for example i have :
A[DATA] : compressed with deflate .
how can i execute an inflate decompression on A[DATA] ?
if it is possible please write an example line ! thanks a lot for your exhilarate :spit:
ILLUSORY VIDEO GAMES ARE MORE IMPORTANT THAN PRECIOUS INTERNATIONAL FOOTBALLs

User avatar
aluigi
VVIP member
VVIP member
Posts: 1917
Joined: Thu Dec 08, 2005 12:26 pm
Location: www.ZENHAX.com
Has thanked: 4 times
Been thanked: 644 times
Contact:

Re: QUICKBMS GUIDE 2 ZLIB

Post by aluigi » Wed Jun 10, 2009 6:07 pm

you can specify the desired compression method using the comtype command.
by default it's the zlib one (RFC 1950, windowbits 15) so if you need the raw inflate (RFC 1951, windowbits -15) it's enough to use: comtype inflate

then the decompression can happen in the classical "clog FILENAME OFFSET ZSIZE SIZE" way but in case you need something in memory (from memory, to memory or both) you can do it through the usage of the memory files supported by quickbms.

you can find an example of the usage of inflate with the zip files here.

User avatar
chrrox
Moderator
Posts: 2537
Joined: Sun May 18, 2008 3:01 pm
Has thanked: 57 times
Been thanked: 1263 times

Re: QUICKBMS GUIDE 2 ZLIB

Post by chrrox » Wed Jun 23, 2010 11:41 pm

here is a list i found of zlib headers.
Here is the full set of 64:

Common:

78 01, 78 5e, 78 9c, 78 da

Rare:

08 1d, 08 5b, 08 99, 08 d7, 18 19, 18 57, 18 95, 18 d3,
28 15, 28 53, 28 91, 28 cf, 38 11, 38 4f, 38 8d, 38 cb,
48 0d, 48 4b, 48 89, 48 c7, 58 09, 58 47, 58 85, 58 c3,
68 05, 68 43, 68 81, 68 de

Very rare:

08 3c, 08 7a, 08 b8, 08 f6, 18 38, 18 76, 18 b4, 18 f2,
28 34, 28 72, 28 b0, 28 ee, 38 30, 38 6e, 38 ac, 38 ea,
48 2c, 48 6a, 48 a8, 48 e6, 58 28, 58 66, 58 a4, 58 e2,
68 24, 68 62, 68 bf, 68 fd, 78 3f, 78 7d, 78 bb, 78 f9

eycaramba
veteran
Posts: 86
Joined: Wed Sep 02, 2009 1:52 pm
Has thanked: 30 times
Been thanked: 1 time

Re: QUICKBMS GUIDE 2 ZLIB

Post by eycaramba » Sat Aug 14, 2010 10:37 am

Hi chrrox,
nice tutorial, comes in very handy :)
So, yeah. I tried this but I am insecure what I am doing wrong. I use this file:
Image

And wrote this script so far:

Code: Select all

idstring "ARC"
get IDNO short
get FILES short
for i = 0 < FILES
getdstring NAME 0x40
get UNK1 long
get ZSIZE long
get SIZE long
get OFFSET long
clog NAME OFFSET ZSIZE SIZE
next i
The compressed files always start at offset 0x8000, so there is much nullspace. Is there anything I have to add? Because BMS is spitting out an error :bleh:
I uploaded the example files here, btw: http://vuze.celes-network.de/tempfiles/ ... rk_arc.zip

(This is my first BMS script so forgive me :lol: )

User avatar
chrrox
Moderator
Posts: 2537
Joined: Sun May 18, 2008 3:01 pm
Has thanked: 57 times
Been thanked: 1263 times

Re: QUICKBMS GUIDE 2 ZLIB

Post by chrrox » Sat Aug 14, 2010 12:31 pm

To help you start off always use offzip.exe to scan a small file so you know what values you are looking for.
In the file you sent take title.arc.
the files extracted give the output.

Code: Select all

- open input file:    G:\title.arc
- enter in directory: G:\
- zip data to check:  32 bytes
- zip windowBits:     15
- seek offset:        0x00000000  (0)

+------------+-------------+-------------------------+
| hex_offset | blocks_dots | zip_size --> unzip_size |
+------------+-------------+-------------------------+
  0x00008000 ....... 13982 --> 460844
  0x0000b69e . 649 --> 1951


- 2 valid zip blocks found
So based on this our bms script would be

Code: Select all

get idstring long
get version short
get files short
goto 0x8
for i = 0 < files
getdstring name 0x40
get extension long
get zsize long
get size THREEBYTE
get fourty byte
get offset long
string name + .
string name + extension
clog name offset zsize size
next i
what pc game is this from capcom monster hunter?

eycaramba
veteran
Posts: 86
Joined: Wed Sep 02, 2009 1:52 pm
Has thanked: 30 times
Been thanked: 1 time

Re: QUICKBMS GUIDE 2 ZLIB

Post by eycaramba » Sat Aug 14, 2010 4:37 pm

Ah, I see! Thanks! Working like a charm :)
It is actually from the benchmark of Monster Hunter Frontier Online (Japan/Korea/Taiwan only).

fire29demon
ultra-n00b
Posts: 3
Joined: Fri May 11, 2012 7:26 pm
Has thanked: 2 times

Re: QUICKBMS GUIDE 2 ZLIB

Post by fire29demon » Sun Dec 15, 2013 8:17 am

To start off i didn't want to post on an old topic but i'm not sure where i would post this to get a reply from OP.

I want to extract an archive "xml.dat" http://rghost.net/50975532
It is encoded and any readable language in a hex editor is nonexistent since the text is in Chinese.

I have a snippet from the hex editor:
Image

My current quickBms script is pretty sad :(

Code: Select all

idstring "UOSEDALB"
get UNK1 byte
get NULL1 long
get NULL2 long
get NULL3 long
get FILES long
get UNK2 byte
get UNK3 byte
get NULL4 long
get NULL5 long
get NULL6 long
get NULL7 long
get NULL8 long
get NULL9 long
get NULL10 long
get NULL11 long
get NULL12 long
get NULL13 long
get NULL14 long
get NULL15 long
get NULL16 long
get NULL17 long
get NULL18 long
get NULL19 short
for i = 0 < FILES
getdstring NAME 

clog NAME OFFSET ZSIZE SIZE
next i
Obviously i'm missing a lot, my question is...on a type of file like this how do i go about figuring out the name of the files and where they stop/new offsets begin etc. since it is all unreadable. I know there's some extractor around these forums for this file type or similar but i want to learn how to do it myself. I have just started to take interest in reverse engineering and the like, so any help or nudge in the right direction would be appreciated, Thanks!

Post Reply