XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Mon Jun 26, 2017 8:09 am

All times are UTC + 1 hour


Forum rules


Please click here to view the forum rules



Post new topic Reply to topic  [ 8 posts ] 
Author Message
 Post subject: Help with SQX files from KGB / Conspiracy
PostPosted: Fri Aug 05, 2016 6:05 am 
Offline
n00b
User avatar

Joined: Sat Sep 29, 2012 10:20 pm
Posts: 15
Has thanked: 2 times
Have thanks: 0 time
Hi! I'm currently trying to reverse engineer the music from Cryo Interactive's 1992 game KGB, re-released as Conspiracy. In order to get to the music files, I have to decompress the files first. The game uses a compression algorithm called SQX, that's exclusive to this game. What's important is that it is very similar to a known format, HSQ (which is used by Dune and MegaRace). Nevertheless, it is still slightly different and I could use your help.

Here's what we know:

    * SQX is very similar to HSQ. Both compression formats are based off of Lemple-Ziv and use Sliding Window RLE compression.

    * Both formats have a six byte header, however SQX's is different.

    * HSQ uses a checksum of 0xAB, which is calculated by adding the first six bytes of the header, to perform decompression. From what it looks like, SQX doesn't use a checksum.

    * Both formats use at least 18-byte chunks, a two-byte header that's used as a bitmap and the proceeding 16 or more bytes is the compressed data.

How do we know SQX is very similar to HSQ?

Because Cryo placed a song from Dune ("MORNING") in the assets for KGB and had it compressed in SQX instead of HSQ. So we have the MORNING song file in SQX, HSQ and in uncompressed format, which allows us to compare between the formats. Now, SQX is supported in the old OPL2/3 player RDOSPlay, but the source code is long gone, so we still don't know how the format works, however I did comfirm that the MORNING.SQX file does play correctly in RDOSPlay and sounds exactly like it's HSQ and uncompressed counterparts.

To help with analysis, I've attached the HSQ, SQX and uncompressed SDB files to this thread. You can easily see how similar SQX is to HSQ in a hex editor. I also attached the source code to a program that handles HSQ decompression so you can see how that format works. In addition, here's a link that shows the logic for HSQ's algorithm.

I understand this is a really obscure game and an even more obscure file format. But, if anyone can help out, it would mean so much to me. Thanks in advance!


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.



Top
 Profile  
 
 Post subject: Re: Help with SQX files from KGB / Conspiracy
PostPosted: Sat Aug 06, 2016 1:07 am 
Offline
ultra-veteran
ultra-veteran

Joined: Fri Nov 06, 2009 12:13 am
Posts: 579
Has thanked: 64 times
Have thanks: 123 times
someone has posted a youtube playlist for this game here - http://www.dosnostalgia.com/?p=664

i found the sourcecode for unhsq here - https://sourceforge.net/p/dunerevival/c ... gs/unhsq.c
it was missing from your post.

also, in another thread you said you haven't coded anything but hex-edited change? hardcode :?

_________________
Useful tool links:


Top
 Profile  
 
 Post subject: Re: Help with SQX files from KGB / Conspiracy
PostPosted: Sun Aug 07, 2016 4:15 am 
Offline
n00b
User avatar

Joined: Sat Sep 29, 2012 10:20 pm
Posts: 15
Has thanked: 2 times
Have thanks: 0 time
WRS wrote:
someone has posted a youtube playlist for this game here - http://www.dosnostalgia.com/?p=664

i found the sourcecode for unhsq here - https://sourceforge.net/p/dunerevival/c ... gs/unhsq.c
it was missing from your post.

also, in another thread you said you haven't coded anything but hex-edited change? hardcode :?


Sorry for forgetting to add the sourcecode to the initial post.

Yes, I created new music content for MegaRace, which uses a slightly updated version of the OPL2 music engine (known as HERAD) that Dune and KGB use, so I could potentially put custom music into those game with very little modification. It fairly easy to hex-edit your own MIDI data into the music files, it just takes time because it's very tedious. I used preexisting MegaRace data for the instruments. I did the majority of reverse-engineering the HERAD music format by playing back hex-edited files in both MegaRace and RDOSPlay.

https://www.youtube.com/watch?v=TqBJedId4g4

https://www.youtube.com/watch?v=bS-kgYqVIbM

KGB is the only game remaining that I don't have direct access to the uncompressed music files. There's some very interesting programming in some of the songs and I wanted to see how exactly those songs were created, in order to better document the music format. Using something like DRO2MIDI, to convert the OPL register values into MIDI isn't going to be enough.


Top
 Profile  
 
 Post subject: Re: Help with SQX files from KGB / Conspiracy
PostPosted: Wed Oct 19, 2016 8:59 pm 
Offline
ultra-n00b
User avatar

Joined: Wed Oct 19, 2016 6:27 pm
Posts: 5
Location: Russia, Moscow
Has thanked: 0 time
Have thanks: 1 time
MrSinistar wrote:
    * Both formats have a six byte header, however SQX's is different.

    * HSQ uses a checksum of 0xAB, which is calculated by adding the first six bytes of the header, to perform decompression. From what it looks like, SQX doesn't use a checksum.

    * Both formats use at least 18-byte chunks, a two-byte header that's used as a bitmap and the proceeding 16 or more bytes is the compressed data.

It's obvious that these two compression algorithms are similar. Most bytes are the same till offset 0x240.

But SQX using different bit mask method, and probably improved due to smaller compressed size.


Top
 Profile  
 
 Post subject: Re: Help with SQX files from KGB / Conspiracy
PostPosted: Wed Oct 19, 2016 10:21 pm 
Offline
ultra-n00b
User avatar

Joined: Wed Oct 19, 2016 6:27 pm
Posts: 5
Location: Russia, Moscow
Has thanked: 0 time
Have thanks: 1 time
Also one interesting thing, according to the game files, there are different types of SQX depending on byte at offset 3.

Default queue bits are:
HSQ: 0xFF
SQX-0: 0x00
SQX-1: 0x55
SQX-2: 0xFF


Top
 Profile  
 
 Post subject: Re: Help with SQX files from KGB / Conspiracy
PostPosted: Thu Oct 20, 2016 1:33 am 
Offline
ultra-n00b
User avatar

Joined: Wed Oct 19, 2016 6:27 pm
Posts: 5
Location: Russia, Moscow
Has thanked: 0 time
Have thanks: 1 time
Value at offset 5 specifies how many bits are used in counter.

Examples:
val = 3: 3 bits for count, 13 bits for offset
val = 4: 4 bits for count, 12 bits for offset
etc.

In SQX-0 algorithm the first bit is checked for zero. Other logic is same as HSQ except variable bits for count/offset.

SQX-1 has much more differences. (SQX-2 is not used in music files, I'm not planning to inspect it.)


Top
 Profile  
 
 Post subject: Re: Help with SQX files from KGB / Conspiracy
PostPosted: Tue Feb 28, 2017 9:36 am 
Offline
ultra-n00b
User avatar

Joined: Wed Oct 19, 2016 6:27 pm
Posts: 5
Location: Russia, Moscow
Has thanked: 0 time
Have thanks: 1 time
I've implemented a decompression program, however the SQX-1 algorithm is still not finished.
The decompressed data at first looks good, but only for few bytes.

Attached a program source with test files (compressed and decompressed - ripped from game memory, running in DOSBox).


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.



Top
 Profile  
 
 Post subject: Re: Help with SQX files from KGB / Conspiracy
PostPosted: Wed Apr 05, 2017 9:54 pm 
Offline
ultra-n00b
User avatar

Joined: Wed Oct 19, 2016 6:27 pm
Posts: 5
Location: Russia, Moscow
Has thanked: 0 time
Have thanks: 1 time
Finally, the universal HSQ / SQX decompressor is done!

Big thanks goes to -=CHE@TER=- who helped a lot with disassembling the original unpacker code from KGB game :keke:
https://www.extractor.ru/ipb/index.php?showtopic=8767

I converted disassembly to Delphi code, and now it works excellent! Attached archive contains the source code and compiled Win32 binary executable.

Have a nice day!

/thread


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.



Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group