Imperium Romanum - HPK Extractor

[Demander]

Hi there!

I made a archive extractor for Imperium Romanum,this game using a HPK archives (I didnt seen this in any game yet).

Into to archive:
Archive structure is pretty complex,its not encrypted but its using a few tricks how to fool hackers

Most of files are compressed with ZLIB(like Textures and XML) but every file may be splitted into 32KB chunks (offsets are in ZLIB Header),few files have "ZLIB" tag,but they are not compressed,usually because of small input buffer (for example chunk with 24 bytes is not compressed,but directly saved into file).You can also find a relative or absolute offsets of files or directories,which are also incorrect.

Game using also in few files a LZSS compression which is not supported in this Extractor ( I cannot find a correct library which will decompress LZSS files(I know structure of "LZSS" filles)).

USAGE:
HPKExtractor.exe [options] -hpk "YourFile.hpk"
Options are: "-zlib" - This will decompress all ZLIB files.

Game with work with compressed and decompressed files too,but if you will using decompressed files then you may notice a slight decrease of game performance.

Game will work with extracted archives,only one thing is need to be done to work:
Move all HPK archives from subfolders into main folder (where is exe located) then extract all archives,delete all hpk archives and folders Local and Packs.
Thats all.

Request:
1. I'm looking for decompiler for a LUA files (5.1)
2. I'm looking for a LZSS library which will be able to decompress "LZSS" files
Any help will be appreciated

Thank you for any help.

[Rheini]

Request:
1. I'm looking for decompiler for a LUA files (5.1)

http://luadec.luaforge.net/
(though this one targets lua 5.02, it might be of help)

2. I'm looking for a LZSS library which will be able to decompress "LZSS" files

Some LZSS decompression code can be found here: viewtopic.php?p=17531#p17531
But of course this might be a variation of LZSS, so you still have to figure out the correct parameters.

[Demander]

Thank you for your reply
1. LUADec works only for 5.0 not 5.1 I tried it sooner and it wont works.

2. I'll look at that,I found in main exe a procedure which will decompress a LZSS,its pretty long procedure but there is no calls in another parts in exe (there is only byte operations),so I ripped that procedure from exe to my app,I know a parameters to this procedure but I'm unable to add them into correct registers (I dont know asm well,so I need to help with that).

LZSS procedure looks like same as ZLIB,but I dont know last parameter.

procedure LZSSDecode (const InBuffer:Pointer;const InSize:Integer;out OutBuffer:Pointer;out OutSize:Integer;Unknown:Integer);

[Rheini]

zlib actually uses LZSS combined with Huffman entropy coding.
you might post a text file containing the code you ripped if you want help with it.

btw Antivir detects the trojan horse TR/Dldr.Delf.T.3 in HPKExtractor.exe

[Demander]

I dont known,if its legal to releaseng parts of code from PE probably not...

But I can post here a LZSS file,which is created by game but its not in game included:
UserConfig.dat

Header:

Magic: 4 Bytes
Original FileSize: 4 Bytes
Chunk Size: 4 Bytes
Unknown: 2 Bytes - Probably parameters to LZSS decompression algo
ChunkOffset: 4 Bytes - Offset to first chunk

Next bytes are compressed data.

There are uncompressed data,LF is linefeed,Tab is tab key .. everything should be without commas in uncompressed file

Code: Select all

ASCII "userconfig.id_old_rect =",LF,"{",LF,TAB,"idTabletDlg = false;",LF,TAB,"idEconomyOverviewDlg = box(point(197, 173), point(826, 595));",LF,TAB,"idSettlementOverviewDlg = box(point(197, 173), point(826, 595));",LF,"};",LF,"userconfig.GameKe"...

[Demander]

zlib actually uses LZSS combined with Huffman entropy coding.
you might post a text file containing the code you ripped if you want help with it.

btw Antivir detects the trojan horse TR/Dldr.Delf.T.3 in HPKExtractor.exe

Really ? it works for me normally,it may be because of used packer,I already uploaded it again with different PE compressor.

[Rheini]

Still detects that trojan horse. What packer do you use?
btw which programming language do you use for your tool?

[Demander]

Maybe its because of your Antivirus program,I'm using latest AVG antivirus,and I dont have any reports because of viruses.
and I'm writing in Object Pascal (delphi)

[Rheini]

Maybe you could just use upx or MEW?

[Demander]

Maybe you could just use upx or MEW?

Okey I packed it for last time If it will report you any virus or horse then you have really bad antivirus

[Rheini]

Thank you for your patience It works now.

I dont known,if its legal to releaseng parts of code from PE probably not...

But you might tell us the function VA (Virtual Address)

Into to archive:
Archive structure is pretty complex,its not encrypted but its using a few tricks how to fool hackers

Care to share your knowledge?

[Demander]

Hi again

1) I can tell you where that procedure begins But better will be a offset where is function called,to see what parameters going to this procedure.

Go to "0066B1C0",this is procedure where is called a procedure of LZSS decompression
(0066A160).
(I'm using patch 1.1)

Maybe somebody can rewrite that procedure to asm block,it doesnt matter which language you are using,so it can be universal to all languages.

But probably,it will be better to new function (because of legal issues)....

2) I'll write a HPK structure later,I still dont know few bytes (most of them are same in every hpk archive,but its better to know them because of archive rebuilding)

3) I found that game using a next compression algo I think its used only for savegame files (*.sav),strange thing is that "magic" is "LZIS",I dont know if its any kind of compression algo or its just LZS.

EDIT:
"LZIS" structure is similar as "ZLIB" and "LZSS":
4 Bytes - magic
4 Bytes - original filesize (yeah original saved games has over 13MB per each)
4 Bytes - chunk size
2 Bytes - unknown (probably parameters to decompression algo)

Next bytes are offsets to chunks,every chunk can be smaller than chunk size,but probably should not be larger.

I made a screenshot of compressed file,selected bytes are beginning of compressed chunk,maybe somebody will recognize a compression algo (beginning of every compressed chunk seems to be same).

http://img150.imageshack.us/img150/1323/86403044id5.png

[Rheini]

I made a screenshot of compressed file,selected bytes are beginning of compressed chunk,maybe somebody will recognize a compression algo (beginning of every compressed chunk seems to be same).

http://img150.imageshack.us/img150/1323/86403044id5.png

As I already said before, there's too much redundancy in the first part (all those 0x6666 and 0x0000 for example)
That part can't be compressed. Did you try to compress it again (e.g. with WinRAR or 7zip)? This way you can get a hint if the data is already compressed.

[Demander]

Its compressed I know it from header... original filesize is saved here (I'm 100% sure)

I splitted one chunk from saved game,maybe it will help to find a compression algo.

[Demander]

Nobody to help ?:P Please its important or I cant finish extractor.
I just need somebody who knows assembler language,for function calling explanation...