Offtopic: Into Commodore 64 (6502) coding, pixeling or music?
Xentax is looking for new members for the C64 activities!
Just drop us a message at forum@xentax.com and join the Scene Team!

Forum rules: Click here

Researching of a .dat archive with a .mnf file.

The Original Forum. Game archives, full of resources. How to open them? Get help here.
stalja
beginner
Posts: 32
Joined: Sat Sep 12, 2009 11:33 am
Has thanked: 10 times
Been thanked: 5 times

Re: Researching of a .dat archive with a .mnf file.

Post by stalja » Sat Jan 11, 2014 8:03 pm

For whatever reason the quickbms script does not extract the zosft file from esodata files. I managed to find the file in eso0002.dat but it contains only a small portion of filenames (something like 30k as compared to over 250k present). It also seems that the quickbms script does not extract some additional files for whatever reason. I might be wrong though.

stalja
beginner
Posts: 32
Joined: Sat Sep 12, 2009 11:33 am
Has thanked: 10 times
Been thanked: 5 times

Re: Researching of a .dat archive with a .mnf file.

Post by stalja » Tue Jan 14, 2014 4:58 pm

That bms script I posted several days ago will not properly unpack everything. Turns out it breaks when some of the empty archives are being called and eso0000.dat has files that are neither zlib nor snappy ( or something in the original bms script is not quite correct). Currently, I am skipping files that cause breaks in the script manually, but any help from the more experienced members would be welcome.

LennardF1989
ultra-n00b
Posts: 4
Joined: Fri Dec 31, 2010 12:54 am

Re: Researching of a .dat archive with a .mnf file.

Post by LennardF1989 » Wed Jan 15, 2014 1:39 pm

Heya,

Never been active on this forum a lot, always lurking and messing around by myself. Let my extraction program run for a good half an hour yesterday , it would go through all archives, decompress the information, and check the header against a local file. I had to skip eso0000 and everything beyond eso0106, because there's stuff in there which I suspect is double compressed (so first Zlib then Snappy or other way around, because those files have multiple entries in the second block of the MNF with different "flags").

There aren't a lot of special files in there though: .xml, .lua, .dds, .gr2, some soundbanks, some audio files, a few custom files and thats about it.

Besides a single ZOSFT in all the scanned eso packages I also encountered a ANFT, which I suspect is some sort of internal filetable. Haven't been able to find another ZOSFT, but it could be its hidden away in side the packages I skipped.

However, as someone in this topic said before: It's not unlikely some files are only used interally.
Disclaimer: I reverse formats for sports only, I have nothing to gain of it but information on how they work and a feeling of accomplishment... much like finally defeating a pesky boss in Dark Souls :D

LennardF1989
ultra-n00b
Posts: 4
Joined: Fri Dec 31, 2010 12:54 am

Re: Researching of a .dat archive with a .mnf file.

Post by LennardF1989 » Wed Jan 15, 2014 7:08 pm

Attached is a rar with logs for all archives.

It's safe to say that Block 3 in the MNFs contains all entries into the DAT files as all bytes are consumed when going through them (except 8 bytes in archive 4 for some reason). The MNF for ESO also contains references to archive 0 which don't exist, so either its residual or it's used for whatever more is in the MNF (eg. the unknown block).

All archives beyond 203 are simply empty, they only have the first 14 bytes of the DAT header. However, the MNF contains over 50k entries into these files, so I guess the content is supposed to be there, but it's stripped/disabled for the beta.

Other than that, as said in my previous post, not a lot of interesting things going on in the files.

With these logs, you/I can check which files are in the ZOSFT to give them a name and which are most likely for internal use only.

EDIT: Logs.rar now also contains list of all filenames found in the ZOSFT, sorted by filename and extension. As you can see, a lot of the files you encounter in the other logs are not mentioned in the ZOSFT.
You do not have the required permissions to view the files attached to this post.
Disclaimer: I reverse formats for sports only, I have nothing to gain of it but information on how they work and a feeling of accomplishment... much like finally defeating a pesky boss in Dark Souls :D

stalja
beginner
Posts: 32
Joined: Sat Sep 12, 2009 11:33 am
Has thanked: 10 times
Been thanked: 5 times

Re: Researching of a .dat archive with a .mnf file.

Post by stalja » Thu Jan 16, 2014 1:38 pm

I came to the same conclusions working with the bms scripts. There is one more thing I'd like to add though - file id is not unique. There are some files that have the same ID but are from diferent archives. So, for example, file with id 9 can be found in eso0000.dat and eso000x.dat and so on. I did not manage to find the correlation between archive_id, file_id and zosft so that I can determine which file from which archive is connected to which filename.

ehnoah
advanced
Posts: 70
Joined: Tue Feb 28, 2012 11:21 pm
Has thanked: 11 times
Been thanked: 4 times

Re: Researching of a .dat archive with a .mnf file.

Post by ehnoah » Fri Jan 31, 2014 1:12 am

While follow this, it looks like ESO use a strong method to save their files.

Droolie
veteran
Posts: 114
Joined: Fri Aug 19, 2005 4:31 pm
Has thanked: 14 times
Been thanked: 33 times

Re: Researching of a .dat archive with a .mnf file.

Post by Droolie » Thu Feb 06, 2014 2:20 pm

After the latest update, it seems that as LennardF1989 said, eso0000.dat is compressed differently, or the MNF records are just wrong for that file. I haven't got the time to analyze it properly, so is anyone interested?

I'm currently using this adapted script to extract the audio and the ZOSFT only (since I'm only interested in those). You can easily adapt it (by using my older scripts) to extract all the files. It only extracts files in archives with 0 < archive number < 204. You can remove those restrictions to be able to extract files from MNFs other than eso.mnf. ;)

LennardF1989
ultra-n00b
Posts: 4
Joined: Fri Dec 31, 2010 12:54 am

Re: Researching of a .dat archive with a .mnf file.

Post by LennardF1989 » Fri Feb 07, 2014 8:44 pm

I purposely waited for the next ESO weekend, because it seems it's not a mature format yet :)
Disclaimer: I reverse formats for sports only, I have nothing to gain of it but information on how they work and a feeling of accomplishment... much like finally defeating a pesky boss in Dark Souls :D

Cagli
ultra-n00b
Posts: 1
Joined: Sat Feb 15, 2014 1:38 pm
Location: France
Has thanked: 1 time

Re: Researching of a .dat archive with a .mnf file.

Post by Cagli » Sat Feb 15, 2014 2:38 pm

I've been lurking this post for a while, and I must say great work!
I'm only interested in the .dds files and have been succesfully getting plenty of those in the eso.mnf/eso00xxx.dat files.

But I don't seem to get the same result for the game.mnf one.
The topic started with the game.mnf file and shifted to the eso.mnf file.
Does anyone have a bms for the game.mnf one? I'm still pretty much a n00b when i comes to figuring it out.
I don't care about the file names, as long as the dds files are exported.
TERAB1T uploaded an exported one, but it seems to be missing a lot of files (perhaps due to not getting the right filenames)
Anyone that can help?

Ferugre
ultra-n00b
Posts: 4
Joined: Sun May 20, 2012 3:56 pm

Re: Researching of a .dat archive with a .mnf file.

Post by Ferugre » Fri Feb 28, 2014 9:18 pm

None of the QuickBMS scripts seem to work, they end up with xmalloc errors. I think incorrect values are being used to extract files. Anyone have working scripts still?

Lord Vaako
advanced
Posts: 42
Joined: Sun Jul 14, 2013 10:28 pm
Has thanked: 6 times
Been thanked: 15 times

Re: Researching of a .dat archive with a .mnf file.

Post by Lord Vaako » Tue Mar 11, 2014 9:06 pm

Extracting main game archives is the first step but what about rev eng data/files inside?
Was any of you able to find structure of any, non simple, files inside?
There are maps (gfx & names), localized text (en/fr/de) containing quests data (titles, descriptions, objectives), lore books texts, npc names, items, names, etc. and even some ids, textures, sounds, ui scripts, etc.
But I'm still missing "connections" between those data - some kind of internal database, i.e. I know quest ids, names and descriptions but I'm missing quest level and category, or connection between quest and its objectives.
Or I'm not able to find connection between map gfx and map id - I know that for most of the zones its simple (just one map and gfx), but multi level zones (i.e the wailing prison) are the problem.

Have any of you found such kind of data? :-)

stalja
beginner
Posts: 32
Joined: Sat Sep 12, 2009 11:33 am
Has thanked: 10 times
Been thanked: 5 times

Re: Researching of a .dat archive with a .mnf file.

Post by stalja » Sat Mar 15, 2014 6:20 pm

None of that has been publicly reverse engineered. No one has yet delved into decrypting eso0000.dat and eso0001.dat, which contain a lot of files not in zosft files and are compressed with an unknown method (not zlib, not snappy). Looking at some websites out there I can see that probably noone reverese engineered the client and they are all sniffing packets or using in-game addons for data collection. I guess there is not enough interest by the pros to go deep into how zenimax encrypted their files.

stalja
beginner
Posts: 32
Joined: Sat Sep 12, 2009 11:33 am
Has thanked: 10 times
Been thanked: 5 times

Re: Researching of a .dat archive with a .mnf file.

Post by stalja » Mon Mar 17, 2014 11:47 am

I was snooping around eso0000 and offzip-ed it. Bunch of files with header

Code: Select all

fa fa eb eb
or

Code: Select all

FB FB EC EC
One of these have multiple zlibed blocks, while other seems to be and index of sorts. When I offzipped the one with zlib blocks I got thousands of new files. Some of them contain some strings and I believe this has something to do with map names and their IDs (possible map tiles?) and possible quest/NPC names (possible map locations?). I also found something strange which I am not able to decipher on my own yet. There are files that apparently have some sort of encrypted or compressed strings. From my crude analysis the files are composed of a header followed by the number of strings contained, followed by 8 byte (Id?), followed by 00, followed by str_length, followed by string, followed by 00.

Attached is one of those files. Does anyone recognize what this is compressed/encrypted with?
You do not have the required permissions to view the files attached to this post.

matreco
beginner
Posts: 20
Joined: Fri Feb 17, 2012 4:56 pm
Been thanked: 1 time

Re: Researching of a .dat archive with a .mnf file.

Post by matreco » Fri Mar 21, 2014 2:12 am

Shame, would be nice to see these models out. :)

Bungie
ultra-n00b
Posts: 1
Joined: Fri Apr 18, 2014 8:09 pm

Re: Researching of a .dat archive with a .mnf file.

Post by Bungie » Sat Apr 19, 2014 7:57 pm

The Strings are encoded with somethink like Base64.

Here the first three links, decoded with Base64 and converted to hex.

Code: Select all

0413787f1451300d0a2f59
047809043108273978044b496969391d64103d0f297b3f5d7a2f3e06480e480120035f01021820082a601055121a254405493040362f3f185e7662682224743d130c6f481b6c48010201064808475d234f
045d78636018050234674470471f471e5919076448123f241a6044474038115903043b443d07112e3c641f5c6040401e66611805581f3a386b

Post Reply