XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Sat Sep 23, 2017 7:36 pm

All times are UTC + 1 hour


Forum rules


Please click here to view the forum rules



Post new topic Reply to topic  [ 46 posts ]  Go to page Previous  1, 2, 3, 4  Next
Author Message
 Post subject: Re: Mafia 3 - .SDS
PostPosted: Wed Oct 12, 2016 3:12 am 
Offline
advanced

Joined: Sat Oct 04, 2014 5:22 am
Posts: 49
Has thanked: 0 time
Have thanks: 2 times

CarLuver69 wrote:
Posting my research here in case anyone can put it to good use. I'll probably make my own tools eventually, but I don't have an ETA on that :P

Thanks to aluigi for his Mafia .SDS (PS3) script, it helped point me in the right direction.

NOTE: I'm probably wrong on a lot of this, so please feel free to make corrections. I'm most likely off on the chunkCount being used to determine how many SDSChunkData's there are.

Code:
struct SDSFile {
    int magic; // 0x534453 ('SDS\0')
   
    int version; // 0x14 (v20 / v1.4?)
   
    enum SDSType : int {
        PC = 0x4350,
       
        /* These are ASSUMED, I only have the PC version */
        PS4 = 0x345350,
        XB1 = 0x314258,
    } type;
   
    int unk_0C; // 0x5DE53FDE
   
    int dataInfoOffset;
    int dataOffset;
   
    int unk_18; // always zero?
    int unk_1C; // offset / size of something?
    int unk_20; // always zero?
    int unk_24; // offset / size of something?
    int unk_28; // always zero?
    int unk_2C; // 1
    int unk_30; // always zero?
    int unk_34; // always zero?
    int unk_38; // always zero?
    int unk_3C; // always zero?
   
    int chunkCount; // how many data chunks? (educated guess)
    int reserved; // checksum, hash, or key of some sort?
   
    /* @ dataInfoOffset */
    struct SDSDataInfo {
        int count; // how many data types there are
       
        struct SDSDataType {
            int typeIndex; // local index to reference this type
           
            int strLen;
            char typeName[strLen]; // no null-terminator
           
            int reserved; // possibly a guard? (always zero?)
        } dataTypes[count];
   
    } dataInfo;
   
    /* @ dataOffset */
    struct SDSData {
        int magic; // 0x6C7A4555 ('UEzl')
       
        int unk_04; // 0x10000? (never seems to change)
        char unk_05; // 4?
       
        struct SDSChunkData {
            int dataSize; // size of SDSChunkData
            char dataType; // reference to SDSDataInfo.dataType
           
            struct SDSChunkData {
                int memorySize; // size in memory? (e.g. uncompressed size)
               
                int bufferOffset; // offset to start of data
               
                int memorySize_2; // SDSData.unk_04 again?
               
                short unk_0A; // number of something? (this format is confusing)
                short unk_0C; // 0x80F?
               
                short bufferSize; // this might be an int, but unlikely
                /*
                    <<< Data padding >>>
                */
                /* @ bufferOffset */
                char buffer[bufferSize];
            } data;
        } chunks[chunkCount];
    };
};


hey man do you have a compiled version? it wont compile for me for some reason in visual studio

You can make the ads go away by registering



Top
 Profile  
 
 Post subject: Re: Mafia 3 - .SDS
PostPosted: Wed Oct 12, 2016 11:15 pm 
Offline
ultra-n00b

Joined: Mon Jun 23, 2014 10:06 am
Posts: 3
Has thanked: 2 times
Have thanks: 3 times
Ekey wrote:
BitEmE wrote:
Also the content seems to be encrypted, I've tried to open an encrypted DDS file -> exception.

They not encrypted, chunks are compressed with Zlib. About XML toc > not exist anymore. Content filenames hashed (fnv64a).

Code:
uint64 m3_gethash(unsigned char *m_String, int dwLength)
{
   uint64 dwHash = 0xCBF29CE484222325;
   for(int i = 0; i < dwLength; i++)
   {
      dwHash *= 0x00000100000001B3;
      dwHash ^= m_String[i];
   }
  return dwHash;
}


Some names

Code:
/gui/pop/garage.dds


For DDS you need skip additional header with 0x2F bytes after decompress. You can do it here in TextureViewer:

Code:
      private void UpdatePreview()
      {
         ...
      }


Try set to 0x2F for code (memory.Position = 0L;) and this will be enought i guess :)


Ok the thing with zlib, I get it.
But i have the following problem at the moment:

My .SDS File has 9 chunks, the alignment is 0x10000 (65335)
The first file has a size of 699239 bytes (682kb)
Image

Of course I can't read the file, because there are too less chunks (at least I'm getting exceptions thrown in my face)
9 Chunks a. 65335 bytes alignment = 588015 bytes

So I will always fail to read that file.

Am I doing something with zlib wrong?
I think the TypeID and Version are correct? Maybe the file sizes too?

I am confused right now. Would be great if someone could tell me what I'm doing wrong or at least giving me a clue :-)
Thanks!


Top
 Profile  
 
 Post subject: Re: Mafia 3 - .SDS
PostPosted: Thu Oct 13, 2016 2:34 pm 
Offline
ultra-n00b

Joined: Sun Dec 19, 2010 10:15 pm
Posts: 4
Has thanked: 4 times
Have thanks: 5 times
BitEmE wrote:
...


Zlib decompression is exactly the same as was in the M2

About files.. It looks like now the file header has bigger size than was before (36 bytes instead of 30, I have such assumption because there was extra 11 bytes between file header and DDS file magic in M2, if we follow the same logic - the header size has increased by 6 bytes)

here is my 010 Editor Templates for sds files and decompressed memory blocks https://github.com/Chipsman/mafia3-010ETemplates
may be it will help you somehow

p.s.: anybody knows where radio music is located? here Mafia III\sds_retail\audio or here Mafia III\edit\audio\packages ?

_________________
Image
RAGE research project, public side: OpenIV (Change log Roadmap) | openFormats


Top
 Profile  
 
 Post subject: Re: Mafia 3 - .SDS
PostPosted: Thu Oct 13, 2016 5:03 pm 
Offline
advanced

Joined: Thu Jul 12, 2012 12:24 pm
Posts: 61
Has thanked: 6 times
Have thanks: 15 times
\edit\audio\packages\radio_global.pck


Top
 Profile  
 
 Post subject: Re: Mafia 3 - .SDS
PostPosted: Thu Oct 13, 2016 11:36 pm 
Offline
ultra-n00b

Joined: Mon Jun 23, 2014 10:06 am
Posts: 3
Has thanked: 2 times
Have thanks: 3 times
Alright, thanks to your help guys, really apprechiate it!

I'm now able to extract/show DDS files. Unfortunately DDs file in Mafia 3 can have the compression types ATI and DXT10, I'm not quite sure how to fix those decompressions with the texture viewer, but that's just a minor problem. (Also extracting is kind of a problem as there are +4 bytes infront of the magic DDs identifier of the file, so external DDs viewer will fail to load the image)
Image

Also I've modified the XML reading and get now 'readable' XML content. Again, unfortunately, the format has changed in Mafia 3. The Problem right now is that I'm not able to say that the next node in memory is an attribute of a node, is a value, a child, or another node.
For example, with this file it's working because the values are only numeric (I'm using a hack at the moment to detect values inside nodes -> TryDouble)
Image

With this file, it's not working because there are node attributes (see xsi:type is a node instead of attribute)
Image

The structure is weird because it looks like this:

node type = 4
6 bytes = 0
String (length is not written in memory, you have to check for null terminator)
String Terminator
node type = 4
6 bytes = 0
String (length is not written in memory, you have to check for null terminator)
... and so on and so on..

So basically there's always a 4 between all nodes (maybe seperator? idk)
I don't know how I should figure out what's a child node, what's an attribute etc..

Again, here's the source (Just based of Rick's Mafia 2 tool) https://github.com/MartinJK/Mafia3SDSExplorer (in case someone of you wants to try it out etc.)

Thanks for you help guys, really apprechiate it. I'm used to reverse engineer and mod games, this is the first time I'm trying to 'reverse' the files format and create a tool for it :-)


Top
 Profile  
 
 Post subject: Re: Mafia 3 - .SDS
PostPosted: Fri Oct 14, 2016 1:33 am 
Offline
M-M-M-Monster veteran
M-M-M-Monster veteran

Joined: Wed Mar 31, 2010 6:54 am
Posts: 1593
Has thanked: 59 times
Have thanks: 748 times
Some new names :)

Code:
/gui/pop/missions.dds
/gui/pop/womaniser_voodoo_dancer.dds
/gui/pop/alternative_shop.dds
/gui/pop/classy_shop.dds
/gui/pop/cloth_shop.dds
/gui/pop/fashion_shop.dds
/gui/pop/gun_shop.dds
/gui/pop/item_shop.dds
/gui/pop/shades_shop.dds
/gui/bc_map_pcp_lab_a.gfx
/gui/config/maps/bc_map_pcp_lab_a
/gui/hideout_district_assignment.swf
/gui/images/icons/button_a.dds
/gui/images/icons/button_b.dds
/gui/images/icons/button_x.dds
/gui/images/icons/other_icons0031.dds
/gui/images/icons/other_icons0037.dds
/gui/images/icons/other_icons0099.dds
/gui/images/icons/other_icons0101.dds
/gui/images/icons/other_icons0103.dds
/gui/images/icons/other_icons0105.dds
/gui/images/icons/other_icons0106.dds
/gui/interact_collectable.swf
/gui/interact_object.swf
/gui/interact_weapon_pickup.swf
/gui/in_world_menu.swf
/gui/pop/garage.dds
/gui/pop/home.dds
/gui/scripted_test.swf

/xbin/cities.xlaybin
/xbin/plotlines.xlaybin
/xbin/missions.xlaybin
/xbin/quests.xlaybin
/xbin/game_events.xlaybin
/xbin/persistent_characters.xlaybin
/xbin/businesses.xlaybin
/xbin/speeches.xlaybin
/xbin/shops.xlaybin
/xbin/cities_game_group.xbin
/xbin/plotlines_group.xbin
/xbin/missions_group.xbin
/xbin/quests_group.xbin
/xbin/game_events_group.xbin
/xbin/persistent_characters_group.xbin
/xbin/businesses_group.xbin
/xbin/streammap_gameinit_group.xbin
/xbin/speeches_game_group.xbin
/xbin/generic_speech_mp.xlaybin
/xbin/generic_speech_mp_group.xbin
/xbin/generic_speech_situations_group.xbin
/xbin/generic_speech_variations_group.xbin
/xbin/shops_group.xbin
/xbin/game_main.xlaybin
/xbin/game_main_group.xbin
/xbin/expressiondata.xlaybin
/xbin/game_expressions_group.xbin
/xbin/charactertable.xlaybin
/xbin/charactertable_cinematics_group.xbin

/sds/maps/mapa_bourbon2.sds
/sds/maps/mapa_testscene.sds

/root/Situations
/root/Variations
/root/CharacterTableMP
/root/StreamMapTableMP
/root/GameEventsTableMP
/root/QuestsTableMP
/root/MissionsTable
/root/SpeechesTableMP
/root/PlotlinesTableMP
/root/BusinessesTableMP
/root/PersistentCharactersTableMP

/fsm/persistent_characters/alvarez_active
/fsm/persistent_characters/alvarez_inactive
/fsm/persistent_characters/biaggio_active
/fsm/persistent_characters/biaggio_inactive
/fsm/persistent_characters/cesar_active
/fsm/persistent_characters/cesar_inactive
/fsm/persistent_characters/clayton_active
/fsm/persistent_characters/clayton_inactive
/fsm/persistent_characters/creed_active
/fsm/persistent_characters/creed_inactive
/fsm/persistent_characters/danny_quest_active
/fsm/persistent_characters/danny_quest_inactive
/fsm/persistent_characters/doctor_active
/fsm/persistent_characters/doctor_inactive
/fsm/persistent_characters/donovan_active
/fsm/persistent_characters/donovan_inactive
/fsm/persistent_characters/mack_quest_active
/fsm/persistent_characters/mack_quest_inactive
/fsm/persistent_characters/marcano_active
/fsm/persistent_characters/marcano_inactive
/fsm/persistent_characters/maria_active
/fsm/persistent_characters/maria_inactive
/fsm/persistent_characters/martinez_active
/fsm/persistent_characters/martinez_inactive
/fsm/persistent_characters/professor_active
/fsm/persistent_characters/professor_inactive
/fsm/persistent_characters/sixto_active
/fsm/persistent_characters/sixto_inactive

/scripts/gamelogic/bc_openworld
/scripts/gamelogic/fpm/fpm_plotline
/scripts/gamelogic/milestones/milestone27/drug_farm_main
/scripts/gamelogic/milestones/milestone27/gameplay_bricks/shooting_from_boat
/scripts/gamelogic/milestones/milestone27/gameplay_bricks/shooting_from_car
/scripts/gamelogic/milestones/milestone28/steal_rum
/scripts/gamelogic/plotline_act2
/scripts/gamelogic/testscenes/bourbon_test_mission/bourbon_test_mission
/scripts/gamelogic/testscenes/districts/west_marais_district_structure_quest
/scripts/gamelogic/testscenes/districts/wharf_district_structure_quest
/scripts/gamelogic/testscenes/game_structure_init
/scripts/gamelogic/testscenes/game_structure_quest
/scripts/gamelogic/testscenes/open_world/main
/scripts/gamelogic/testscenes/racing_focustest/main
/scripts/gamelogic/testscenes/test_bartek/simple
/scripts/gamelogic/testscenes/test_racing_pod/main


Top
 Profile  
 
 Post subject: Re: Mafia 3 - .SDS
PostPosted: Fri Oct 14, 2016 2:28 am 
Offline
advanced
User avatar

Joined: Fri Aug 22, 2014 6:11 pm
Posts: 66
Location: Earth
Has thanked: 1 time
Have thanks: 0 time
BitEmE wrote:
Alright, thanks to your help guys, really apprechiate it!

Thanks for you help guys, really apprechiate it. I'm used to reverse engineer and mod games, this is the first time I'm trying to 'reverse' the files format and create a tool for it :-)


We owe you bro, thanks. Can you share compiled version of this script? And i want a question, this tool does it work to subtitles.sds files, can you extract and repack clearly?


Top
 Profile  
 
 Post subject: Re: Mafia 3 - .SDS
PostPosted: Fri Oct 14, 2016 8:52 pm 
Offline
M-M-M-Monster veteran
M-M-M-Monster veteran

Joined: Wed Mar 31, 2010 6:54 am
Posts: 1593
Has thanked: 59 times
Have thanks: 748 times
Info for PCKG archives

Code:
struct PCKGHeader
{
   uint32_t   dwID; // Always PCKG (0x50434B47)
   uint32_t   dwVersion; // 1
   uint32_t   dwTotalFiles;
   uint32_t   dwFlag; // 0xFFFFFFFF
};

struct PCKGEntry
{
   uint64_t   dwHash; // Hash of filename?
   uint32_t   dwOffset;
   uint32_t   dwSize;
   uint32_t   dwZSize;
};


Top
 Profile  
 
 Post subject: Re: Mafia 3 - .SDS
PostPosted: Sat Oct 15, 2016 5:41 pm 
Offline
n00b

Joined: Fri Jun 05, 2015 7:35 pm
Posts: 18
Has thanked: 2 times
Have thanks: 0 time
BitEmE wrote:
Alright, thanks to your help guys, really apprechiate it!

I'm now able to extract/show DDS files. Unfortunately DDs file in Mafia 3 can have the compression types ATI and DXT10, I'm not quite sure how to fix those decompressions with the texture viewer, but that's just a minor problem. (Also extracting is kind of a problem as there are +4 bytes infront of the magic DDs identifier of the file, so external DDs viewer will fail to load the image)
Image

Also I've modified the XML reading and get now 'readable' XML content. Again, unfortunately, the format has changed in Mafia 3. The Problem right now is that I'm not able to say that the next node in memory is an attribute of a node, is a value, a child, or another node.
For example, with this file it's working because the values are only numeric (I'm using a hack at the moment to detect values inside nodes -> TryDouble)
Image

With this file, it's not working because there are node attributes (see xsi:type is a node instead of attribute)
Image

The structure is weird because it looks like this:

node type = 4
6 bytes = 0
String (length is not written in memory, you have to check for null terminator)
String Terminator
node type = 4
6 bytes = 0
String (length is not written in memory, you have to check for null terminator)
... and so on and so on..

So basically there's always a 4 between all nodes (maybe seperator? idk)
I don't know how I should figure out what's a child node, what's an attribute etc..

Again, here's the source (Just based of Rick's Mafia 2 tool) https://github.com/MartinJK/Mafia3SDSExplorer (in case someone of you wants to try it out etc.)

Thanks for you help guys, really apprechiate it. I'm used to reverse engineer and mod games, this is the first time I'm trying to 'reverse' the files format and create a tool for it :-)

Can you please post here builded version of your tool? Thx ;)


Top
 Profile  
 
 Post subject: Re: Mafia 3 - .SDS
PostPosted: Sun Oct 16, 2016 9:17 am 
Offline
ultra-n00b

Joined: Sun Dec 19, 2010 10:15 pm
Posts: 4
Has thanked: 4 times
Have thanks: 5 times
Audio packages format info (AKPK)

Code:
struct Header
{
    char    dwMagic[4];   // AKPK (0x4B504B41h)
    DWORD   dwHeaderSize; // +0x4
    DWORD   _f8;
    DWORD   _fC;
    DWORD   _f10;
    DWORD   _f14;
    DWORD   _f18;
    DWORD   _f1C;
    DWORD   _f20;
    DWORD   _f24;
    DWORD   _f28;
    DWORD   _f2C;
    DWORD   _f30;
    DWORD   dwFileCount;    //+0x34
};

struct FileEntry
{
    DWORD   dwHash;     // +0x0
    DWORD   _f4;        // always 16,  block size ? or aligment ?
    DWORD   dwSize;     // +0x8
    DWORD   dwOffsetInBlocks; //+0xC // block size == 16
    DWORD   _f10;       // always 0, // or dwOffsetInBlocks is QWORD
};


Audio is stored in .wav container but the format/codec inside is unknown

here is 010 template https://github.com/Chipsman/mafia3-010ETemplates/blob/master/m3-pck.bt

_________________
Image
RAGE research project, public side: OpenIV (Change log Roadmap) | openFormats


Top
 Profile  
 
 Post subject: Re: Mafia 3 - .SDS
PostPosted: Sat Oct 22, 2016 10:44 pm 
Offline
advanced

Joined: Sat Oct 04, 2014 5:22 am
Posts: 49
Has thanked: 0 time
Have thanks: 2 times
any progress on decrypting the dat files?


Top
 Profile  
 
 Post subject: Re: Mafia 3 - .SDS
PostPosted: Thu Oct 27, 2016 5:55 pm 
Offline
advanced
User avatar

Joined: Fri Aug 22, 2014 6:11 pm
Posts: 66
Location: Earth
Has thanked: 1 time
Have thanks: 0 time
Skrillex wrote:
BitEmE wrote:
Alright, thanks to your help guys, really apprechiate it!

Thanks for you help guys, really apprechiate it. I'm used to reverse engineer and mod games, this is the first time I'm trying to 'reverse' the files format and create a tool for it :-)


We owe you bro, thanks. Can you share compiled version of this script? And i want a question, this tool does it work to subtitles.sds files, can you extract and repack clearly?


Top
 Profile  
 
 Post subject: Re: Mafia 3 - .SDS
PostPosted: Mon Oct 31, 2016 9:28 pm 
Offline
veteran

Joined: Mon Oct 13, 2014 6:54 am
Posts: 106
Has thanked: 52 times
Have thanks: 6 times
I just wanted to add my support for anyone thinking of putting tools together for this game! Love it! Great character models, and vehicles, too! Would love to be able to get in and take a proper look at some of them...!


Top
 Profile  
 
 Post subject: Re: Mafia 3 - .SDS
PostPosted: Sat Nov 19, 2016 2:19 pm 
Offline
double-veteran
double-veteran

Joined: Tue Mar 24, 2015 8:12 pm
Posts: 1178
Has thanked: 19 times
Have thanks: 548 times
iambosh wrote:
any progress on decrypting the dat files?


they are not encrypted


Top
 Profile  
 
 Post subject: Re: Mafia 3 - .SDS
PostPosted: Sun Nov 20, 2016 5:49 am 
Offline
advanced

Joined: Sat Oct 04, 2014 5:22 am
Posts: 49
Has thanked: 0 time
Have thanks: 2 times
daemon1 wrote:
iambosh wrote:
any progress on decrypting the dat files?


they are not encrypted


yeah when decrypted with the tool the results from the sds are dat files with random names and they are encrypted. so no actual assets



Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 46 posts ]  Go to page Previous  1, 2, 3, 4  Next

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 8 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group