XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Thu Oct 19, 2017 6:17 am

All times are UTC + 1 hour




Post new topic Reply to topic  [ 236 posts ]  Go to page 1, 2, 3, 4, 5 ... 16  Next
Author Message
 Post subject: Beyond: Two Souls - Reversing | Porting (THAT'S ALL FOLKS?)
PostPosted: Sat Apr 12, 2014 1:30 pm 
Offline
veteran
User avatar

Joined: Thu Apr 03, 2014 9:49 am
Posts: 129
Location: The Netherlands
Has thanked: 62 times
Have thanks: 166 times

____Big Update List

____01/12/2014 We all knew it had to happen some day!


Model-to-do-list Disclaimer: list subject to change ;)
Todo
  • Make a to-do list :lol:

Doing
  • Research SCRIPT_ZONE's (ZONE_SHAPE_BOX, ZONE_SHAPE_SPHERE, ZONE_SHAPE_HEXAHEDRON)
  • Research areas.

Done
I forgot to update this list #lol.
Except for the skeleton & materials everything is done: meshes are parsed, textures decoded.


    Sequence-to-do-list Disclaimer: list subject to change ;)
    Todo
    • Implement MOVIE_VIEWPORT_FADE_FX_ACTION
    • Implement MOVIE_PAD_VIBRATION_FX_ACTION

    Doing
    • Implement PAR_ELEMENT_STANDARD
    • Implement PAR_ELEMENT_HOLD
    • Implement PAR_ELEMENT_LOOP
    • Implement PAR_ELEMENT_MASH
    • Implement MPAR_ELEMENT_AXIS_TRANSLATION
    • Implement MPAR_ELEMENT_AXIS_ROTATION
    • Implement MPAR_ELEMENT_ABC (?)
    • Implement MPAR_ELEMENT_STANDARD
    • Implement MPAR_ELEMENT_HOLD
    • Implement MPAR_ELEMENT_LOOP
    • Implement MPAR_ELEMENT_MASH
    • Implement MOVIE_USER_ACTION_CONDITION

    Done
    • Create a "flexible" interface for playing / controlling sequences.
    • Implement SEQUENCE_CONTROLLER (SEQ_CHK_), which only defines basic (but important) stuff like the root movie.
    • Implement MOVIE_SCRIPT_CONDITION
    • Implement SEQUENCE_CONTROLLER_PLAY[
    • Rewrite audio streamer for the third time :cry: (this time asynchronous)
    • Make it better.
    • Perform first tests!!
    • Write a custom high-resolution Timer class (using a thread and a stopwatch!)
    • Implement MOVIE_PLAY_ANIM_ACTION
    • Implement MOVIE_GO_TO_MOVIE_CONDITION
    • Implement PLAY_SCRIPT_TOP
    • Implement MOVIE_PLAY_SHOT_ACTION
    • Implement MOVIE_WATCH_FX_ACTION
    • Implement MOVIE_CAMERA_MODIFIER_FX_ACTION
    • Implement MOVIE_CAMERA_QUAKE_FX_ACTION
    • Implement MOVIE_PLAY_SOUND_ACTION
    • Implement MOVIE_DIALOG_GROUP_CONDITION
    • Implement MOVIE_PLAY_DIALOG_ACTION
    • Implement MPA_CHK_ and the other entity Communicators (camera shots, sounds, dialog, etc.)
    • Research camera "sub-shots".
    • Implement MOV_ELEM (also relies on other stuff)
    • Implement MOV_MISC
    • Implement MOV_GUID
    • Implement MOV_ENT_ (relies on other stuff though)
    • Find out what PAR/MPAR stands for (Physical Action Reaction / Motion Physical Action Reaction)


    Kernel-to-do-list Disclaimer: list subject to change ;)
    Todo
    • Implement version 2 of CAC_____ (as seen in 0802 The Party Clash -> main scene com).
    • Check out different DATA_CONTAINER => DC_DATA version formats and implement
    • Implement SCRIPT_ZONEs
    • Start my own blog(???)
    • Check out non-empty SC_FRM_W (in 1802 Hunted Train)
    • Audio features (stop, pause, resume, seek, insert silence)

    Doing
    • Perform tests to make sure everything works!
    • Set up MOGRE (now Paradox) environment
    • Spam XeNTaX with pointless update videos

    Done
    • Research PAR/MPAR events and choice events.
    • Implement Lua "actions" system (partly completed)
    • Re-implement dynamic loading for SCRIPT_FUNCTION_CALL.
    • Proper meta-data reading (needs: SINT_SEC reader rewrite, modification to DC_DATA_ reader and preloader FIX)
    • FIX: preloading of resources
    • Partly rewrite, partly port the Sequence parser/player.
    • Port usable code over from older tools (mainly InfraViewer)
    • Completely revise attribute reading and data loading.
    • Finish audio streaming engine (needs minor performance adjustments)
    • Change FMOD audio engine to BASS audio engine (much better support for custom data streaming and easier DSP mixing)
    • Research loading of resources.
    • MILESTONE: Execute -FULL- scene, including sub-scripts, dynamic loading (100% with Broken (scene after Prologue) and Prologue)
    • Implement SendEvent(...)
    • Make some sort of Lua execution manager to make sure there are no cross-thread executions. (seems to be fixed using <lock>)
    • [HIGH PRIORITY] Dynamic loading for LOADING_ZONEs & LOADING_ZONE_TIMERs. (seems to work fine, needs more testing though!)
    • Implement SCRIPT_FUNCTION and SCRIPT_FUNCTION_CALL (major)
    • Reorder classes into separate .cs files...
    • MILESTONE: perform test using scene name, -> load scene loadzone, load scene datacontainer, execute scene (and load resources) -> call <scene>.OnInit().
    • Research COM_INSTANCE_LIST's
    • Implement EndVarFreedom()
    • Converter for PS3 (PPC, big endian) LUAC chunks to PC (little endian) chunks.
    • Temporary video play management, with threaded pre-loading and threaded playing. Note: needs actual playing! XD
    • Audio system (FMOD) with global functions, events, and threading. (3D support will be written later)
    • Implement SCRIPT_DELAY
    • Implement MESSAGE_LAUNCHER & MESSAGE_CONTROLLER (TESTED)
    • Proper resource management and properly working clean up functions (stress tested!)
    • Make multithreaded loading manager with managed stacks and event callbacks
    • "BinaryCore" for BigFiles (IDM file reading, handle assignments)
    • Implement runtime.sdat reading
    • Implement PARAM.SFO reader
    • Implement Dynamic Communicator system (register, attributes, events, paramaters, actions (partly), unregister, linking)
    • Implement Lua "variables" system.
    • Implement Lua "events" system.
    • Implement Lua "attributes" system.
    • Neatly working kernel enviroment with easy-to-plug-in features and support.
    • Create basic "instances" environment.


    Support

    As you may know, this category of XeNTaX doesn't get as much attention as we'd like it to get!
    That's why I made this badge:

    Image
    Code:
    [url=http://forum.xentax.com/viewtopic.php?f=32&t=11411][img]http://i.imgur.com/EUcneAM.png[/img][/url]


    So now, you can show your support of this project by putting the badge in your signature!

    Donations

    Some people have asked me if they can donate somewhere, so I decided to put a PayPal donation thing up here:

    Image

    To anyone who donates: THANK YOU! You're the best.
    After you donate, make sure to send the receipt to hugo ~a t~ djek ~d o t~ nl with your nickname, so you can be in the credits once any tools / ports are released.

    Where will the money go to? I haven't really thought of any situations where this project would cost money, but it will be primarily spent on this project. This means we could for instance set up a website, or pay people to reverse some assembly for us :twisted: It could also be spent on hardware / software (such as the PS4 version of Beyond, when it comes out).

    WARNING!!! The info under here is inaccurate and very outdated!

    Image

    Here's my second tool since joining here!

    This tool can extract all data blocks from Beyond: Two Souls, including audio (SFX, Dialog, Music & Ambience (multi channel)), segs entries (and decompress them using QuickBMS), scripts and dialog text (very limited support right now).

    It also partly works with Heavy Rain. Let me know if you want me to investigate those formats too.

    Here's a screenshot:
    Image


    What's coming?
    Okay, first a little explanation about how Beyond works.
    Beyond (and Heavy Rain) store its data in blocks called Communicators.

    (The following text is correct for around 60% :keke: )
    A Communicator is a table containing data for a certain function in the game. These are formatted as precompiled Lua "function chunks". The values of these "settings" are stored either compressed (as a segs entry, which is a split up ZLIB archive), or uncompressed (as a QZIP entry, which is funny, because "ZIP" normally indicates a compressed file entry). Aside from storing just values as Lua functions, modified versions of Communicators are also used for storing the "actual" scene scripts, model data, animation data, (very small amounts of) audio, video (like the sport scenes and cartoons on the in-game TVs), textures, dialog sequence text, and more.

    Right now, I'm working on a universal parser for these blocks. That's basically what the extractor does, but the difference is that this new parser can get useful data out of the blocks, rather than just extracting them.

    >> Download Beyond: Two Souls -Extractor- v1.0
    ANOTHER NOTE: Even though this is v1.0 it's still very much a beta. Please let me know if you get errors during export!!

    So, there you go! I wonder if I will be getting any comments on this one!

    You can make the ads go away by registering


    _________________
    Image

    "I'm going to throw my shit at you" —The Great Mighty Poo
    _______________________>> check my incomplete portfolio I wish the [center] bbcode was turned on
    _______________________>> or my twitter
    _______________________>> or my game dev FB page


    Last edited by HugoPeters on Thu Jan 15, 2015 11:52 pm, edited 96 times in total.

    Top
     Profile  
     
     Post subject: Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
    PostPosted: Sat Apr 12, 2014 6:02 pm 
    Offline
    ultra-veteran
    ultra-veteran
    User avatar

    Joined: Fri Apr 27, 2012 9:37 am
    Posts: 430
    Has thanked: 29 times
    Have thanks: 178 times
    I bow down to you sir good fucken work, I gave up when all those seg blocks were chopped up textures and such. If we can get a decent unpacker for each game asset would be easier to reverse.

    -EDIT-

    Ok so I tested it on the main BIGFILE
    Image

    I get this not sure why.
    Also BigFile_PS3.d08 throws this
    Image

    Have you looked at parsing the model/animation/texture blocks yet?

    _________________
    Devblog


    Top
     Profile  
     
     Post subject: Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
    PostPosted: Tue Apr 15, 2014 4:44 pm 
    Offline
    veteran
    User avatar

    Joined: Thu Apr 03, 2014 9:49 am
    Posts: 129
    Location: The Netherlands
    Has thanked: 62 times
    Have thanks: 166 times
    cra0 wrote:
    I bow down to you sir good fucken work, I gave up when all those seg blocks were chopped up textures and such. If we can get a decent unpacker for each game asset would be easier to reverse.

    -EDIT-

    Ok so I tested it on the main BIGFILE
    Image

    I get this not sure why.
    Also BigFile_PS3.d08 throws this
    Image

    Have you looked at parsing the model/animation/texture blocks yet?

    Thanks!

    Damn, that were some things I forgot to fix... sorry for that! I'll post an update later. Slammed with school stuff right now... :(

    From your list, I have only been looking at the ANIMDATA blocks. I always first try to reverse engineer the file entries / tab header, etc. but I can't figure it out completely... will post some info below.

    About the segs archives: I haven't really looked at them too much, but from what I can see is that we're still missing something. After each(?) segs archive there are more chunks that start with 0x10... these define their own sizes as well... not sure what's up with that, maybe they belong to the segs archives, because every individual file is aligned to 0x800 (2048 bytes), and these only come after the segs stuff, unaligned.

    I've been looking through the decrypted version of EBOOT.BIN (EBOOT.ELF), and found some strings that might be able to give us some clues about these archives..
    PSST, don't tell anyone: I got the source code for EdgeZlib (it's a special kind of Zlib that can be offloaded to the ?SPU? that comes with the PS3 Edge SDK), will be checking that out soon...

    0x00EF86F0: EDGE ZLIB ERROR: outputUncompSkipBeginSize(%d) + outputUncompSkipEndSize(%d) > expectedUncompSize(%d)
    0x00EF8760: EDGE ZLIB ERROR: edgeZlibFetchAndInflateLargeRawData failed (%d)
    0x00EF87B0: EDGE ZLIB ERROR: edgeZlibInflateRawData failed (%d)
    0x00EF87F0: EDGE ZLIB ERROR: DecompressInflateQueueElement returned (%d)
    0x00EF8830: EDGE ASSERTION FAILURE: %s(%d)
    0x00EF8850: 1.2.3.0-PS3-SPU-EDGE
    0x00EF8870: EDGE ZLIB ERROR: inflate error (%d)
    0x00EF88A0: incorrect header check
    0x00EF88C0: unknown compression method
    0x00EF88E0: invalid window size
    0x00EF8900: unknown header flags set
    0x00EF8920: header crc mismatch
    0x00EF8940: invalid block type
    0x00EF8960: invalid stored block lengths
    0x00EF8980: too many length or distance symbols
    0x00EF89B0: invalid code lengths set
    0x00EF89D0: invalid bit length repeat
    0x00EF89F0: invalid literal/lengths set
    0x00EF8A10: invalid distances set
    0x00EF8A30: invalid literal/length code
    0x00EF8A50: invalid distance code
    0x00EF8A70: invalid distance too far back
    0x00EF8A90: incorrect data check
    0x00EF8AB0: incorrect length check
    0x00EF9380: EDGE ZLIB ERROR: inflateEnd error (%d)
    0x00EF93B0: EDGE ZLIB ERROR: Stream decompressed to size different from expected (%d != %d)
    0x00EF9520: invalid distance code
    0x00EF9540: invalid literal/length code
    0x00EF9560: invalid distance too far back
    0x00EF972C: SPUNAME
    0x00EF9734: ..\..\Output\PS3\SPU Release\Ker

    0x00D80384: Out of memory while calling edgeZlibAddInflateQueueElement. The game will crash soon !

    0x00D84448: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:70 (0 == ret)
    0x00D84488: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:96 CELL_OK == ret
    0x00D844CC: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:188 (maxNumQueueEntries > 0)
    0x00D8451C: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:148 (bufferSize == edgeZlibGetInflateQueueSize( maxNumQueueEntries ))
    0x00D84594: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:149 (bufferSize == (128 + (maxNumQueueEntries * 32)))
    0x00D845FC: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:150 ((((uint32_t)pBuffer) & 0x7F) == 0)
    0x00D8465C: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:164 (0 == ret)
    0x00D8469C: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:167 (0 == ret)
    0x00D846DC: EDGE ZLIB ERROR: Input compressed data is on stack (which is illegal)

    0x00D84724: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:401 (!sys_process_is_stack((void*)pInputCompressedData))
    0x00D84794: EDGE ZLIB ERROR: Output uncompressed data is on stack (which is illegal)

    0x00D847E4: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:404 (!sys_process_is_stack(pOutputUncompPartialBuff))
    0x00D8484C: EDGE ZLIB ERROR: outputUncompSkipBeginSize(%d) + outputUncompSkipEndSize(%d) > expectedUncompressedSize(%d)

    0x00D848BC: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:408 (outputUncompSkipBeginSize + outputUncompSkipEndSize <= expectedUncompressedSize)
    0x00D84944: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:410 (pWorkToDoCounter)
    0x00D8498C: Warning: Compressed data at address 0x%08X is *bigger* than master data (%d > %d). Pointless?

    0x00D849EC: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:423 ((((uint32_t)pWorkToDoCounter) & 0x3) == 0)
    0x00D84A54: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:442 (0 == ret)
    0x00D84A9C: Buff))
    0x00D84AA4: dSize)
    0x00D84AB4: Sources\Zlib\EdgeZLib\edgezlib_inflate_queue.cpp:583 (0 == ret)


    Furthermore, here are the different types of Dynamic Communicator blocks I could find, most of them with a description of some sort.
    I put them in a nice switch statement (yes, I did this manually...):

    NOTE: an asterisk (*) means the description is a guess by me, and where there's just the description it means I haven't found the short name for that one yet.

    Code:
    switch (BlockType)
    {
        case "COM_CONT":  // COMMUNICATOR_CONTAINER
        case "LOADCONT": // *LOADING_CONTAINER
        case "LOADZONE":  // LOADING_ZONE
        case "LOADZTIM": // LOADZING_ZONE_TIMER
        case "SINT_SEC":
        case "SFE_LOAD":
        case "ANIMDATA": // *ANIMATION_DATA
        case "LOCALIZ_":  // *LOCALIZATION
        case "PARTITIO":  // COMMUNICATOR_PARTITION
       
        case "SFESAV  ": // SCRIPT_FRAMEWORK_ENTITY_CONTROLLER_ABC
        case "SCN_CHK ":
        case "SINT_SCN":
        case "VPCCTRL ": // VIDEO_PLAYER_CONTROLLER
        case "MCR_CHK ":
       
        case "SCRIPT  ":
        case "SCPT_DAT":
           
        /* Unknown */
        case "SC_FRM_W": // SCRIPT_FRAMEWORK
        case "SENT_CHK":
        case "MK_INFO_":
        case "LOADZCAM": // LOADING_ZONE_CAMERA
        case "SINT_WLD":
        case "SCSAV   ":
        case "SFESAV  ":
        case "VLACRFC ": // VIEWPORT_LAYOUT_ANIMATION_CONTROLLER_RUN_FROM_CURRENT
        case "VLACR   ": // VIEWPORT_LAYOUT_ANIMATION_CONTROLLER_RUN
        case "VCFOCTRL": // VIEWPORT_CONTROLLER_FADE_OPACITY
        case "VCFCTRL ": // VIEWPORT_CONTROLLER_FADE
        case "CP_ATTR ": // SCRIPT_FRAMEWORK_ENTITY_CONTROLLER_ABC
        case "SCN_CHK ":
        case "SCENE   ":
        case "ADDR_CHK":
        case "SINT_ADR":
        case "VPCCTRL ": // VIDEO_PLAYER_CONTROLLER
        case "MMCSAV  ":
        case "MENUSCPM": // MENUS_MASTER_COM_PARAMETERS
        case "LMCRCTRL": // LOAD_MENU_CONTROLLER_RUN
        case "FADECTRL": // FADE_CONTROLLER
        case "MCRCTRL ": // MENU_CONTROLLER_RUN
        case "SEQ_CHK ":
        case "GEN_SEQ ":
        case "SECSAVE ": // SCRIPT_EVENT_CONTROLLER
        case "SEQUENCE":
        case "SCDLPARA": // SCRIPT_DIALOG_PARAMETERS
        case "SDPARAMS":
        case "SD_PARMS":
        case "GAP_CHK ": // GENERIC_ANIM_PARAMETERS
        case "UA_CHK  ": // USER_ACTION
        case "PAR_CONT":
        case "UACABC  ": // UA_CONTROLLER_ABC
        case "UAECABC ": // UA_EVENT_CONTROLLER_ABC
        case "EVENTCOM":
        case "EVNTMPAR":
        case "UAECPAR ": // UA_EVENT_CONTROLLER_PAR
        case "MPARELAD": // MPAR_ELEMENT_AXIS_DIAGONAL
        case "PARELEM ": // MPAR_ELEMENT_NO_EVENT
        case "PARELSHK": // PAR_ELEMENT_BLOCK/HIT
        case "SINT_TPG":
        case "MPCTRL  ": // MAGNETIC_SEGMENT_CONTROLLER
        case "TPGCTRL ": // THIRD_PERSON_GAMEPLAY_CONTROLLER
        case "SINT_GC ":
        case "GCTRLER ": // GAME_CONTROLLER
        case "SINT_GM ":
        case "SINTGM  ":
        case "SINT_GRI": // GAME_RUNTIME_INFO
        case "MAGCON  ": // MAGNETIC_CONNECTION
        case "TPGNP_CK":
        case "MPASAV  ": // MAGNETIC_PATH
        case "MPCSAV  ": // MAGNETIC_PATH_CONTAINER
        case "SDCTRL  ": // SCRIPT_DELAY
        case "STIM_CHK":
        case "GMK_COMG": // GMK_MOVEMENT_GRAPH
        case "GMK_ANIM": // GMK_ANIMATION_DATABASE
        case "RBSACTIO":
        case "RBSACTOR":
        case "RBSBRAIN":
        case "MPAREABC": // MPAR_ELEMENT_ABC
        case "PARELABC": // PAR_ELEMENT_ABC
        case "MPARELAR": // MPAR_ELEMENT_AXIS_ROTATION
        case "MPARELAT": // MPAR_ELEMENT_AXIS_TRANSLATION
        case "PARELLOP": // MPAR_ELEMENT_LOOP
        case "PARELMSH": // MPAR_ELEMENT_MASH/PAR_ELEMENT_MASH
        case "PARELHLD": // PAR_ELEMENT_HOLD
        case "PARELLP ": // PAR_ELEMENT_LOOP
        case "PARELSTD": // PAR_ELEMENT_STANDARD
        case "CBSPGP  ":
        case "CBFLWR  ": // ?CAMERA_BEHAVIOR_FOLLOW_VEHICLE
        case "CBFOLWVH": // ?CAMERA_BEHAVIOR_SEQUENCE
        case "ADCI_CHK":
        case "ADBSAV  ":
        case "GSCCSAV ": // GAME_SAVE_CONTROLLER
        case "GSD_CHK ": // ?GAME_SAVE_DATA
        case "GSDSAV  ":
        case "SRG_CHK ":
        case "AG_SAV  ":
        case "AORSAV  ":
        case "AROSAV  ":
        case "AGNC_CHK": // ANAEL_GAMEPLAY_NAVIGATION_CONFIG
        case "ABCSAV  ":
        case "AGNC_CHK": // ANAEL_GAMEPLAY_SELECTOR_CONFIG
        case "ANCSAV  ":
        case "AGNC_CHK":
        case "ASCSAV  ":
        case "AEMCSAV ": // ANAEL_EASY_MODE_CONTROLLER
        case "AFCSAV  ": // ANAEL_FX_CONTROLLER
        case "PIPO    ": // ?not sure =
        case "AFCCSAV ": // ANAEL_FX_CONTROLLER_CONFIG
        case "AGCSAV  ": // ANAEL_GO_TO_CONTROLLER
        case "AICSAV  ": // ANAEL_INTERACTION_CONTROLLER
                         // ANAEL_ENERGY_MANAGER
                         // ANAEL_ENERGY_SOURCE_CONFIG
                         // COM_ATTRIBUTE_CONTAINER_ABC
        case "AESSAV  ":
        case "AEM_CHK ":
        case "AEMSAV  ":
        case "AFC     ":
        case "ATCSAV  ":
        case "ATFSAV  ":
        case "CARGCSAV": // CAR_GAMEPLAY_CONTROLLER
        case "CARGPIN ": // CTRL_TRAIL_SPEED
        case "IEPSAV  ": // IW_ENTITY_TAIL_SHADER_CONTROLLER
        case "IECSAV  ": // INFRAWORLD_ENTITY_CONFIG
        case "IEP_CHK ": // INFRAWORLD_ENTITY_PACK
        case "IEPSAV  ": // INFRAWORLD_ENTITY_PACK_CONTROLLER
        case "OCSAV   ": // ?PURSUIT_BEHAVIOR
        case "PSSCTCFG": // POSSESSION_CENTRE_CONFIG
        case "PCCSAV  ": // POSSESSION_CENTRE_CONTROLLER
        case "VTSSAV  ": // VIEW_TARGET_SELECTION_CONTROLLER
       
        case "GFCCFG  ": // GROUP_FOLLOW_CONTROLLER_CONFIG
        case "ARRVLPOS": // ARRIVAL_POSE
        case "APSAV   ":
        case "SGCSAV  ": // ?STABILIZE_GAMEPLAY_CONTROLLER
        case "STCTLSV ": // STEALTH_CONTROLLER
       
        case "FSSFCTRL": // FOOTSTEP_SURFACE_CONTROLLER
       
        case "ICCSAV  ": // IK_CONSTRAINT_CONTROLLER
                         // IK_CONSTRAINT_CONTROLLER_EFFECTOR
        case "IQCSAV  ": // IK_QUADRIPED_CONTROLLER
        case "WICSAV  ": // WATCH_ENTITY_CONTROLLER
        case "SUBMGPIN": // SUBMARINE_GAMEPLAY_INPUT
        case "BIKE_CHK": // BIKE_GAMEPLAY
        case "BGCSAV  ": // BIKE_GAMEPLAY_CONTROLLER
        case "PRCANMDT": // PROC_ANIM_COM_DATA
                         // PROJECTILE_MANAGER
        case "VICSAV  ": // VIBRATE_CONTROLLER
        case "LCSAV   ": // LOADING_CONTROLLER
        case "LCFSAV  ": // LOADING_CONTROLLER_FUNCTION
        case "MPST_CHK": // PLAY_SCRIPT_TOP
        case "TSFX_CHK": // MOVIE_TIME_SCALE_FX_ACTION
        case "ACT_CHK ":
        case "COND_CHK":
        case "FX_CHK  ":
        case "CCTSAV  ": // COLLISION_CONTROLLER
        case "TCTSAV  ": // THROW_CONTROLLER
        case "PSESAV  ": // PHYSICS_SPECIAL_EFFECTS
        case "REPSAVE ": // REPULSOR
                         // PHYSICS_LOD_CONFIGURATOR
        case "PHYLC   ":
        case "PLCSAV  ": // CPN_FLMG_CORRECTION
                         // CPN_OBSTACLE_CORRECTION
                         
        case "CDEC_SAV": // CHARACTER_DIALOG_EVENT_CONTROLLER
        case "CEGASAV ": // CONTROLLER_ENTITY_GOTO_ABC
                         // CONTROLLER_ENTITY_GOTO_ADDRESS
        case "CEGAASAV": // CONTROLLER_ENTITY_GOTO_ADDRESS_ABC
        case "CEOTTSV ": // CONTROLLER_ENTITY_ORIENT_TO_TARGET
        case "SECPRANM": // SCRIPT_ENTITY_CONTROLLER_PLAY_PROCEDURAL_ANIM
        case "SSMSAV  ": // SCRIPT_ENTITY_CONTROLLER_SKELETON_MAPPING
        case "SKEL_MAP":
        case "SMCSAV  ": // SKELETON_MAPPING_CONFIG
        case "STZOCTRL": // STAIRS_ZONE_CONTROLLER
        case "SEPSAV  ": // SYNCHRONIZE_ENTITY_POSITIONS_CONTROLLER
        case "PHAD_CHK":
        case "SLSAV   ": // SCRIPT_LOCATOR
        case "FBCSAV  ": // FX_BANK_CONTROLLER
        case "TS_FX   ": // TIME_SCALE_FX
        case "LCALTCFG": // LOCATION_ALERT_CONFIG
        case "LACSAV  ":
                         // LOCATION_ALERT_CONTROLLER
                         // RAISE_EVENT_COUNT_CONTROLLER
                         // RAISE_EVENT_PER_ITEM_CONTROLLER
        case "SWSAV   ": // SCRIPT_WIND
                         // SEQUENCE_INSTANCE
        case "MGCDE   ": // MIX_GROUP_CONTROLLER_DISABLE_EFFECT
        case "MGCEE   ": // MIX_GROUP_CONTROLLER_ENABLE_EFFECT
        case "MGCSFRE ": // MIX_GROUP_CONTROLLER_SET_FDN_REVERB_EFFECT
                         // MIX_GROUP_CONTROLLER_MIX_GROUP
        case "MGCSMCE ": // MIX_GROUP_CONTROLLER_SET_MULTICHANNEL_COMPRESSOR_EFFECT
        case "MGCSPSE ": // MIX_GROUP_CONTROLLER_SET_PITCH_SHIFTER_EFFECT
        case "SSDFFCTL": // SCRIPT_SOUND_CONTROLLER_DISTANCE_FILTER
                         // FOOTSTEP_SOUND_MANAGER
        case "FSMSAV  ": // COLLISION_SOUND_MANAGER
        case "COSOMG  ":
        case "COSOMGSV": // SCRIPT_SOUND_BANK
        case "SBCSAV  ": // SCRIPT_SOUND_BANK_CONTROLLER
                         // SCRIPT_SOUND_BANK_INSTANCE
                         // SCRIPT_SOUND_MULTITRACK_CONTAINER
        case "S_SND_MC": // SCRIPT_SOUND_RANDOM_CONTAINER
                         // SCRIPT_SOUND_RANDOM_CONTAINER_MODE
        case "S_SND_RC":
        case "S_SND_SC":
        case "AI_AGENT":
        case "AIMSAV  ": // AI_PATH_INTERACTOR_WITH_ZONE
        case "AIPIWZON":
                         // CONTROLLER_AGENT_DELAY_PATH
                         // CONTROLLER_AGENT_FOLLOW
                         // CONTROLLER_AGENT_GOTO_ABC
                         // CONTROLLER_AGENT_GOTO_ADDRESS
                         // CONTROLLER_AGENT_GOTO_ADDRESS_ANIM
                         // CONTROLLER_AGENT_GOTO_AGENT
                         // CONTROLLER_AGENT_GOTO_ZONE
        case "SDCTRL  ": // CONTROLLER_SYNC
        case "GCMCTRL ": // GMK_COMMAND_MESSAGE_CONTROLLER
        case "MKSP_CHK": // ?GMK_SIMULATE_PARAMS
                         // MOTION_GRAPH_TESTER_CONTROLLER
                         // MOTION_GRAPH_PLANNING_CONTROLLER
                         // MOTION_GRAPH_CONTROLLER
                         // QUATERNION_INTERPOLATION_TESTER_CONTROLLER
        case "HDAMCT  ": // HEAD_ANIMO_CONTAINER
        case "WANIMO  ": // WEIGHTED_ANIMO
        case "AREA_CHK":
        case "SINT_ARA":
        case "SINT_CD ":
        case "CSB_CHK ":
        case "SINT_CSB":
        case "CBABC   ": // ?CAMERA_BEHAVIOR_ANAEL
        case "CBSPGP  ": // ?CAMERA_BEHAVIOR_CHASE
        case "CBDCIRC ": // CAMERA_BEHAVIOR_CONTAINER
        case "CBCONTAI": // CAMERA_BEHAVIOR_CROWD_CHASE
                         // CAMERA_BEHAVIOR_FIRST_PERSON
        case "PANIMOCT": // PLAYABLE_ANIMO_CONTROLLER
        case "TGUI_CK ":
       
        case "CATALOG ": // AREA_CATALOG
        case "QFOBJECT": // QPF_COM_CLASS
       
        case "SCPTBANK": // SCRIPT_BANK
        case "GCFG_CHK":
        case "GAME_CFG":
        case "CFG_ELEM": // GAME_CFG_ELEMENT_ABC
        case "RUNT_CFG": // GAME_CFG_RUNTIME_ELEMENT_ABC
        case "STAT_CFG": // GAME_CFG_STATIC_ELEMENT_ABC
       
        case "DLG_CFG ": // DIALOG_CFG
        case "GMG_CFG ": // GAME_MANAGER_CFG
        case "ST_MPAR_":
        case "MPAR_CFG":
        case "RT_ID_  ":
        case "ID_CFG  ": // INPUT_DEVICE_CFG
        case "RT_LOC_ ":
        case "LOC_CFG ": // LOCALIZATION_CFG
        case "RT_TPG_ ":
        case "TPG_CFG ": // THIRD_PERSON_GAMEPLAY_CFG
       
        case "GENPAT  ": // GENERIC_PATH
        case "GENCON  ": // GENERIC_PATH_CONNECTION
        case "GENSEG  ": // GENERIC_PATH_SEGMENT
       
        case "STORYBRD": // *Storyboard
        case "SINTSTRY":
        case "STBRDELM": // STORYBOARD_ELEMENT
        case "STRBDSCN": // STORYBOARD_SCENE
        case "STBSCELM": // STORYBOARD_SCENE_ELEMENT
       
        case "SF_SAV  ": // SCRIPT_FUNCTION
        case "SFCSAV  ": // SCRIPT_FUNCTION_CALL
       
        case "CMODFXCK": // MOVIE_CAMERA_MODIFIER_FX_ACTION
        case "CQFX_CHK": // MOVIE_CAMERA_QUAKE_FX_ACTION
        case "PADVIB  ": // MOVIE_PAD_VIBRATION_FX_ACTION
        case "PLAVIDFX": // MOVIE_PLAY_VIDEO_FX_ACTION
        case "RVLA_CHK": // MOVIE_RUN_VIEWPORT_LAYOUT_ANIM_ACTION
        case "VPFX_CHK": // MOVIE_VIEWPORT_FADE_FX_ACTION
        case "MDGC_CHK": // MOVIE_DIALOG_GROUP_CONDITION
        case "SCPTCOND": // MOVIE_SCRIPT_CONDITION
        case "MSIC    ": // MOVIE_SOFT_INTERRUPT_CONDITION
        case "MUAC_CHK": // MOVIE_USER_ACTION_CONDITION
       
        case "MOV_ENT_":
        case "MOV_EVT_":
        case "MOV_TOP_":
        case "MOV_FX__":
        case "MOV_GUID":
        case "MOV_MISC":
        case "MOV_COND":
        case "MOV_ELEM":
       
        case "PARDISPL": // UA_ELEMENT_DISPLAY
        case "PARINPUT": // UA_ELEMENT_INPUT
       
        case "MPA_CHK_":
        case "MPABC___": // MOVIE_PLAYABLE_ABC
        case "MCAA_CHK": // MOVIE_COM_ATTRIB_ACTION
        case "MPAA_CHK": // MOVIE_PLAY_ANIM_ACTION
        case "MPDA_CHK": // MOVIE_PLAY_DIALOG_ACTION
        case "NSHA_CHK": // MOVIE_PLAY_SHOT_ACTION
        case "MSA_CHK_": // MOVIE_PLAY_SOUND_ACTION
       
        /* 3D objects */
        case "OBJ_____":
        case "MOV_ENT_":
        case "MESHDATA":
        case "PRIMEDGE":
        case "CLUPSKME":
        case "ENTITY  ":
        case "MESH____":
        case "NODE    ":
        case "STPHDATA": // STATIC_PHYSIC_DATA
        case "DYPHDATA": // DYNAMIC_PHYSIC_DATA
        case "SHADCUST":
        case "FX_EVNT_": // FX_EVENTS
       
        /* Audio */
        case "PARTOFFS": // OFFSET_PARTITION
        case "STREAMAB": // STREAMABLE
        case "HEADER__":
        case "S_SND_E_":
        case "TBOFFSET":
    }



    About the ANIMDATA, I haven't researched it very deeply, but I did find out some things:

    0x8: Block ID (ANIMDATA)
    0x4: Unknown identifier (see below)
    0x4: Unknown identifier (see below)
    0x4: Unknown identifier (see below)
    0x4: Unknown identifier (see below)
    0x4: Big "tab" entries amount (after the header tab)
    0x4: Unknown identifier... (see below)
    0x4: Unknown identifier... (see below)
    0x4: Header "tab" entries amount

    Then the header tab starts, every entry consists of 6 ushorts(?):
    0x2: Unknown ushort
    0x2: Unknown ushort
    0x2: Unknown ushort
    0x2: Unknown ushort
    0x2: Unknown ushort
    0x2: Unknown ushort

    Then there's 2 (padding?) bytes, and then the "big" tab thing starts, every entry consisting of 4 ushorts(?):
    0x2: Unknown ushort
    0x2: Unknown ushort
    0x2: Unknown ushort
    0x2: Unknown ushort

    Then there's some more data... might be the same size as the header entries amount (??????)

    So yeah, haven't looked at it very long :P

    I may have a lead on this one as well though, I found this in the extracted strings from EBOOT.ELF:

    0x00D7B0B0: ANIMDATA
    0x00D7B0C0: ANIMPARM
    0x00D7B0D0: TRACKDAT
    0x00D7B0DC: ANIM_DATA
    0x00D7B0E8: KEYFRAME_ANIMATION
    0x00D7B108: Evt_Foot_Left
    0x00D7B118: Evt_Foot_Right
    0x00D7B128: Evt_SYNCHstep_Left
    0x00D7B140: Evt_SYNCHstep_Right
    0x00D7B15C: MBI_SKA
    0x00D7B164: STREAMABLE_KEYFRAME_ANIMATION
    0x00D7B184: STREAM_ANIMATION_OFFSET
    0x00D7B19C: AnimData
    0x00D7B1AC: Ref_AnimData
    0x00D7B1BC: Ref Frame
    0x00D7B1CC: Start Frame
    0x00D7B1DC: End Frame
    0x00D7B1EC: Blend Time
    0x00D7B1FC: Animation Type
    0x00D7B20C: Streamable AnimData
    0x00D7B224: Ref Streamable AnimData
    0x00D7B23C: SetAnimData
    0x00D7B24C: GetAnimData
    0x00D7B25C: SetStartFrame
    0x00D7B26C: GetStartFrame
    0x00D7B27C: SetEndFrame
    0x00D7B28C: GetEndFrame
    0x00D7B29C: SetRefAnimData
    0x00D7B2AC: GetRefAnimData
    0x00D7B2BC: SetReferenceFrame
    0x00D7B2D4: GetRefFrame
    0x00D7B2E4: SetStreamableAnimData
    0x00D7B2FC: GetStreamableAnimData
    0x00D7B314: SetRefStreamableAnimData
    0x00D7B334: GetRefStreamableAnimData
    0x00D7B354: Main
    0x00D7B35C: Idle Frame
    0x00D7B36C: Idle Duration Min
    0x00D7B384: Idle Duration Offset
    0x00D7B39C: Animation Types
    0x00D7B3AC: One Shot
    0x00D7B3BC: Loop
    0x00D7B3C4: Attachment Offset Translation
    0x00D7B3E4: Attachment Offset Rotation X
    0x00D7B404: Attachment Offset Rotation Y
    0x00D7B424: Attachment Offset Rotation Z
    0x00D7B444: Attachment Animation
    0x00D7B45C: Pause Attachment Anim
    0x00D7B474: ANIMMOD
    0x00D7B47C: ANIMATION_MODIFIER
    0x00D7B490: LeftUpLeg
    0x00D7B4A0: LeftLeg
    0x00D7B4A8: LeftFoot
    0x00D7B4B8: RightUpLeg
    0x00D7B4C8: RightLeg
    0x00D7B4D8: RightFoot
    0x00D7B4E4: LeftToeBase
    0x00D7B4F4: RightToeBase
    0x00D7B504: LeftArm
    0x00D7B50C: LeftForeArm
    0x00D7B51C: LeftHand
    0x00D7B52C: RightArm
    0x00D7B53C: RightForeArm
    0x00D7B54C: RightHand
    0x00D7B55C: ANIMATION_OPTIONS
    0x00D7B570: DisplayMotionKitCommands
    0x00D7B590: DisplayInvisiblePlanes
    0x00D7B5A8: DisplayMagneticPathConstraints
    0x00D7B5C8: DisplayAnimationPoses
    0x00D7B5E0: DisplayBoneLock
    0x00D7B5F0: DisplayPoseValidityCheck
    0x00D7B610: DisplayCanDebug
    0x00D7B620: DisplayHierarchyFadeIn
    0x00D7B638: EnableRemoteDebugging

    And this:

    0x00DAC61C: ThisCommunicator
    0x00DAC630: None
    0x00DAC638: InBounds
    0x00DAC648: OnEvent
    0x00DAC650: AliasCRC
    0x00DAC660: nFrameLowerBound
    0x00DAC678: nFrameUpperBound
    0x00DAC690: sAnimDataName
    0x00DAC6A0: EventNameCRC
    0x00DAC6B0: TriggerOnce
    0x00DAC6C0: nAnimDataID
    0x00DAC6CC: ThisCommunicator
    0x00DAC6E0: Invoke
    0x00DAC6E8: Success
    0x00DAC6F0: AliasCRC
    0x00DAC700: EventNameCRC
    0x00DAC714: Invoke
    0x00DAC71C: Success
    0x00DAC724: EventNameCRC
    0x00DAC734: EntityCatalogID
    0x00DAC744: EntityCatalogName
    0x00DAC758: ThisCommunicator
    0x00DAC774: Offset_Translation
    0x00DAC78C: Offset_Rotation

    I don't know if that helps at all, but it could give some hints to what the data means...

    By the way, I haven't found any texture blocks yet.... do they start without a Communicator ID? Because I did see a few of those that looked like they could be textures...

    _________________
    Image

    "I'm going to throw my shit at you" —The Great Mighty Poo
    _______________________>> check my incomplete portfolio I wish the [center] bbcode was turned on
    _______________________>> or my twitter
    _______________________>> or my game dev FB page


    Top
     Profile  
     
     Post subject: Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
    PostPosted: Wed Apr 16, 2014 10:28 pm 
    Offline
    veteran
    User avatar

    Joined: Thu Apr 03, 2014 9:49 am
    Posts: 129
    Location: The Netherlands
    Has thanked: 62 times
    Have thanks: 166 times
    Major Update / Find / Thing

    Alright, I decided to take a look at (the decrypted version of) BigFile_PS3.sdat first, which starts with the id QUANTICDREAMTABIDMEM.
    I suspected this file would give indices into the other BigFile's, and I was right!

    The file starts off with a header, which I haven't figured out completely yet, but at least it gives the amount of entries, and the block offset for each block (+ 0x18 for the header).

    Then the very first block of entries in the sdat file are indices into BigFile_PS3.dat and are for the LOCALIZ_ blocks (PS3 error codes / Sony messages), the second block is also for BigFile_PS3.dat and this one only has one entry, for the (huge) uncompiled Lua initalization script.

    Then after that there are huge blocks of indices. First up is a tab index I haven't figured out yet :]
    After that come the indices into the files, each 0x14 in size. They give block offsets, block sizes and the file ID's.

    Here's a sample output of 1 of the blocks of BigFile_PS3.sdat (only the first 100 entries of the total of 10,069...):

    Code:
    META
    {
        File: G:\Projects\Tools\BeyondTwoSouls\Indexing\BigFile_PS3.(s)dat
        Header ID: QUANTICDREAMTABIDMEM
        Version: 3
    }

    HEADER TAB
    {
        HeaderTabEntries: 7
        UnknownPadding: 0x0
         ----------
        Data Type Entry #1: Unknown: 0x3F8, TypeEntries: 0x2, BlockOffset: 0x60
        Data Type Entry #2: Unknown: 0x1078, TypeEntries: 0x1, BlockOffset: 0x488
        Data Type Entry #3: Unknown: 0x869, TypeEntries: 0x1, BlockOffset: 0x89C
        Data Type Entry #4: Unknown: 0x1D, TypeEntries: 0x2755, BlockOffset: 0xCB0
        Data Type Entry #5: Unknown: 0x897, TypeEntries: 0x15, BlockOffset: 0x32354
        Data Type Entry #6: Unknown: 0x3F2, TypeEntries: 0x3398, BlockOffset: 0x328F8
        Data Type Entry #7: Unknown: 0xFFB, TypeEntries: 0x1FC8, BlockOffset: 0x734D8
    }

    /* TEMP TEMP TEMP: STARTING ANALYSE @ 0x10C4 */

    TEMP FILE OUTPUT
    {
        Entry #1: BlockID: 256, BlockOffset: 0xADD4000, BlockSize: 0x77431F, u1: 0x6ACBC7, FileID: 0 -> BigFile_PS3.dat
        Entry #2: BlockID: 1024, BlockOffset: 0x46A46000, BlockSize: 0x68, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
        Entry #3: BlockID: 1792, BlockOffset: 0x12DAE800, BlockSize: 0x12B7D8, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #4: BlockID: 2048, BlockOffset: 0x4C783800, BlockSize: 0x29, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #5: BlockID: 2560, BlockOffset: 0x240EA800, BlockSize: 0x782A, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #6: BlockID: 2816, BlockOffset: 0xBDB3000, BlockSize: 0x4F5B, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #7: BlockID: 3584, BlockOffset: 0x5B259800, BlockSize: 0x3B118, u1: 0x3A6CB, FileID: 0 -> BigFile_PS3.dat
        Entry #8: BlockID: 4608, BlockOffset: 0x61F3000, BlockSize: 0x6953B, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #9: BlockID: 6400, BlockOffset: 0x7DE5A000, BlockSize: 0x2ADB6A, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #10: BlockID: 8192, BlockOffset: 0x4DD74000, BlockSize: 0x120C7, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #11: BlockID: 8960, BlockOffset: 0x5985D000, BlockSize: 0x9D14, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #12: BlockID: 9984, BlockOffset: 0x2A398000, BlockSize: 0xE59E7, u1: 0xE45C4, FileID: 2 -> BigFile_PS3.d02
        Entry #13: BlockID: 10496, BlockOffset: 0x39932800, BlockSize: 0xBCC1, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
        Entry #14: BlockID: 12032, BlockOffset: 0x551D7000, BlockSize: 0x123AFE, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
        Entry #15: BlockID: 12800, BlockOffset: 0x7DA49800, BlockSize: 0x3CFFD56, u1: 0x3C9F87B, FileID: 3 -> BigFile_PS3.d03
        Entry #16: BlockID: 13568, BlockOffset: 0x12BDB000, BlockSize: 0x1F90A2, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #17: BlockID: 13824, BlockOffset: 0xBADE000, BlockSize: 0x29, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #18: BlockID: 14592, BlockOffset: 0x4FBEF800, BlockSize: 0x3B3, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #19: BlockID: 14848, BlockOffset: 0x6F185800, BlockSize: 0x1EA4D, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #20: BlockID: 16128, BlockOffset: 0x64C32800, BlockSize: 0x2B42, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #21: BlockID: 16640, BlockOffset: 0x39570800, BlockSize: 0xB8C8A, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #22: BlockID: 18176, BlockOffset: 0x1AEE6800, BlockSize: 0xB9D5B, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #23: BlockID: 19200, BlockOffset: 0x3094D800, BlockSize: 0x29, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #24: BlockID: 19968, BlockOffset: 0x30D5C800, BlockSize: 0x3AE, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #25: BlockID: 20992, BlockOffset: 0x5A712000, BlockSize: 0x29, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
        Entry #26: BlockID: 22016, BlockOffset: 0x379F5800, BlockSize: 0x29, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
        Entry #27: BlockID: 22272, BlockOffset: 0x5D2F8800, BlockSize: 0x9C9B, u1: 0x3207, FileID: 1 -> BigFile_PS3.d01
        Entry #28: BlockID: 24064, BlockOffset: 0x2A7D800, BlockSize: 0x247E9B, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
        Entry #29: BlockID: 24576, BlockOffset: 0x396D0000, BlockSize: 0x1A19F4D, u1: 0x19F050E, FileID: 4 -> BigFile_PS3.d04
        Entry #30: BlockID: 27904, BlockOffset: 0x68C5E000, BlockSize: 0x7A6BF, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #31: BlockID: 28672, BlockOffset: 0x5D495000, BlockSize: 0xB98BE, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #32: BlockID: 29696, BlockOffset: 0x6AF8A800, BlockSize: 0x4F5D2, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #33: BlockID: 30208, BlockOffset: 0x52ED3800, BlockSize: 0x22B1D, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #34: BlockID: 30976, BlockOffset: 0x31093000, BlockSize: 0x29, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #35: BlockID: 31232, BlockOffset: 0x3A18000, BlockSize: 0x2073C, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
        Entry #36: BlockID: 32512, BlockOffset: 0x4E9B800, BlockSize: 0x79CE38, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #37: BlockID: 32768, BlockOffset: 0x63FB8000, BlockSize: 0x81448, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #38: BlockID: 33024, BlockOffset: 0x4825000, BlockSize: 0x28BF5, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
        Entry #39: BlockID: 35072, BlockOffset: 0x255CD800, BlockSize: 0x29, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
        Entry #40: BlockID: 35840, BlockOffset: 0x3D411800, BlockSize: 0x11C954, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
        Entry #41: BlockID: 36608, BlockOffset: 0x33F23800, BlockSize: 0x5D884C, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #42: BlockID: 38912, BlockOffset: 0x372E2800, BlockSize: 0x2ABA9D, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #43: BlockID: 39680, BlockOffset: 0x1F782000, BlockSize: 0xE417, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #44: BlockID: 40448, BlockOffset: 0x73AAD000, BlockSize: 0x29, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #45: BlockID: 40704, BlockOffset: 0xBE20800, BlockSize: 0x7D26, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
        Entry #46: BlockID: 41216, BlockOffset: 0x564DE800, BlockSize: 0xED8A, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #47: BlockID: 43520, BlockOffset: 0x2BB60800, BlockSize: 0x2DFC, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
        Entry #48: BlockID: 43776, BlockOffset: 0x6B89000, BlockSize: 0xADF, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
        Entry #49: BlockID: 513, BlockOffset: 0x396000, BlockSize: 0x464, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #50: BlockID: 1025, BlockOffset: 0x46A44800, BlockSize: 0x68, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
        Entry #51: BlockID: 1281, BlockOffset: 0x832A000, BlockSize: 0x6957A1, u1: 0x686825, FileID: 2 -> BigFile_PS3.d02
        Entry #52: BlockID: 1537, BlockOffset: 0x235C6800, BlockSize: 0x3F0C9, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #53: BlockID: 1793, BlockOffset: 0x12EDA000, BlockSize: 0xC79A, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #54: BlockID: 2817, BlockOffset: 0xBDB8000, BlockSize: 0x18201B, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #55: BlockID: 3585, BlockOffset: 0x5B295000, BlockSize: 0xFF4C7, u1: 0xA3EFB, FileID: 0 -> BigFile_PS3.dat
        Entry #56: BlockID: 3841, BlockOffset: 0xE388800, BlockSize: 0x70F77, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #57: BlockID: 4097, BlockOffset: 0x4DB8B800, BlockSize: 0x41AE2, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #58: BlockID: 5121, BlockOffset: 0x4FAA1800, BlockSize: 0x6B137E, u1: 0x6A0839, FileID: 1 -> BigFile_PS3.d01
        Entry #59: BlockID: 7681, BlockOffset: 0x63847800, BlockSize: 0x3067, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #60: BlockID: 8193, BlockOffset: 0x4DD86800, BlockSize: 0x29, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #61: BlockID: 8961, BlockOffset: 0x59867000, BlockSize: 0x6D704, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #62: BlockID: 10497, BlockOffset: 0x3993E800, BlockSize: 0x58BE, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
        Entry #63: BlockID: 11009, BlockOffset: 0x1141E000, BlockSize: 0xBE6, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #64: BlockID: 12801, BlockOffset: 0x734AB800, BlockSize: 0x5A3062, u1: 0x58888A, FileID: 0 -> BigFile_PS3.dat
        Entry #65: BlockID: 13825, BlockOffset: 0xBADE800, BlockSize: 0x42B2A, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #66: BlockID: 14337, BlockOffset: 0x764FE800, BlockSize: 0x37FAA8, u1: 0x313500, FileID: 0 -> BigFile_PS3.dat
        Entry #67: BlockID: 16385, BlockOffset: 0x65162800, BlockSize: 0x12447A, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #68: BlockID: 19201, BlockOffset: 0x3094E000, BlockSize: 0x79A66, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #69: BlockID: 19969, BlockOffset: 0x30D5D000, BlockSize: 0x46CB0, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #70: BlockID: 20481, BlockOffset: 0x3CD5D000, BlockSize: 0x76FD0, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #71: BlockID: 20993, BlockOffset: 0x7251E000, BlockSize: 0x1BD8BD, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #72: BlockID: 22017, BlockOffset: 0x379F6000, BlockSize: 0x29, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
        Entry #73: BlockID: 24065, BlockOffset: 0x2CC5800, BlockSize: 0x2602F, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
        Entry #74: BlockID: 30209, BlockOffset: 0x53F21800, BlockSize: 0xF4D, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #75: BlockID: 30977, BlockOffset: 0x31093800, BlockSize: 0x29, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #76: BlockID: 31745, BlockOffset: 0x5EE4F800, BlockSize: 0x60066, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #77: BlockID: 32257, BlockOffset: 0x46CA800, BlockSize: 0x3B1, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
        Entry #78: BlockID: 32769, BlockOffset: 0x64039800, BlockSize: 0x2F2B9, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #79: BlockID: 35073, BlockOffset: 0x255CE000, BlockSize: 0x2D872, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
        Entry #80: BlockID: 35841, BlockOffset: 0x3D52E800, BlockSize: 0xED651, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
        Entry #81: BlockID: 37633, BlockOffset: 0xD529800, BlockSize: 0x1964, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
        Entry #82: BlockID: 37889, BlockOffset: 0x116C0000, BlockSize: 0x71E3, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #83: BlockID: 39681, BlockOffset: 0x1F790800, BlockSize: 0x41D28, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #84: BlockID: 40193, BlockOffset: 0x22F54000, BlockSize: 0x1A616, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #85: BlockID: 40449, BlockOffset: 0x73AAD800, BlockSize: 0x29, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #86: BlockID: 41217, BlockOffset: 0x564ED800, BlockSize: 0x1DC63, u1: 0x0, FileID: 0 -> BigFile_PS3.dat
        Entry #87: BlockID: 41729, BlockOffset: 0x61035800, BlockSize: 0x14044, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #88: BlockID: 43777, BlockOffset: 0x69D0800, BlockSize: 0x5BE12, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
        Entry #89: BlockID: 258, BlockOffset: 0x8FA2000, BlockSize: 0x7CBC5D, u1: 0x72BFB2, FileID: 0 -> BigFile_PS3.dat
        Entry #90: BlockID: 770, BlockOffset: 0x5305C000, BlockSize: 0x241422B, u1: 0x237FBD7, FileID: 1 -> BigFile_PS3.d01
        Entry #91: BlockID: 1026, BlockOffset: 0x46A45000, BlockSize: 0x464, u1: 0x0, FileID: 4 -> BigFile_PS3.d04
        Entry #92: BlockID: 1538, BlockOffset: 0x23606000, BlockSize: 0x29, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #93: BlockID: 2818, BlockOffset: 0xBF3A800, BlockSize: 0x7EA1, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #94: BlockID: 3586, BlockOffset: 0x5B394800, BlockSize: 0xAE8F9, u1: 0x79511, FileID: 0 -> BigFile_PS3.dat
        Entry #95: BlockID: 8450, BlockOffset: 0x39708800, BlockSize: 0x3CEB, u1: 0x0, FileID: 1 -> BigFile_PS3.d01
        Entry #96: BlockID: 8962, BlockOffset: 0x598D4800, BlockSize: 0x20EBB, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #97: BlockID: 10242, BlockOffset: 0x265C2800, BlockSize: 0xF3203, u1: 0xBE680, FileID: 0 -> BigFile_PS3.dat
        Entry #98: BlockID: 11010, BlockOffset: 0x1141F000, BlockSize: 0x138212, u1: 0x0, FileID: 2 -> BigFile_PS3.d02
        Entry #99: BlockID: 12034, BlockOffset: 0x552FB000, BlockSize: 0xA5FB2, u1: 0x0, FileID: 3 -> BigFile_PS3.d03
        Entry #100: BlockID: 12802, BlockOffset: 0x7F43C000, BlockSize: 0xE2CC2, u1: 0xB8880, FileID: 0 -> BigFile_PS3.dat
    }


    There are a few things to note here:
    1. The thing that I called "BlockID" (which I'm not sure of if that's what it actually is) gets bigger every entry but then suddenly gets smaller again. I'm suspecting these are "groups" of assets that belong to each other.
    2. The u1 stands for Unknown1. No idea what this indicates yet, but it's only non-0x0 for some entries...

    So, that's it for now!

    _________________
    Image

    "I'm going to throw my shit at you" —The Great Mighty Poo
    _______________________>> check my incomplete portfolio I wish the [center] bbcode was turned on
    _______________________>> or my twitter
    _______________________>> or my game dev FB page


    Top
     Profile  
     
     Post subject: Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
    PostPosted: Thu Apr 17, 2014 10:19 pm 
    Offline
    veteran
    User avatar

    Joined: Thu Apr 03, 2014 9:49 am
    Posts: 129
    Location: The Netherlands
    Has thanked: 62 times
    Have thanks: 166 times
    [bruce]
    The "groups" inside BigFile_PS3.(s)dat seem to be neatly ordered by content! There's a group that indexes just .bik files (video) of the different BigFile's, then there's a group that seems to be indexing only the audio files, and there are groups that index all over the place (BigFile_PS3.dat - .d04). Then, within the big content groups, there seem to be 255 subgroups (always that number for some reason), that's the tab thing I didn't understand in my previous post. These subgroups are defined before the actual file indices start of that content group. I'm not sure if these groups actually mean something or not right now. Today I made a pretty good parser for the index sdat with basic GUI / treeview of the individual entries of every BigFile, together with sorting per content group / subgroup. Will port some of the better code of the extractor over tomorrow so I can check if these groups are actual groups.

    _________________
    Image

    "I'm going to throw my shit at you" —The Great Mighty Poo
    _______________________>> check my incomplete portfolio I wish the [center] bbcode was turned on
    _______________________>> or my twitter
    _______________________>> or my game dev FB page


    Top
     Profile  
     
     Post subject: Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
    PostPosted: Fri Apr 18, 2014 12:11 pm 
    Offline
    veteran

    Joined: Fri Jun 01, 2012 10:08 am
    Posts: 122
    Has thanked: 31 times
    Have thanks: 64 times
    I have some files taken from the ps3s cache whilst the game is running, I can send them to you if you think they may help.


    Top
     Profile  
     
     Post subject: Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
    PostPosted: Fri Apr 18, 2014 3:13 pm 
    Offline
    ultra-veteran
    ultra-veteran
    User avatar

    Joined: Fri Apr 27, 2012 9:37 am
    Posts: 430
    Has thanked: 29 times
    Have thanks: 178 times
    HugoPeters wrote:
    PSST, don't tell anyone: I got the source code for EdgeZlib

    hehehe XD nice! hope that comes in handy


    HugoPeters wrote:
    Furthermore, here are the different types of Dynamic Communicator blocks I could find, most of them with a description of some sort.

    Ah nice yeah those look about right

    case "MESHDATA":
    case "OBJ_____":

    seem interesting most likely OBJ would be the static objects in the scenes.


    HugoPeters wrote:
    By the way, I haven't found any texture blocks yet.... do they start without a Communicator ID? Because I did see a few of those that looked like they could be textures...

    The textures are in DXT1 DXT5 (DXTATI2N - NORMALS), if I recall the segs script EKEY wrote dumped some texture files.


    JayK wrote:
    I have some files taken from the ps3s cache whilst the game is running, I can send them to you if you think they may help.

    Someone's mentioned this to me before would you mind if I took a look at the ram dump as well to at least see how there model formats are stored and hopefully
    figure them out for when Hugo's finished his parser tool?

    Oh lastly great job HugoPeters Image

    _________________
    Devblog


    Top
     Profile  
     
     Post subject: Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
    PostPosted: Fri Apr 18, 2014 4:29 pm 
    Offline
    veteran
    User avatar

    Joined: Thu Apr 03, 2014 9:49 am
    Posts: 129
    Location: The Netherlands
    Has thanked: 62 times
    Have thanks: 166 times
    JayK wrote:
    I have some files taken from the ps3s cache whilst the game is running, I can send them to you if you think they may help.

    Yeah, that would be very useful! That could help with identifying the different groups of files the game defines, or what to expect from the decompressed file entries. Btw, cache dump = RAM dump?

    cra0 wrote:
    HugoPeters wrote:
    HugoPeters wrote:
    Furthermore, here are the different types of Dynamic Communicator blocks I could find, most of them with a description of some sort.

    Ah nice yeah those look about right

    case "MESHDATA":
    case "OBJ_____":

    seem interesting most likely OBJ would be the static objects in the scenes.


    From what I can tell the objects are very complex. Here's some strings related to the model structure:

    0x00DCA3FC: hkVersionPatchManager
    0x00DCA414: hk_2011.1.0-r1
    0x00DCA424: hk.DataObjectType
    0x00DCA438: The object of class
    0x00DCA451: is about to be removed and all references to it will be invalid.
    However, the object is referenced (
    0x00DCA4B8: ) from outside the world and it will lead to unexpected behavior or crash.
    0x00DCA508: d:/Perforce/Main/Havok2011/Source/Common/Serialize/Data/Dict/hkDataObjectDict.cpp
    0x00DCA55C: typeName
    0x00DCA610: hk.PostFinish
    0x00DCA654: __classnames__
    0x00DCA664: variant
    0x00DCA66C: memSizeAndFlags
    0x00DCA67C: value
    0x00DCA684: name
    0x00DCA68C: time
    0x00DCA694: namedVariants
    0x00DCA6A4: className
    0x00DCA6B4: hkRootLevelContainer
    0x00DCA6CC: hkRootLevelContainerNamedVariant
    0x00DCA6F4: hkReferencedObject
    0x00DCA70C: hkBaseObject
    0x00DCA71C: type
    0x00DCA724: data
    0x00DCA72C: hkxMaterial
    0x00DCA73C: floatData
    0x00DCA74C: floatStride
    0x00DCA75C: uint8Stride
    0x00DCA76C: hkxIndexBuffer
    0x00DCA77C: userChannels
    0x00DCA78C: mapping
    0x00DCA794: hkxMaterialTextureStage
    0x00DCA7AC: uint8Data
    0x00DCA7BC: decls
    0x00DCA7C4: hkxMeshSection
    0x00DCA7D4: attributes
    0x00DCA7E4: hkxVertexDescriptionElementDecl
    0x00DCA804: numVerts
    0x00DCA814: subMaterials
    0x00DCA824: length
    0x00DCA82C: indexBuffers
    0x00DCA83C: annotations
    0x00DCA84C: indexType
    0x00DCA85C: uint16Data
    0x00DCA86C: hkxMesh
    0x00DCA874: uint32Stride
    0x00DCA884: vectorData
    0x00DCA894: attributeGroups
    0x00DCA8A4: ambientColor
    0x00DCA8B4: byteOffset
    0x00DCA8C4: hkxVertexBufferVertexData
    0x00DCA8E4: indices16
    0x00DCA8F4: indices32
    0x00DCA904: hkpSphereMotion
    0x00DCA914: hkxVertexDescription
    0x00DCA92C: hkxAttributeGroup
    0x00DCA944: texture
    0x00DCA94C: diffuseColor
    0x00DCA95C: material
    0x00DCA96C: stages
    0x00DCA974: usage
    0x00DCA97C: userChannelInfos
    0x00DCA994: specularColor
    0x00DCA9A4: mesh
    0x00DCA9AC: vectorStride
    0x00DCA9BC: byteStride
    0x00DCA9CC: sections
    0x00DCA9DC: hkxAttribute
    0x00DCA9EC: uint32Data
    0x00DCA9FC: desc
    0x00DCAA04: uint16Stride
    0x00DCAA14: tcoordChannel
    0x00DCAA24: hkxAttributeHolder
    0x00DCAA3C: vertexBuffer
    0x00DCAA4C: emissiveColor
    0x00DCAA5C: hkxVertexBuffer
    0x00DCAA6C: vertexBaseOffset
    0x00DCAA84: hkxAnimatedVector
    0x00DCAA9C: vectors
    0x00DCAAA4: hkxMeshUserChannelInfo
    0x00DCAABC: usageHint
    0x00DCAACC: isHierarchicalCompound
    0x00DCAAE4: extraData
    0x00DCAAF4: hkpShapeInfo
    0x00DCAB04: childShapeNames
    0x00DCAB14: localFrame
    0x00DCAB24: childTransforms
    0x00DCAB34: hkdShapesCollected
    0x00DCAB4C: hkLocalFrame
    0x00DCAB64: children
    0x00DCAB74: floats
    0x00DCAB7C: parent
    0x00DCAB84: hkMemoryResourceHandleExternalLink
    0x00DCABAC: hkMemoryResourceHandle
    0x00DCABC4: hkResourceContainer
    0x00DCABDC: references
    0x00DCABEC: hkResourceHandle
    0x00DCAC04: memberName
    0x00DCAC14: hkMemoryResourceContainer
    0x00DCAC34: hkResourceBase
    0x00DCAC44: externalId
    0x00DCAC54: resourceHandles
    0x00DCAC64: near
    0x00DCAC6C: materials
    0x00DCAC7C: leftHanded
    0x00DCAC8C: hkxTextureInplace
    0x00DCACA4: angle
    0x00DCACAC: selectedNodes
    0x00DCACBC: position
    0x00DCACCC: hkxScene
    0x00DCACDC: cameras
    0x00DCACE4: rootNode
    0x00DCACF4: filename
    0x00DCAD04: modeller
    0x00DCAD14: bindPose
    0x00DCAD24: selected
    0x00DCAD34: hkxNodeAnnotationData
    0x00DCAD4C: fileType
    0x00DCAD5C: asset
    0x00DCAD64: far
    0x00DCAD6C: object
    0x00DCAD74: meshes
    0x00DCAD7C: userProperties
    0x00DCAD8C: hkxNode
    0x00DCAD94: externalTextures

    There's a lot more, but I don't know if that has to do with the actual model file...
    What's interesting is that (some of) the physics data is also stored in the model file.
    It looks like the mesh data is stored in chunks... not sure if one chunk equals one primitive, but it's worth considering...

    Btw, here's a screenshot from the output of the data as how it's stored in the BigFile indices file (BigFile_PS3.sdat):
    Image
    It's the beginning of my all-in-one parser tool thingy :3

    cra0 wrote:
    HugoPeters wrote:
    By the way, I haven't found any texture blocks yet.... do they start without a Communicator ID? Because I did see a few of those that looked like they could be textures...

    The textures are in DXT1 DXT5 (DXTATI2N - NORMALS), if I recall the segs script EKEY wrote dumped some texture files.

    Do the texture files start with a Communicator ID? like COM_CONT, ANIMDATA, PARTITIO_, etc?
    Can you attach a sample texture file? So I'm not "groping in the dark"?

    _________________
    Image

    "I'm going to throw my shit at you" —The Great Mighty Poo
    _______________________>> check my incomplete portfolio I wish the [center] bbcode was turned on
    _______________________>> or my twitter
    _______________________>> or my game dev FB page


    Top
     Profile  
     
     Post subject: Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
    PostPosted: Sat Apr 19, 2014 1:23 am 
    Offline
    veteran

    Joined: Fri Jun 01, 2012 10:08 am
    Posts: 122
    Has thanked: 31 times
    Have thanks: 64 times
    Nah the cache dump is different to the ram dump, but I dumped the ram as well soI can give you that also. The cache dump has the scene and and index and DAT file etc, like TheEmbassy.idx TheEmbassy.DAT there's lots of Segs and other stuff insid. I became less interested as time went on but I still have the files. I'll send them to you both soon


    Top
     Profile  
     
     Post subject: Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
    PostPosted: Sun Apr 27, 2014 11:56 pm 
    Offline
    veteran
    User avatar

    Joined: Thu Apr 03, 2014 9:49 am
    Posts: 129
    Location: The Netherlands
    Has thanked: 62 times
    Have thanks: 166 times
    Update!
    Alright, got quite a lot of stuff going on!
    cra0 and I are working on the ultimate Beyond tool!

    I'm doing most of the tool itself, the GUI, parser, structure, base features, decompressor, extractor, etc. while cra0 is reversing the model format! And another big thing for me: I've made the switch from GameMaker to C# in Visual Studio..... and damn. It's like heaven. I really want to thank cra0 for helping we with all my noob questions, I'm already getting the hang of it!

    Here are some screenshots from the tool in its current form (which is: very unfinished / very early):

    Image
    Left is where you select the main entries group. The entries are defined from the decrypted BigFile_PS3.sdat file. Then once you loaded the entries group, you get a nice display of all the entries in that group!

    Image
    Before you can actually begin looking in the entries to see what data they contain you must first Inspect them. You can Inspect every individual entry manually by selecting it, then going to the Inspector tab and then clicking the Inspect button, but you can also Inspect all entries at once by clicking the Tools menu.
    Once entries have been Inspected you can do things like searching the entries for types of content (Communicators). The filter can be set by selecting the "Filter" tab on the left... you can also choose between two filter modes: the entry must contain all Communicators in the filter, or the entry must contain one or more of the Communicators in the filter! When you click "Add Communicator" you get a nice popup with a combo box consisting of all known Communicators.

    Image
    You can also filter the currently loaded entries on offset!

    Image
    This is the Inspector tab. On the Inspector's Base tab, you can see an overview of all Communicators found within the entries. As you can see in this example, it contains both Communicators set in the filter!

    Image
    This is a feature in a very early stage: an output of how the entry is constructed.

    Image
    Here's a neat one: when you double click a Communicator in the Inspector's Base tab, you will be taken to the Hex Viewer, and the selected offset will be automatically set to the offset of that Communicator!

    Also, don't forget the decompressor! I found out more about how these files are built, and most of them consist of more than one segs archive!

    So yeah, let me know what you guys think! If you have any suggestions, please let me know!

    _________________
    Image

    "I'm going to throw my shit at you" —The Great Mighty Poo
    _______________________>> check my incomplete portfolio I wish the [center] bbcode was turned on
    _______________________>> or my twitter
    _______________________>> or my game dev FB page


    Top
     Profile  
     
     Post subject: Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
    PostPosted: Thu May 01, 2014 9:50 pm 
    Offline
    veteran
    User avatar

    Joined: Thu Apr 03, 2014 9:49 am
    Posts: 129
    Location: The Netherlands
    Has thanked: 62 times
    Have thanks: 166 times
    another huge update with a title in a huge font

    Thanks to JayK's files I reverse engineered the DC_INFO sections in Beyond's file structure.

    Too tired for an explanation now and since no one ever replies here I'm not going to bother posting the structure until someone asks me too :dance:

    Here's a sample:

    Image

    _________________
    Image

    "I'm going to throw my shit at you" —The Great Mighty Poo
    _______________________>> check my incomplete portfolio I wish the [center] bbcode was turned on
    _______________________>> or my twitter
    _______________________>> or my game dev FB page


    Top
     Profile  
     
     Post subject: Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
    PostPosted: Fri May 02, 2014 11:07 pm 
    Offline
    veteran

    Joined: Fri Jun 01, 2012 10:08 am
    Posts: 122
    Has thanked: 31 times
    Have thanks: 64 times
    Nice, glad them files came in handy.


    Top
     Profile  
     
     Post subject: Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
    PostPosted: Sun May 04, 2014 11:28 pm 
    Offline
    veteran
    User avatar

    Joined: Thu Apr 03, 2014 9:49 am
    Posts: 129
    Location: The Netherlands
    Has thanked: 62 times
    Have thanks: 166 times
    Alrighty, I now partly understand the way the game accesses its content.
    Again, not going to explain too deeply because I suck at explaining and this is pretty complicated xD

    Here's a quick look though:

    Image

    This is for dialog. For example, the first red square gives an index into the FLOW_DIALOG content group, which is included in these:

    Image

    The enum value for FLOW_DIALOG(_OFFSET) is 0xffc / 0xffb.
    Then, the File ID is the next int, in this example it's 0x131a.

    Using my tool I can quickly seek to the file bound to that ID, and then hear the audio, and it matches the sentence! :D

    This ID, 0x3f2 stands for AUDIO_DATA. This is from one of JayK's files, The Party.dat or something like that. When I go to the file with that ID, surprise surprise, it's the "Beautiful Body" song!
    Image

    _________________
    Image

    "I'm going to throw my shit at you" —The Great Mighty Poo
    _______________________>> check my incomplete portfolio I wish the [center] bbcode was turned on
    _______________________>> or my twitter
    _______________________>> or my game dev FB page


    Top
     Profile  
     
     Post subject: Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
    PostPosted: Tue May 06, 2014 11:50 am 
    Offline
    veteran
    User avatar

    Joined: Thu Apr 03, 2014 9:49 am
    Posts: 129
    Location: The Netherlands
    Has thanked: 62 times
    Have thanks: 166 times
    I've made a video demonstrating the tool and how I find matching audio with text dialog (FLOW_DIALOG):

    https://www.youtube.com/watch?v=vunRjVMVXY0

    _________________
    Image

    "I'm going to throw my shit at you" —The Great Mighty Poo
    _______________________>> check my incomplete portfolio I wish the [center] bbcode was turned on
    _______________________>> or my twitter
    _______________________>> or my game dev FB page


    Top
     Profile  
     
     Post subject: Re: [PS3] Beyond: Two Souls -Extractor- + Beginnings of pars
    PostPosted: Tue May 06, 2014 3:25 pm 
    Offline
    veteran
    User avatar

    Joined: Thu Apr 03, 2014 9:49 am
    Posts: 129
    Location: The Netherlands
    Has thanked: 62 times
    Have thanks: 166 times
    Texture format is reversed!

    Image


    _________________
    Image

    "I'm going to throw my shit at you" —The Great Mighty Poo
    _______________________>> check my incomplete portfolio I wish the [center] bbcode was turned on
    _______________________>> or my twitter
    _______________________>> or my game dev FB page


    Top
     Profile  
     
    Display posts from previous:  Sort by  
    Post new topic Reply to topic  [ 236 posts ]  Go to page 1, 2, 3, 4, 5 ... 16  Next

    All times are UTC + 1 hour


    Who is online

    Users browsing this forum: No registered users and 1 guest


    You cannot post new topics in this forum
    You cannot reply to topics in this forum
    You cannot edit your posts in this forum
    You cannot delete your posts in this forum
    You cannot post attachments in this forum

    Search for:
    Jump to:  
    Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group