XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Thu Nov 23, 2017 1:12 am

All times are UTC + 1 hour




Post new topic Reply to topic  [ 20 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Heavy Rain Tool
PostPosted: Sun Sep 03, 2017 2:42 pm 
Offline
ultra-n00b

Joined: Sat Sep 02, 2017 1:09 pm
Posts: 4
Has thanked: 2 times
Have thanks: 7 times

The beginnings of the ultimate Heavy Rain extractor tool, or even PC port in the future - Heavy Rain Tool

Image

After finding @HugoPeters work on Beyond, I became really interested in the concept of reverse-engineering a game engine, so I started working on my favourite game - Heavy Rain.

Currently I managed to:
  • Read the BigFile index (the sdat encrypted file)
  • Find and display a list of entries grouped by type
  • Unpack all of the segs archived entries
  • Read the inner structure of the entries
  • Parse some of the blocks of the entries to extract data from them:
  • Textures (some of them are misaligned)
  • Scripts that are not compiled
  • BIK Video files
  • Some of the audio files + Multi-Channel audio exporting (Mostly soundtrack)

What I'm working on:
  • Linking the entries together by parsing the storyboard
  • Properly aligning textures

Tool download link: https://drive.google.com/open?id=0B32koR8mJOyBU0FkSXg2SHMtMzg
(Last updated: 2017-09-12, PreAlpha 3)
GitHub Repository: https://github.com/MushroomLT/qdengine

Tool usage:
* Run the tool to generate "engine.json" configuration file.
* Open the configuration file and set "qdGameDirectory" to the game file directory location
* Set vlcExeLocation to the location of your VLC Media Player installation to be able to play sounds directly from the tool
* Set loadBigFile to the BigFile index file you want to load.

The tool can extract any entry into a separate binary file by selecting an entry and clicking the Export button.
To view the inner blocks of the entry double click it in the tree. Selecting a block will automatically adjust the offset in the hex view.
To export video from an entry double click the entry, and double click the VIDEDATA block.
To export sounds from an entry double click the entry, and double click the PARTITIO block.
To play the exported sound using VLC player, double click the entry inside of the PARTITIO block.

To view scripts, double click the SCRIPT block, and select the inner script entry, it will be displayed under the Contents tab.

Extracted soundtrack demo: https://drive.google.com/open?id=0B32koR8mJOyBNGRzckhoX3plN28

You can make the ads go away by registering



Last edited by MushroomLT on Tue Sep 12, 2017 2:03 pm, edited 3 times in total.

Top
 Profile  
 
 Post subject: Some progress
PostPosted: Fri Sep 08, 2017 5:17 pm 
Offline
ultra-n00b

Joined: Sat Sep 02, 2017 1:09 pm
Posts: 4
Has thanked: 2 times
Have thanks: 7 times
I managed to perform a RAM dump of RPCS3 running the game by modifying it's source, and from the dump I found the LUA initialisation code for probably all of the data types. Now the tool is capable of displaying what data each entry has:
Code:
CommunicatorBlock Contents:
DCINFO Entries:
   Entry 0: SOUND_OFFSET; 64112
   Entry 1: SOUND_DATA; 64108
   Entry 2: SCRIPT_SOUND_ENTITY; 15116
   Entry 3: SOUND_OFFSET; 64116
   Entry 4: SOUND_DATA; 64112
   Entry 5: SCRIPT_SOUND_ENTITY; 15118
   Entry 6: SOUND_OFFSET; 66806
   Entry 7: SOUND_DATA; 66803
   Entry 8: SCRIPT_SOUND_ENTITY; 15117
   Entry 9: SOUND_OFFSET; 46980
   Entry 10: SOUND_DATA; 46976
   Entry 11: SCRIPT_SOUND_ENTITY; 10944
   Entry 12: SOUND_OFFSET; 64114
   Entry 13: SOUND_DATA; 64110
   Entry 14: SCRIPT_SOUND_ENTITY; 15120
   Entry 15: SOUND_OFFSET; 62123
   Entry 16: SOUND_DATA; 62119
   Entry 17: SCRIPT_SOUND_ENTITY; 10947
   Entry 18: SCRIPT_SOUND_ENTITY; 10945
   Entry 19: SCRIPT_SOUND_ENTITY; 15119
   Entry 20: LOADING_ZONE; 5681
   Entry 21: LOADING_ZONE; 5756
   Entry 22: LOADING_ZONE; 6602
   Entry 23: LOADING_ZONE; 6630
   Entry 24: LOADING_ZONE; 6676
   Entry 25: LOADING_ZONE; 6718
   Entry 26: LOADING_ZONE; 6767
   Entry 27: LOADING_ZONE; 6779
   Entry 28: LOADING_ZONE; 22721
   Entry 29: LOADING_ZONE; 22724
   Entry 30: LOADING_ZONE; 22727
   Entry 31: LOADING_ZONE; 22729
   Entry 32: LOADING_ZONE; 22754
   Entry 33: LOADING_ZONE; 38606
   Entry 34: STORYBOARD; 46
   Entry 35: LOADING_ZONE; 5674
   Entry 36: LOADING_ZONE; 5675
   Entry 37: LOADING_ZONE; 5677
   Entry 38: LOADING_ZONE; 5678
   Entry 39: LOADING_ZONE; 5679
   Entry 40: LOADING_ZONE; 5689
   Entry 41: LOADING_ZONE; 5690
   Entry 42: LOADING_ZONE; 5691
   Entry 43: LOADING_ZONE; 5704
   Entry 44: LOADING_ZONE; 5739
   Entry 45: LOADING_ZONE; 5740
   Entry 46: LOADING_ZONE; 5741
   Entry 47: LOADING_ZONE; 5754
   Entry 48: LOADING_ZONE; 6498
   Entry 49: LOADING_ZONE; 6569
   Entry 50: LOADING_ZONE; 6659
   Entry 51: LOADING_ZONE; 6662
   Entry 52: LOADING_ZONE; 6669
   Entry 53: LOADING_ZONE; 6670
   Entry 54: LOADING_ZONE; 6679
   Entry 55: LOADING_ZONE; 6686
   Entry 56: LOADING_ZONE; 6689
   Entry 57: LOADING_ZONE; 6696
   Entry 58: LOADING_ZONE; 6697
   Entry 59: LOADING_ZONE; 6698
   Entry 60: LOADING_ZONE; 6701
   Entry 61: LOADING_ZONE; 6703
   Entry 62: LOADING_ZONE; 6705
   Entry 63: LOADING_ZONE; 6706
   Entry 64: LOADING_ZONE; 6707
   Entry 65: LOADING_ZONE; 6709
   Entry 66: LOADING_ZONE; 6711
   Entry 67: LOADING_ZONE; 6714
   Entry 68: LOADING_ZONE; 6716
   Entry 69: LOADING_ZONE; 6719
   Entry 70: LOADING_ZONE; 6720
   Entry 71: LOADING_ZONE; 6723
   Entry 72: LOADING_ZONE; 6724
   Entry 73: LOADING_ZONE; 6725
   Entry 74: LOADING_ZONE; 6728
   Entry 75: LOADING_ZONE; 6730
   Entry 76: LOADING_ZONE; 6732
   Entry 77: LOADING_ZONE; 6733
   Entry 78: LOADING_ZONE; 6734
   Entry 79: LOADING_ZONE; 6738
   Entry 80: LOADING_ZONE; 6740
   Entry 81: LOADING_ZONE; 6742
   Entry 82: LOADING_ZONE; 6745
   Entry 83: LOADING_ZONE; 6748
   Entry 84: LOADING_ZONE; 6753
   Entry 85: LOADING_ZONE; 6755
   Entry 86: LOADING_ZONE; 6758
   Entry 87: LOADING_ZONE; 6760
   Entry 88: LOADING_ZONE; 6771
   Entry 89: LOADING_ZONE; 6773
   Entry 90: LOADING_ZONE; 6774
   Entry 91: LOADING_ZONE; 6776
   Entry 92: LOADING_ZONE; 6777
   Entry 93: LOADING_ZONE; 6785
   Entry 94: LOADING_ZONE; 6788
   Entry 95: LOADING_ZONE; 6789
   Entry 96: LOADING_ZONE; 6792
   Entry 97: LOADING_ZONE; 6795
   Entry 98: LOADING_ZONE; 6797
   Entry 99: LOADING_ZONE; 6798
   Entry 100: LOADING_ZONE; 6799
   Entry 101: LOADING_ZONE; 6800
   Entry 102: LOADING_ZONE; 6801
   Entry 103: LOADING_ZONE; 6805
   Entry 104: LOADING_ZONE; 6807
   Entry 105: LOADING_ZONE; 6808
   Entry 106: LOADING_ZONE; 6810
   Entry 107: LOADING_ZONE; 6811
   Entry 108: LOADING_ZONE; 6814
   Entry 109: LOADING_ZONE; 6816
   Entry 110: LOADING_ZONE; 6817
   Entry 111: LOADING_ZONE; 6818
   Entry 112: LOADING_ZONE; 6825
   Entry 113: LOADING_ZONE; 6828
   Entry 114: LOADING_ZONE; 6830
   Entry 115: LOADING_ZONE; 6832
   Entry 116: LOADING_ZONE; 6834
   Entry 117: LOADING_ZONE; 6835
   Entry 118: LOADING_ZONE; 6837
   Entry 119: LOADING_ZONE; 7700
   Entry 120: LOADING_ZONE; 10309
   Entry 121: LOADING_ZONE; 10419
   Entry 122: LOADING_ZONE; 17933
   Entry 123: LOADING_ZONE; 18595
   Entry 124: LOADING_ZONE; 20832
   Entry 125: LOADING_ZONE; 20834
   Entry 126: LOADING_ZONE; 20836
   Entry 127: LOADING_ZONE; 20850
   Entry 128: LOADING_ZONE; 20852
   Entry 129: LOADING_ZONE; 22359
   Entry 130: LOADING_ZONE; 23752
   Entry 131: LOADING_ZONE; 23756
   Entry 132: LOADING_ZONE; 23761
   Entry 133: LOADING_ZONE; 23763
   Entry 134: LOADING_ZONE; 23969
   Entry 135: LOADING_ZONE; 25561
   Entry 136: LOADING_ZONE; 28431
   Entry 137: LOADING_ZONE; 29901
   Entry 138: LOADING_ZONE; 29907
   Entry 139: LOADING_ZONE; 29909
   Entry 140: LOADING_ZONE; 29911
   Entry 141: LOADING_ZONE; 29913
   Entry 142: LOADING_ZONE; 29915
   Entry 143: LOADING_ZONE; 29917
   Entry 144: LOADING_ZONE; 29920
   Entry 145: LOADING_ZONE; 29924
   Entry 146: LOADING_ZONE; 29928
   Entry 147: LOADING_ZONE; 29932
   Entry 148: LOADING_ZONE; 29937
   Entry 149: LOADING_ZONE; 29953
   Entry 150: LOADING_ZONE; 29956
   Entry 151: LOADING_ZONE; 33069
   Entry 152: LOADING_ZONE; 33215
   Entry 153: LOADING_ZONE; 33239
   Entry 154: LOADING_ZONE; 33241
   Entry 155: LOADING_ZONE; 34655
   Entry 156: LOADING_ZONE; 35557
   Entry 157: LOADING_ZONE; 35863
   Entry 158: LOADING_ZONE; 38507
   Entry 159: LOADING_ZONE; 39037
   Entry 160: LOADING_ZONE; 39041
   Entry 161: LOADING_ZONE; 39269
   Entry 162: LOADING_ZONE; 39448
   Entry 163: LOADING_ZONE; 39519
   Entry 164: LOADING_ZONE; 39521
   Entry 165: LOADING_ZONE; 39523
   Entry 166: LOADING_ZONE; 40373
   Entry 167: STORYBOARD; 35
   Entry 168: ENGINE_TEXTURE_FILE; 49948
   Entry 169: ENGINE_TEXTURE_FILE; 50553
   Entry 170: ENGINE_TEXTURE_FILE; 54206
   Entry 171: ENGINE_TEXTURE_FILE; 54207
   Entry 172: ENGINE_TEXTURE_FILE; 54208
   Entry 173: ENGINE_TEXTURE_FILE; 54209
   Entry 174: ENGINE_TEXTURE_FILE; 54210
   Entry 175: ENGINE_TEXTURE_FILE; 54211
   Entry 176: ENGINE_TEXTURE_FILE; 54212
   Entry 177: ENGINE_TEXTURE_FILE; 54213
   Entry 178: ENGINE_TEXTURE_FILE; 54214
   Entry 179: ENGINE_TEXTURE_FILE; 62181
   Entry 180: ENGINE_TEXTURE_FILE; 62182
   Entry 181: ENGINE_TEXTURE_FILE; 62183
   Entry 182: ENGINE_TEXTURE_FILE; 62184
   Entry 183: ENGINE_TEXTURE_FILE; 62185
   Entry 184: ENGINE_TEXTURE_FILE; 62186
   Entry 185: ENGINE_TEXTURE_FILE; 62187
   Entry 186: ENGINE_TEXTURE_FILE; 62188
   Entry 187: ENGINE_TEXTURE_FILE; 62921
   Entry 188: ENGINE_TEXTURE_FILE; 64307
   Entry 189: ENGINE_TEXTURE_FILE; 64308
   Entry 190: ENGINE_TEXTURE_FILE; 64309
   Entry 191: ENGINE_TEXTURE_FILE; 69866
   Entry 192: ENGINE_TEXTURE_FILE; 75541
   Entry 193: ENGINE_TEXTURE_FILE; 75542
   Entry 194: ENGINE_TEXTURE_FILE; 75771
   Entry 195: ENGINE_TEXTURE_FILE; 76140
   Entry 196: ENGINE_TEXTURE_FILE; 78367
   Entry 197: ENGINE_TEXTURE_FILE; 78368
   Entry 198: ENGINE_TEXTURE_FILE; 78369
   Entry 199: ENGINE_TEXTURE_FILE; 78370
   Entry 200: ENGINE_TEXTURE_FILE; 78371
   Entry 201: ENGINE_TEXTURE_FILE; 78372
   Entry 202: ENGINE_TEXTURE_FILE; 78373
   Entry 203: ENGINE_TEXTURE_FILE; 78374
   Entry 204: ENGINE_TEXTURE_FILE; 78375
   Entry 205: ENGINE_TEXTURE_FILE; 78376
   Entry 206: ENGINE_TEXTURE_FILE; 78377
   Entry 207: ENGINE_TEXTURE_FILE; 78378
   Entry 208: ENGINE_TEXTURE_FILE; 79345
   Entry 209: ENGINE_TEXTURE_FILE; 62189
   Entry 210: ENGINE_TEXTURE_FILE; 69848
   Entry 211: MENUS_MASTER_FONT_DATA; 9
   Entry 212: MENUS_MASTER_DATA; 28
   Entry 213: CHARACTER; 3
   Entry 214: CHARACTER; 345
   Entry 215: CHARACTER; 7
   Entry 216: CHARACTER; 6

DCDATA Size: 5336713; Entries: 7
   Entry 0: 0
   Entry 1: 48
   Entry 2: 0
   Entry 3: 0
   Entry 4: 256
   Entry 5: 0
   Entry 6: 1


Top
 Profile  
 
 Post subject: Textures reversed
PostPosted: Fri Sep 08, 2017 9:28 pm 
Offline
ultra-n00b

Joined: Sat Sep 02, 2017 1:09 pm
Posts: 4
Has thanked: 2 times
Have thanks: 7 times
Textures reversed!
Image


Top
 Profile  
 
 Post subject: Re: Heavy Rain Tool
PostPosted: Sat Sep 09, 2017 10:18 pm 
Offline
ultra-veteran
ultra-veteran

Joined: Fri Nov 06, 2009 12:13 am
Posts: 593
Has thanked: 70 times
Have thanks: 127 times
Awesome keep it up!

_________________
Useful tool links:


Top
 Profile  
 
 Post subject: Re: Heavy Rain Tool
PostPosted: Tue Sep 12, 2017 1:54 pm 
Offline
ultra-n00b

Joined: Sat Sep 02, 2017 1:09 pm
Posts: 4
Has thanked: 2 times
Have thanks: 7 times
DataContainer structure reversed - working on the storyboard now, which links all the scenes.
Image


Top
 Profile  
 
 Post subject: Re: Heavy Rain Tool
PostPosted: Sat Sep 16, 2017 11:25 pm 
Offline
veteran
User avatar

Joined: Thu Apr 03, 2014 9:49 am
Posts: 129
Location: The Netherlands
Has thanked: 62 times
Have thanks: 166 times
Great to see my stuff inspired a follow-up! :up:

_________________
Image

"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my incomplete portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
_______________________>> or my game dev FB page


Top
 Profile  
 
 Post subject: Re: Heavy Rain Tool
PostPosted: Sun Sep 17, 2017 11:55 am 
Offline
n00b

Joined: Fri Jul 22, 2011 4:39 pm
Posts: 19
Has thanked: 21 times
Have thanks: 13 times
Awesome work dude! Also these are the scenes that were cut from the game in Beta status, fully playable but unpolished:

0501 Sleepless Night Underwater
0701 Going to School Wakeup
1002 Swimming Again

Since they have almost been completed and use mostly existing assets, everything you could find in regards to them would be quite amazing. I doubt you can fully restore them, but maybe they have a lot of traces left :D

_________________
Image


Top
 Profile  
 
 Post subject: Re: Heavy Rain Tool
PostPosted: Sun Sep 17, 2017 8:42 pm 
Offline
Moderator

Joined: Sun Mar 27, 2011 8:42 pm
Posts: 935
Has thanked: 10 times
Have thanks: 120 times
well i hope u finnish it since Hugo had to drop it.. :)

_________________
Quick BMS Editor GUI - simple easy to use
Goto : viewtopic.php?uid=34229&f=29&t=6797&start=0

Downloads from DropBox : https://dl.dropboxusercontent.com/u/


Top
 Profile  
 
 Post subject: Re: Heavy Rain Tool
PostPosted: Sun Sep 24, 2017 11:38 pm 
Offline
advanced

Joined: Tue Jun 30, 2009 11:35 am
Posts: 61
Has thanked: 28 times
Have thanks: 5 times
Any news?


Top
 Profile  
 
 Post subject: Re: Heavy Rain Tool
PostPosted: Sun Oct 01, 2017 1:23 pm 
Offline
double-veteran
double-veteran

Joined: Tue Mar 24, 2015 8:12 pm
Posts: 1233
Has thanked: 23 times
Have thanks: 604 times
Quantic Dream game formats were reversed all the way, and tools released to public. But it was forbidden to talk about this engine here on xentax. Any changes?

Image
Image


Top
 Profile  
 
 Post subject: Re: Heavy Rain Tool
PostPosted: Mon Oct 02, 2017 4:06 pm 
Offline
veteran
User avatar

Joined: Thu Apr 03, 2014 9:49 am
Posts: 129
Location: The Netherlands
Has thanked: 62 times
Have thanks: 166 times
daemon1 wrote:
Quantic Dream game formats were reversed all the way, and tools released to public. But it was forbidden to talk about this engine here on xentax. Any changes?

<snip>

No they were not, there was only a model extractor which just brute-force scans files for model data.

_________________
Image

"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my incomplete portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
_______________________>> or my game dev FB page


Top
 Profile  
 
 Post subject: Re: Heavy Rain Tool
PostPosted: Mon Oct 02, 2017 4:18 pm 
Offline
double-veteran
double-veteran

Joined: Tue Mar 24, 2015 8:12 pm
Posts: 1233
Has thanked: 23 times
Have thanks: 604 times
HugoPeters wrote:
daemon1 wrote:
Quantic Dream game formats were reversed all the way, and tools released to public. But it was forbidden to talk about this engine here on xentax. Any changes?

<snip>

No they were not, there was only a model extractor which just brute-force scans files for model data.


Yes, they were. You probably just don't know about that. And a proper tool was released which is able to extract all models and levels with textures, skeletons and fully weighted including cloth, from both Heavy Rain & Beyond Two Souls. I posted images with models extracted with that tool.

My question is: can I now talk about this on xentax?


Top
 Profile  
 
 Post subject: Re: Heavy Rain Tool
PostPosted: Tue Oct 03, 2017 6:31 am 
Offline
veteran

Joined: Wed Jun 17, 2009 4:33 pm
Posts: 118
Has thanked: 21 times
Have thanks: 4 times
I think the tool was on the Cg in Games forum (QDtools)

but it seems it cannot be downloaded any longer

It's to bad since I sold my copy of Heavy Rain for PS3 a while back because I figured there was never going to be a way to get the models
I would probably go buy another copy if I knew the tools were available
(actually I bet a lot of people might buy a copy)


Top
 Profile  
 
 Post subject: Re: Heavy Rain Tool
PostPosted: Tue Oct 03, 2017 4:16 pm 
Offline
double-veteran
double-veteran

Joined: Tue Mar 24, 2015 8:12 pm
Posts: 1233
Has thanked: 23 times
Have thanks: 604 times
Modman69 wrote:
I think the tool was on the Cg in Games forum (QDtools)


Tools can still be downloaded from CG and also Facepunch, and maybe other sites.


Top
 Profile  
 
 Post subject: Re: Heavy Rain Tool
PostPosted: Wed Oct 04, 2017 10:50 am 
Offline
veteran
User avatar

Joined: Thu Apr 03, 2014 9:49 am
Posts: 129
Location: The Netherlands
Has thanked: 62 times
Have thanks: 166 times
daemon1 wrote:
HugoPeters wrote:
daemon1 wrote:
Quantic Dream game formats were reversed all the way, and tools released to public. But it was forbidden to talk about this engine here on xentax. Any changes?

<snip>

No they were not, there was only a model extractor which just brute-force scans files for model data.


Yes, they were. You probably just don't know about that. And a proper tool was released which is able to extract all models and levels with textures, skeletons and fully weighted including cloth, from both Heavy Rain & Beyond Two Souls. I posted images with models extracted with that tool.

My question is: can I now talk about this on xentax?

Again, no they were not. Apparently you think models are the only thing what makes a game ;)
The tool you're talking about, like I said, brute-force scans files for model data. I didn't say it doesn't produce proper models.
Give me a tool that can also interpret animation data and all the rest that comes with it and can produce a scene like I showcase on my portfolio (http://hugo.fyi/projects/reverse-engineering-b2s) - then we'll talk :bleh:


_________________
Image

"I'm going to throw my shit at you" —The Great Mighty Poo
_______________________>> check my incomplete portfolio I wish the [center] bbcode was turned on
_______________________>> or my twitter
_______________________>> or my game dev FB page


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 20 posts ]  Go to page 1, 2  Next

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group