[PS3] GTA V .CDR 3D Drawables Reversing edge related

[psxmodder]

Hello people, a lot of work have been done about drawables of RAGE engine on pc and even on x360, for example on x360 there is this xdr2obj https://github.com/tgascoigne/xdr2obj but nothing for the ps3 version because there is a different approach to store index and vertex data.

There is a documentation even from that xdr2obj git "xdr.org" with many common things with the ".wdr" doc here: http://gta.wikia.com/wiki/WDR and our .cdr from PS3 is really close but just at some header fields later when starts the ps edge compression approach the things becomes different!.

I started looking around console format's some time ago had done a simple app to decompress RSC resource archive but now i'm working on reversing the 3d data from RAGE engine on GTA V but PS3 version. There are .cdr,cft and cdd formats right now i'm focus on .cdr drawables there are a lot of wiki's and info about the x360 one .xdr and obviusly for pc .wdr (gta iv) the thing is totally different on ps3 -x360, i mean on old gen consoles the format have more things in common with the old .WDR from GTA IV even there was that xdr2obj for x360 because the .xdr(xbox360) and .wdr old gta iv pc are really the same as i see when searching with hxd BUT the thing for PS3 is really different!, have some days and at least with the .cdr i managed to get vertex extracted (i believe they are) (format of vertex is the most common v(x,y,z) etc) and have done reverse engineering of many offsets in the file i'm writing a doc about my findings.

Offsets documented:




The section 0x1D10 that i use is from asteroid .cdr but every .CDR have one section like that maybe is some kind of vertex buffer because have important information for the model.

Here are some .cdr files maybe any other dev with experience can help me. About the .cdr they were decompressed from zlib rockstar it uses RSC format with my experimental program written on java i managed to get it unpacked on ps3 version they just use zlib on x360 they use lzx. because that there is no RSC header on the files.

https://drive.google.com/open?id=0B7ouK ... jItV2R5SjQ
That is a image of what i have done in the few time. is an asteroid model with only points.

And this is .zip with .cdr and images with some analysis patterns even i have extracted some blocks that are on the file with hxd. They are three blocks i think i got vertex data from the one named "asteroid_18800-0x2D60-block" being 18800 the offset on that block in the file and 0x2D60 his length and when i check 0x2D60/10 it gives me 0x2D6 that is actually the verts count so maybe i'm not so wrong. the other two i don't know anything must be skeleton or shaders related.

https://drive.google.com/open?id=0B7ouK ... zBGamJHSWM

That file contains an asteroid model just like this one:



I made some more research but now i know more about this format. Rockstar use Playstation Edge in order to optimize vertex cache and indexes to better load on spus of cell cpu just like this doc says:

http://www.jonolick.com/uploads/7/9/2/1 ... _final.pdf

Sony PS Edge vertex and index format is used in many games from GOW 3, Playstation all stars, etc. In any heavy game with lot of 3d stuff it should be used.

Here in this forums i started to see some topics about ps edge compression and see the work of chrrox (thanks to him for his scripts).

There is a function on noesis include's "rapi.decompressEdgeIndices(edgeData, indexCount)" that does all the magic obviusly when you know on the file where is located the index offset, vertex count, index count, index size, etc. I managed today to get that more documented in a doc file there is it:
https://drive.google.com/file/d/0B7ouKH ... sp=sharing

For faster results i just use the asteroid.cdr file so in the doc are some fixed offsets and values for this file only but should work many of that fields with even all .cdr that i look around. Well this doc like i say is usefull when you have the asteroid cdr.

viewtopic.php?f=18&t=11954 Here is the link to noesis-plugin that chrrox write for ps all stars and i use that src to check if there is that kind of format and YES i get somethings in common between ps all stars and maybe every edge related format with our .CDR from RAGE GTA V on ps3 and that is what i had wrote on the document.

Here is another picture too with colors of that block (asteroid.cdr:0x1D10 fixed offset) that i quote in the previous document:

https://drive.google.com/file/d/0B7ouKH ... sp=sharing

Hurray! I managed to get decompressed the index data! using that offsets in the doc and as the picture says and using the python plugin with noesis func "rapi.decompressEdgeIndices" but that is just faces and triangles and this time is Ok the data from what i see.

bs.seek(indexOffset, NOESEEK_ABS)
edgeData = bs.readBytes(indexSize)
edgeDecomp = rapi.decompressEdgeIndices(edgeData, indexCount)
(this is from chrrox plugin to ps all stars)

I get this file:
https://drive.google.com/file/d/0B7ouKH ... sp=sharing

here is a picture of how it looks index data now that is decompressed (v1,v2,v3) short values:
https://drive.google.com/file/d/0B7ouKH ... sp=sharing

Later i make a simple java app to convert every short to integer values of two bytes as the index data should normally be ( for example: f 1 2 3, f 3 2 0, etc) and get :

http://pastebin.com/csQjWwd4

Looks according to index information that there are 724 verts in our asteroid cdr sample at least and in the offset that i believe it is vert count says that there are 726 or 0x2D6 so maybe we are more close to the goal

Here is the "experimental" python (noesis compatible based on chrrox ps all stars, again thanks to him for marvelous work!) this is a modified script to work with (asteroid.cdr but can work with other .cdr with only one geometry i think and changing the value 0x3000 to where graphics section of resource starts according to .RSC format) BUT IT works just to decompress edges at this time don't try to export .obj or anything this script is just to fwrite decompressed faces to another file. The python script doesn have some offsets rights.

http://pastebin.com/7EEiw3LA

Again this python script for noesis is just to use this:

Code: Select all

edgeDecomp = rapi.decompressEdgeIndices(edgeData, indexCount)

Until that step the python script works ok but as i'm just in the step of make documentation i'm just using hex editor, building quickly with java, python and anything that can help me in the process of reversing.

Here are some more .cdr files if anyone want to look on this.
https://drive.google.com/file/d/0B7ouKH ... sp=sharing

Now need more work with kind of file, it should be great if we can do something with the format because i think that even Red dead redemption work in this way and maybe in the far future we can mod console versions. I need help from more experienced devs with this 3d reversing stuff i'm a programmer but i'm new in these stuff. Again thanks to everyone!. sorry for my bad english.

[psxmodder]

Good news i think i managed to get vertex values (x,y,z,w?) floats extracted from the block that i "BELIEVE" that have vertex positions, as i say on the post is the block located always maybe all .cdr from what i see on (0x15800+CPU SECTION LENGTH) in the case of asteroid cdr it was 0x3000 so it becomes 0x18800 and according to the possibly "vert count" 0x2D6 after doing (vert count * 0x10) by the vertex stride (4b floats) i get the length 0x2D60 of that block. well there is it on .obj but only vertex.

http://pastebin.com/nCcH7rAU vertex

and the thing is that i get some errors because there are some "NAN" not a number values on some vertex,

Again as the first post says i already get decompressed face indexed list here is the sample from asteroid too:

http://pastebin.com/csQjWwd4

The index list says that there should be 724 verts and the vertex data stuff says that too but when i get this two vertex and faces on only one sample .OBJ i get errors. Need more help with this i think

[shakotay2]

http://pastebin.com/csQjWwd4

The index list says that there should be 724 verts and the vertex data stuff says that too but when i get this two vertex and faces on only one sample .OBJ i get errors. Need more help with this i think

f 0/0 1/1 2/2

wavefront obj face indices are 1-based. Add 1 to each face index.


btw: very interesting that you deal with edge decompression!
But you're producing walls of text; sadly I don't have the time to go through it.

[psxmodder]

f 0/0 1/1 2/2

wavefront obj face indices are 1-based. Add 1 to each face index.

btw: very interesting that you deal with edge compression!
But you're producing walls of text; sadly I don't have the time to go through it.

Yeah i wrote a lot of text i will try to make it shorter, Thanks for that gonna add +1 to each index and i have seen some of your works (codes) too thank you for everything you have done.

Yeah i had spend a few days until i figured out that is using edge compression, i'm starting to use asteroid sample because it just have one geometry from what i see on hxD but i been doing some more research and now today finger out how to work with many geometries or mdls in one file.

We should do some kind of topic about edge compression in his many ways because i see that many games uses it but they change the offsets and the way they likely mix the pc format with the sony edge compression method from what i see.

Edit: Well now Blender open it but is showing this weird geo:



And in the center is where are a lots of vertex and faces even when i come to wire view i see how there is really the asteroid form (yeah maybe i'm crazy xD) but there is bug that make faces goes beyond every limit and make a ugly form like in the picture well need to finger out what is happening.

I think that i need to do something to the vertex before just take 0x10 bytes and convert to IEEE floats maybe there is something more with the positions of they, or maybe is java converter that is not taking the numbers well, here is the vertex data it is in (4b floats (x,y,z,w)) maybe you can see if i'm not doing a good intbitstofloat convertion.

https://drive.google.com/file/d/0B7ouKH ... sp=sharing

[shakotay2]

this all is a little bit confusing to me (maybe you should give your thoughts more structure? )

First of all I don't think that it's floats in your asteroid_18800-0x2D60-vertex.block

Code: Select all

Offset    0  1  2  3  4  5  6  7   8  9  A  B  C  D  E  F

000000   5D 4C 7D C0 2C 53 30 03  11 FF 74 FF CF 80 7F FF

2ndly when I use your python script on prop_asteroid_01.cdr.out (where I cut off the .out since you claimed it to be a .cdr (!?) in your opening post) I get these face indices:
...
f 164/164/164 163/163/163 160/160/160
f 165/165/165 163/163/163 164/164/164
f 166/166/166 165/165/165 164/164/164

which are far from 724

Also (0x1B560 - 0x18800) / 16 = 726

[psxmodder]

this all is a little bit confusing to me (maybe you should give your thoughts more structure? )

First of all I don't think that it's floats in your asteroid_18800-0x2D60-vertex.block

Code: Select all

Offset    0  1  2  3  4  5  6  7   8  9  A  B  C  D  E  F

000000   5D 4C 7D C0 2C 53 30 03  11 FF 74 FF CF 80 7F FF

2ndly when I use your python script on prop_asteroid_01.cdr.out (where I cut off the .out since you claimed it to be a .cdr (!?) in your opening post) I get these face indices:
...
f 164/164/164 163/163/163 160/160/160
f 165/165/165 163/163/163 164/164/164
f 166/166/166 165/165/165 164/164/164

which are far from 724

Also (0x1B560 - 0x18800) / 16 = 726

Uhmm no that python script was just to use the function of rapi.decompress edge and use fwrite to get that index decompressed to the file uploaded before. Don't use the file .obj that noesis exports because i forgot to say that is not correctly generated as you see is not yet using correct vertex info and maybe because that it shrinks the face index number. If you are using the python script with the .cdr file there should be a new "index-data-decompressed.block" in NOESIS folder i'm working with that file, sorry because i forgot to say that but the thing is that there are a lot of info and i'm working with other stuff too.

The python script is usefull just until it comes to decompressedges part xD.

i'm using a simple java app ("REALLY SIMPLE") to get 2 bytes integer from "index-data-decompressed.block" outputted from noesis to floats in v(a,b,c) patterns and with that i make that .txt with faces index list. Well that stuff looks Ok but maybe as you say i'm wrong with the thing about vertex data that maybe the one i choosen was not the floats that we need.

here is the code from that java app:
http://pastebin.com/PNFhr5U7

Also (0x1B560 - 0x18800) / 16 = 726

0x18800 to 0x1B560 = 0x2D60 Length / 16 = 726

Yeah that was my thought but as you say it doesn't look to be floats in that data but at least there is the pattern of 0x10(I would have to be blind not to see it xD) that looks good.

[shakotay2]

i'm using a simple java app ("REALLY SIMPLE") to get 2 bytes integer from "index-data-decompressed.block"

great!
That's what most beginners lack in: to gain basic coding knowledges.
So java is a good start!

As for your asteroid; I've a deja vue but I can't really recall from my tired mind what that could mean.

So I used your index list to get some mesh (at no avail starting from 0x18800)
but seems I got the normals (because of the sphere).

prop_asteroid_01-cdr.JPG

[psxmodder]

i'm using a simple java app ("REALLY SIMPLE") to get 2 bytes integer from "index-data-decompressed.block"

great!
That's what most beginners lack in: to gain basic coding knowledges.
So java is a good start!

As for your asteroid; I've a deja vue but I can't really recall from my tired mind what that could mean.

So I used your index list to get some mesh (at no avail starting from 0x18800)
but seems I got the normals (because of the sphere).

prop_asteroid_01-cdr.JPG

Thats good but how many verts did you find with your tool?, it looks like they were a few right? or you get all the 726 verts?, well your tool have some alternative way to get the floats because in the ones that i did extract i have not found any 47.??? float but if that are the normals then in the header should be more info about it.

[shakotay2]

Thats good but how many verts did you find with your tool?

I entered 726 as vertex count. (Normally hex2obj offers the maximum face index (FI) as the vertex count but it can't handle edgecompressed FIs.)

to get the floats because in the ones that i did extract i have not found any 47.??? float

that's the reason why I've switched to ShortAll ('short' means 'signed int', 'All' means verts AND uvs, which isn't proven so far).

[psxmodder]

that's the reason why I've switched to ShortAll ('short' means 'signed int', 'All' means verts AND uvs, which isn't proven so far).

First you say that in our 0x10 or 16 bytes lines instead of v(x,y,z,w?) x 4b there is v(x,y,z),vn(x,y,z),vt(u,v) x 2b right?. THat make sense according to the .WDR format on PC already documented here : http://gta.wikia.com/wiki/WDR

Code: Select all

GRAPHICS section (Size found from the flags @ 0x08 of RSC file header)
---------------------------------------------------------------------------------------
VertexData (BlockSize VertexCount*0x24 or 0x34 when the Geometry is using a normal map)
---------------------------------------------------------------------------------------
0x00	4b	FLOAT	Position x
0x04	4b	FLOAT	Position y
0x08	4b	FLOAT	Position z
0x0C	4b	FLOAT	Normal x
0x10	4b	FLOAT	Normal y
0x14	4b	FLOAT	Normal z
0x18	4b	DWORD	Color (RGBA)
0x1C	4b	FLOAT	Texcoord u
0x20	4b	FLOAT	Texcoord v
0x24	4b	FLOAT	Tangent/Bi-Normal x   // Only when there is a normal map applied to the Geometry
0x28	4b	FLOAT	Tangent/Bi-Normal y   // Only when there is a normal map applied to the Geometry
0x2C	4b	FLOAT	Tangent/Bi-Normal z   // Only when there is a normal map applied to the Geometry
0x30	4b	FLOAT	Tangent/Bi-Normal w   // Only when there is a normal map 

Well obviusly on pc is maybe more suitable the use 4b arrays.

Another question how did you try to work with half floats convert? sorry if it is a dumb question

[shakotay2]

it's all wild guessing atm, the data might be compressed, shrinked, whatever. You could switch to HF_all in hex2obj but after pressing the go2 button you'll see that the result is not senseful:

asteroid_HF.JPG

v 339 94208 0.067566 or v 5436 0.006973 0.06427 for example are assumed to be wrong vertices.

btw: "shrinked" would mean x xn y yn z zn.
(That's a very wild guess, admitted but I've seen something similar for tx, ty in a PC 3D format, tx tx1 ty ty1.)

[psxmodder]

it's all wild guessing atm, the data might be compressed, shrinked, whatever. You could switch to HF_all in hex2obj but after pressing the go2 button you'll see that the result is not senseful:

asteroid_HF.JPG

v 339 94208 0.067566 or v 5436 0.006973 0.06427 for example are assumed to be wrong vertices.

btw: "shrinked" would mean x xn y yn z zn.
(That's a very wild guess, admitted but I've seen something similar for tx, ty in a PC 3D format, tx tx1 ty ty1.)

Humm but how did your program exactly get those normal verts ?, maybe is relevant to know what offsets etc. I don't see how they could shrink those bytes but maybe the function that is not properly working on the .py script for noesis can help us if we modify it in a better way here:

Code: Select all

bs.seek(vertexDataOffset, NOESEEK_ABS)
	#vertexSize = 0x10 * vertexCount
	print('Vertex Size: ' + hex(vertexDataSize))
	vertbuff = bs.readBytes(vertexDataSize)
	print('vertbuff Size: ' + hex(len(vertbuff)))
	rapi.rpgBindPositionBufferOfs(vertbuff, noesis.RPGEODATA_FLOAT, 0x10, 0)
	rapi.rpgSetName(str(i))
	#rapi.rpgCommitTriangles(None, noesis.RPGEODATA_USHORT, (vertexSize // 16), noesis.RPGEO_POINTS, 1)
	rapi.rpgCommitTriangles(edgeDecomp, noesis.RPGEODATA_USHORT,vertexCount, noesis.RPGEO_TRIANGLE, 1)
	mdl = rapi.rpgConstructModel()
	mdlList.append(mdl)
	f.close()
	rapi.rpgClearBufferBinds()	

Actually the code from "edgeDecomp = rapi.decompressEdgeIndices(edgeData, indexCount)" works great with the edges as we get those 726 face indexes but maybe there is another EDGE stuff related to our 0x2D60 bytes that contains our vertex data.

In the script the rpgCommitTriangles and rapi.rpgBindPositionBufferOfs(vertbuff, noesis.RPGEODATA_FLOAT, 0x10, 0) are not properly working as you see in .obj outputted by noesis export.

You know how we can check the output's from those functions without only writing with rpgConstructModel?. so We can check where is failling.

[shakotay2]

I don't see how they could shrink those bytes

that was an example only. If I knew what they did I would have told you.

Humm but how did your program exactly get those normal verts ?

It's nothing special, something like xpos = Highbyte*256 + Lowbyte ;
(big endian data)
To handle signed values I use if (xpos>32767.0) xpos -= 65536.0;
xpos /= 256.0 ;
(the divider might be different -> scaling)

but maybe there is another EDGE stuff related to our 0x2D60 bytes that contains our vertex data.

To my knowledge Sony's edge compression is used for face indices only (but I may be wrong).

You know how we can check the output's from those functions without only writing with rpgConstructModel?

I don't get what you mean.
Also rapi.rpgBindPositionBufferOfs(vertbuff, noesis.RPGEODATA_FLOAT, 0x10, 0) can NOT work properly when used on data that is NOT float.

[psxmodder]

It's ok but i think that the "short all" method must be the one used because you get the normals, how are you converting those 2 bytes in "short all" method? i was thinking that you were using hf kind of conversion but now i see that you were referring to another type.

[shakotay2]

It's ok but i think that the "short all" method must be the one used because you get the normals, how are you converting those 2 bytes in "short all" method?

like described in my post before

i was thinking that you were using hf kind of conversion but now i see that you were referring to another type.

didn't you understand my post as of Mon Jan 09, 2017 8:43 am? Read carefully, please.
I wrote that the result is not senseful when chosing HF_all. Because the data is not half floats, obviously.