WebGL ripping of a model from sketchfab

Post questions about game models here, or help out others!
Post Reply
fxcyrex
ultra-n00b
Posts: 7
Joined: Wed Oct 10, 2012 6:56 pm
Been thanked: 1 time

WebGL ripping of a model from sketchfab

Post by fxcyrex » Thu Aug 08, 2013 2:01 pm

I know it has been asked before, but I really need help with a model.

https://sketchfab.com/show/c3oFUCpyRK49ji7C9wKjktdrsls

I think this is the model

https://sketchfab.com/media/show-df4e51 ... e805b65.js

This model is saved in a js file, but it must be possible to get it to a usable format somehow. If someone with more knowledge than me are willing to take a look at it I would be forever grateful :)

User avatar
shakotay2
MEGAVETERAN
MEGAVETERAN
Posts: 2652
Joined: Fri Apr 20, 2012 9:24 am
Location: Nexus, searching for Jim Kirk
Has thanked: 653 times
Been thanked: 1360 times

Re: WebGL ripping of a model from sketchfab

Post by shakotay2 » Fri Aug 09, 2013 5:42 pm

fxcyrex wrote:I know it has been asked before
So why opening a new thread and not continue an existing one?
I think this is the model
https://sketchfab.com/media/show-df4e51 ... e805b65.js

This model is saved in a js file, but it must be possible to get it to a usable format somehow
You know the word function is contained 4482 times?

Why do you think the model data contained in here?
For me it's just a js graphics library.

I searched for model data in an array but couldn't find any.

btw: what about finale's python script? (viewtopic.php?f=16&t=8663&hilit=webgl&start=15 third post)

Did you use it? edit: didn't get it working

But this is an example how a json file would look like:

Code: Select all

{
	"objectName" : ["Plane"],
	"texturePath" : ["Material.001_40249_1557887544163_1"],
	"vertexPositions" : [-0.000001,0.007504,0.055036,-0.000001,0.007504,1.000000,0.000001,2.007504,-0.000000],
	"vertexTextureCoords" : [1.000000,0.000000,0.500000,1.000000,0.000000,0.000000],
	"indices" : [],
	"texIndices" : []
}
Bigchillghost, Reverse Engineering a Game Model: viewtopic.php?f=29&t=17889
extracting simple models: viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-jm9.zip
"You quoted the whole thing, what a mess."

finale00
M-M-M-Monster veteran
M-M-M-Monster veteran
Posts: 2387
Joined: Sat Apr 09, 2011 1:22 am
Has thanked: 170 times
Been thanked: 296 times

Re: WebGL ripping of a model from sketchfab

Post by finale00 » Sat Aug 10, 2013 4:52 pm

I don't think webGL has any particular specifications for how the input should be structured; rather, it provides the API for rendering a model and it's up to developers to pick how they want to store the data and feed it to the API.

The plugin I wrote was for that 3DCG website. They happened to store their data in a particular format for all of their submissions.
I imagine most websites that offer 3D rendering services convert models into their own internal format just for sanity purposes on the dev's end, so I would approach it by writing different plugins for different websites.

There is no actual "webgl" format after all, and I should rename the plugin linked up there to "webgl_3dcg.net"

Sketchfab's files are stored like

https://d3iahq3akd4kx2.cloudfront.net/u ... e.osgjs.gz

Code: Select all

{
  "Generator": "OpenSceneGraph 3.1.4", 
  "Version": 7, 
  "osg.Node": {
    "Children": [ {
      "osg.Node": {
        "UniqueID": 0, 
        "Name": "Small Crania.stl", 
        "Children": [ {
          "osg.Node": {
            "UniqueID": 1, 
            "Children": [ {
              "osg.Geometry": {
                "UniqueID": 2, 
                "PrimitiveSetList": [ {
                  "DrawArrays": {
                    "UniqueID": 4, 
                    "Count": 1689144, 
                    "First": 0, 
                    "Mode": "LINES"
                  }
                } ], 
                "VertexAttributeList": {
                  "Vertex": {
                    "UniqueID": 3, 
                    "Array": { 
                      "Float32Array": { 
                        "File": "model_wireframe.bin.gz",
                        "Size": 1689144,
                        "Offset": 0
                      }
                    }, 
                    "ItemSize": 3, 
                    "Type": "ARRAY_BUFFER"
                  }
                }
              }
            } ]
          }
        } ]
      }
    } ]
  }
}
And you can see that it references other files where the rest of the data is stored.

User avatar
zaykho
mega-veteran
mega-veteran
Posts: 218
Joined: Fri Dec 03, 2010 1:20 pm
Location: France
Has thanked: 143 times
Been thanked: 34 times

Re: WebGL ripping of a model from sketchfab

Post by zaykho » Sat Aug 10, 2013 10:01 pm

finale00 wrote:Sketchfab's files are stored like

https://d3iahq3akd4kx2.cloudfront.net/u ... e.osgjs.gz

Code: Select all

{
  "Generator": "OpenSceneGraph 3.1.4", 
  .......
And you can see that it references other files where the rest of the data is stored.
Well it depend of the format used by the user, obj, 3ds etc....
All models are named model_file and converted in their .bin format before being compressed by gzip.


Here the JavaScript root data where stored all information about the model:
https://sketchfab.com/v1/models/c3oFUCp ... wKjktdrsls

Here the JavaScript root data where stored all information about the 3D model for the plugin OpenSceneGraph:
https://d3iahq3akd4kx2.cloudfront.net/u ... e.osgjs.gz

And here, the 3d model converted to their .bin file format and compressed by gzip:
Be aware this file is heavy, be sure to have a good browser with good stability, do " File > Save As " to save it.
https://d3iahq3akd4kx2.cloudfront.net/u ... ile.bin.gz


I'm really surprised that they didn't use an obfuscator or a PHP sidehack to make the decrypt data a bit harder............

User avatar
shakotay2
MEGAVETERAN
MEGAVETERAN
Posts: 2652
Joined: Fri Apr 20, 2012 9:24 am
Location: Nexus, searching for Jim Kirk
Has thanked: 653 times
Been thanked: 1360 times

Re: WebGL ripping of a model from sketchfab

Post by shakotay2 » Sun Aug 11, 2013 11:46 am

zaykho wrote:And here, the 3d model converted to their .bin file format and compressed by gzip:
Thx!
And here the first submesh (of 13): Image
Bigchillghost, Reverse Engineering a Game Model: viewtopic.php?f=29&t=17889
extracting simple models: viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-jm9.zip
"You quoted the whole thing, what a mess."

User avatar
zaykho
mega-veteran
mega-veteran
Posts: 218
Joined: Fri Dec 03, 2010 1:20 pm
Location: France
Has thanked: 143 times
Been thanked: 34 times

Re: WebGL ripping of a model from sketchfab

Post by zaykho » Sun Aug 11, 2013 12:15 pm

shakotay2 wrote:
zaykho wrote:And here, the 3d model converted to their .bin file format and compressed by gzip:
Thx!
And here the first submesh (of 13): Image
Very Nice ! :)

fxcyrex
ultra-n00b
Posts: 7
Joined: Wed Oct 10, 2012 6:56 pm
Been thanked: 1 time

Re: WebGL ripping of a model from sketchfab

Post by fxcyrex » Tue Aug 13, 2013 6:50 am

shakotay2 wrote:
zaykho wrote:And here, the 3d model converted to their .bin file format and compressed by gzip:
Thx!
And here the first submesh (of 13): Image
Vert Nice. I wish I could do magic like you :)

User avatar
shakotay2
MEGAVETERAN
MEGAVETERAN
Posts: 2652
Joined: Fri Apr 20, 2012 9:24 am
Location: Nexus, searching for Jim Kirk
Has thanked: 653 times
Been thanked: 1360 times

Re: WebGL ripping of a model from sketchfab

Post by shakotay2 » Tue Aug 13, 2013 8:09 am

fxcyrex wrote:Vert Nice. I wish I could do magic like you :)
There's nothing being magic, no. It's totally simple.

If you're having a basic understanding of vertices and faces I can tell you how to get it.

I was missing the texture coordinates therefore I somewhat lost the interest.

This is a wavefrront obj of crania anatomia, first submesh: [link removed due to a demand of the copy right holders]

(I seldom create normals because they can be generated in blender automatically.)
Last edited by shakotay2 on Mon Nov 18, 2013 12:16 pm, edited 2 times in total.
Bigchillghost, Reverse Engineering a Game Model: viewtopic.php?f=29&t=17889
extracting simple models: viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-jm9.zip
"You quoted the whole thing, what a mess."

oamio
veteran
Posts: 100
Joined: Sun Mar 06, 2011 2:52 pm
Has thanked: 20 times
Been thanked: 8 times

Re: WebGL ripping of a model from sketchfab

Post by oamio » Tue Aug 13, 2013 8:12 am


User avatar
shakotay2
MEGAVETERAN
MEGAVETERAN
Posts: 2652
Joined: Fri Apr 20, 2012 9:24 am
Location: Nexus, searching for Jim Kirk
Has thanked: 653 times
Been thanked: 1360 times

Re: WebGL ripping of a model from sketchfab

Post by shakotay2 » Tue Aug 13, 2013 8:22 am

oamio wrote:http://www.3dcg-arts.net/art/975[...]
do u know how to rip these webgl format?
Sorry I can't read the language on that site.

If you could give me informations like finale00 and zaykho I would give it a try.
I need this (triangles, vertices, offsets)

Code: Select all

{
  "Generator": "OpenSceneGraph 3.1.4", 
  "Version": 7, 
  "osg.Node": {
    "Children": [ {
      "osg.Node": {
        "UniqueID": 0, 
        "Name": "Small Crania.stl.resolve", 
        "Children": [ {
          "osg.Node": {
            "UniqueID": 1, 
            "Children": [ {
              "osg.Geometry": {
                "UniqueID": 2, 
                "PrimitiveSetList": [ {
                  "DrawElementsUShort": {
                    "UniqueID": 5, 
                    "Indices": {
                      "Array": { 
                        "Uint16Array": { 
                          "File": "model_file.bin.gz",
                          "Size": 65535,
                          "Offset": 0
                        }
                      }, 
                      "ItemSize": 1, 
                      "Type": "ELEMENT_ARRAY_BUFFER"
                    }, 
                    "Mode": "TRIANGLES"
                  }
                } ], 
and the model_file.bin.gz.
Bigchillghost, Reverse Engineering a Game Model: viewtopic.php?f=29&t=17889
extracting simple models: viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-jm9.zip
"You quoted the whole thing, what a mess."

oamio
veteran
Posts: 100
Joined: Sun Mar 06, 2011 2:52 pm
Has thanked: 20 times
Been thanked: 8 times

Re: WebGL ripping of a model from sketchfab

Post by oamio » Tue Aug 13, 2013 8:45 am

http://www.3dcg-arts.net/api/model/get/three_format/975

You can get api code like that
but I dont know how to get gz files

fxcyrex
ultra-n00b
Posts: 7
Joined: Wed Oct 10, 2012 6:56 pm
Been thanked: 1 time

Re: WebGL ripping of a model from sketchfab

Post by fxcyrex » Tue Aug 13, 2013 9:57 am

shakotay2 wrote:
fxcyrex wrote:Vert Nice. I wish I could do magic like you :)
There's nothing being magic, no. It's totally simple.

If you're having a basic understanding of vertices and faces I can tell you how to get it.

I was missing the texture coordinates therefore I somewhat lost the interest.

This is a wavefrront obj of crania anatomia, first submesh: http://www.uploadmb.com/dw.php?id=1376377338

(I seldom create normals because they can be generated in blender automatically.)
I have some programming experience, but not with 3D. If you could try to explain to me how to get the rest of the submeshes I'll try to do it myself. But I don't know where to start and what tools I need.

User avatar
shakotay2
MEGAVETERAN
MEGAVETERAN
Posts: 2652
Joined: Fri Apr 20, 2012 9:24 am
Location: Nexus, searching for Jim Kirk
Has thanked: 653 times
Been thanked: 1360 times

Re: WebGL ripping of a model from sketchfab

Post by shakotay2 » Tue Aug 13, 2013 10:41 am

oamio wrote:http://www.3dcg-arts.net/api/model/get/three_format/975

You can get api code like that
but I dont know how to get gz files
Thx; Seems the required being contained in your linked file.
I will have a look soon.

edit: very soon:
Image

(Some faces wrong.)
Last edited by shakotay2 on Tue Aug 13, 2013 12:52 pm, edited 1 time in total.
Bigchillghost, Reverse Engineering a Game Model: viewtopic.php?f=29&t=17889
extracting simple models: viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-jm9.zip
"You quoted the whole thing, what a mess."

User avatar
shakotay2
MEGAVETERAN
MEGAVETERAN
Posts: 2652
Joined: Fri Apr 20, 2012 9:24 am
Location: Nexus, searching for Jim Kirk
Has thanked: 653 times
Been thanked: 1360 times

Re: WebGL ripping of a model from sketchfab

Post by shakotay2 » Tue Aug 13, 2013 10:47 am

edit: be informed that this crash course is obsolete because most sketchfab models seem to use varint encoding now
fxcyrex wrote:I have some programming experience, but not with 3D.
Well, then it's gonna going to become a little bit harder.
But I can give you an overview only - understanding vertices and faces may take some weeks.

This is a crash course where I show how to convert the (uncompressed) model_file.bin data into a wavefront obj model (see attached in my third post).

You'll need a hex editor for viewing the bin file and a coding language
to create the obj file (in ASCII format).

U need to uncompress the file.osgjs.gz
In it search for "osg.Geometry" as a start of a submesh.
(I'm dealing with the first submesh only.)

Code: Select all

                          "Size": 65535,
                          "Offset": 0
                    "Mode": "TRIANGLES"
face indices (Triangles block)
One face consists of three face indices (2 bytes)
There are 65536 of them so we get 21845 faces
Starting from file offset 0:

Code: Select all

Offset   0  1  2  3  4  5  6  7   8  9  A  B  C  D  E  F
00000   00 00 01 00 02 00 03 00  04 00 05 00 06 00 07 00 08 00
You have to add one to the faceindex for wavefront obj format

f 1 2 3
f 4 5 6
f 7 8 9
...
end of faces block at 0x 1FFFD, last face f 65533 65534 65535

Now have a look at the vertices:

Code: Select all

                  "Vertex": {
                    "UniqueID": 3, 
                    "Array": { 
                      "Float32Array": { 
                        "File": "model_file.bin.gz",
                        "Size": 65536,
                        "Offset": 917504
                      }
                    }, 
there are 65536 vertices (each vertex having 3 floats x,y,z, each float consisting of 4 hex bytes)
(65536 * 12 bytes = 0xC0000, 0xE0000 + 0xC0000 = 0x1A0000)

(you'll need floating point format IEEE-754 for converting 4 hex bytes to a floating point value)

starting from file offset 917504 = 0xE0000

Code: Select all

Offset  0  1  2  3  4  5  6  7   8  9  A  B

0E0000 FA CB A6 3F 60 32 D5 3D  9F AB 3D 3F
converted to
v 1.303100 0.104100 0.740900

end of vertex block at 1A0000
last vertex: v 1.941700 1.668900 3.312900

That's all. Save the data lines (faces and vertices) into an obj file
and import it into blender or any other 3D modelling tool supporting wavefront obj.

And yes: the texture data is missing.
Last edited by shakotay2 on Sun Aug 30, 2015 7:46 am, edited 1 time in total.
Bigchillghost, Reverse Engineering a Game Model: viewtopic.php?f=29&t=17889
extracting simple models: viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-jm9.zip
"You quoted the whole thing, what a mess."

fxcyrex
ultra-n00b
Posts: 7
Joined: Wed Oct 10, 2012 6:56 pm
Been thanked: 1 time

Re: WebGL ripping of a model from sketchfab

Post by fxcyrex » Tue Aug 13, 2013 11:24 am

I'm not interested in the textures so thanks for the info. Will try to do the rest of the work myself :)

Post Reply