WebGL ripping of a model from sketchfab

Post questions about game models here, or help out others!
episoder
mega-veteran
mega-veteran
Posts: 186
Joined: Fri Oct 16, 2015 8:05 pm
Has thanked: 3 times
Been thanked: 80 times

Re: WebGL ripping of a model from sketchfab

Post by episoder » Fri Feb 08, 2019 4:24 pm

well. seems sketchfab introduced a 'fixed' format against exactly this extraction script. very recent. not to say, i pinned it around 4-6 days ago. on a monday? makes sense. now, somebody will have to dig thru the russian code and figure where to skip the trash (out of range) index data they added, somewhere. i don't even know. i haven't digged in the binaries (yet?). do i want to? ugh. this encoded stuff is so hard to read. it's all trial and error depack debug. i shouldn't spend time on this. :|

User avatar
M0fD
mega-veteran
mega-veteran
Posts: 173
Joined: Sat Nov 01, 2014 10:05 pm
Location: Les Gobelins
Has thanked: 17 times
Been thanked: 26 times

Re: WebGL ripping of a model from sketchfab

Post by M0fD » Sun Feb 10, 2019 1:14 am

episoder wrote:well. seems sketchfab introduced a 'fixed' format against exactly this extraction script. very recent. not to say, i pinned it around 4-6 days ago. on a monday? makes sense. now, somebody will have to dig thru the russian code and figure where to skip the trash (out of range) index data they added, somewhere. i don't even know. i haven't digged in the binaries (yet?). do i want to? ugh. this encoded stuff is so hard to read. it's all trial and error depack debug. i shouldn't spend time on this. :|
Awww, if only someone had seen it coming and someone had heard it, what a shame.

I vaguely can remember someone saying that's it's impossible for Sketchfab to fix the ripping method so it's ok to share it.

A bad memory I have, or am I ?
1. Ninja Ripper doesn't work with Sketchfab

2. Don't distribute the extraction method

3. Three models only

4. No buyable models


ImageImage

episoder
mega-veteran
mega-veteran
Posts: 186
Joined: Fri Oct 16, 2015 8:05 pm
Has thanked: 3 times
Been thanked: 80 times

Re: WebGL ripping of a model from sketchfab

Post by episoder » Sun Feb 10, 2019 1:33 am

M0fD wrote:Awww, if only episoder had seen it coming and episoder had heard it, what a shame.

I vaguely can remember episoder saying that's it's impossible for Sketchfab to fix the ripping method so it's ok to share it.

A bad memory I have, or am I ?
you really tryna do that (f)lame? i never 'said' it's impossible to fix it. i even thought and 'mentioned' how to xchange and fix this format myself. hehe. and... i started tryna recode the importer today. it will take a while tho. i'm not familiar with blender python. gotta learn all of it on the way.

what you doing? :lol:

User avatar
M0fD
mega-veteran
mega-veteran
Posts: 173
Joined: Sat Nov 01, 2014 10:05 pm
Location: Les Gobelins
Has thanked: 17 times
Been thanked: 26 times

Re: WebGL ripping of a model from sketchfab

Post by M0fD » Sun Feb 10, 2019 1:42 am

What I'm doing ? Guess what, I'm searching for a solution too lol

Btw, the blue one was me so it's "Awww, if only M0fD had seen it coming and episoder had heard it, what a shame.

I vaguely can remember episoder saying that's it's impossible for Sketchfab to fix the ripping method so it's ok to share it."

But I didn't mention you in the first place :lol:
1. Ninja Ripper doesn't work with Sketchfab

2. Don't distribute the extraction method

3. Three models only

4. No buyable models


ImageImage

episoder
mega-veteran
mega-veteran
Posts: 186
Joined: Fri Oct 16, 2015 8:05 pm
Has thanked: 3 times
Been thanked: 80 times

Re: WebGL ripping of a model from sketchfab

Post by episoder » Sun Feb 10, 2019 2:21 am

you just searching?... okay. huh. when you start to work on things let us know. can you assist coding a uptodate importer? if not... just stfu, thx. sorry or maybe not. :]

User avatar
M0fD
mega-veteran
mega-veteran
Posts: 173
Joined: Sat Nov 01, 2014 10:05 pm
Location: Les Gobelins
Has thanked: 17 times
Been thanked: 26 times

Re: WebGL ripping of a model from sketchfab

Post by M0fD » Sun Feb 10, 2019 2:43 am

So mean as always I see.
When I mean searching it's meaning I'm working on it. Dont just take things like that XD
If some people want to work together to solve this problem, I don't mind to join them, if not that's fine I can deal whit it myself.

So, like you said, stfu too.
1. Ninja Ripper doesn't work with Sketchfab

2. Don't distribute the extraction method

3. Three models only

4. No buyable models


ImageImage

Szkaradek123
mega-veteran
mega-veteran
Posts: 297
Joined: Wed May 05, 2010 8:21 pm
Location: Poland Głogów
Has thanked: 21 times
Been thanked: 582 times

Re: WebGL ripping of a model from sketchfab

Post by Szkaradek123 » Sun Feb 10, 2019 8:34 am

the code lacks one unknow condition to skip indices decoding
Replace this code for models only with indices problem

Code: Select all

def getIndices(itemsize,size,offset,type,g,mode,magic):
	if type!="Uint8Array":bytes=decodeVarint(g,offset,size*itemsize,type)
	else:
		g.seek(offset)
		bytes=list(g.B(size*itemsize))		
	write(log,[magic,mode,type],0)
	write(log,bytes,0)		
	
	IMPLICIT_HEADER_LENGTH=3
	IMPLICIT_HEADER_MASK_LENGTH=1
	IMPLICIT_HEADER_PRIMITIVE_LENGTH=0
	IMPLICIT_HEADER_EXPECTED_INDEX=2
	highWatermark=2
	
	MissingCondition=None
	if MissingCondition!=None:
		
		if mode=='"TRIANGLE_STRIP"':
					k=IMPLICIT_HEADER_LENGTH+bytes[IMPLICIT_HEADER_MASK_LENGTH]
					bytes=decodeDelta(bytes,k)	
					#write(log,[magic,k],0)	
					#write(log,bytes,0)		
					bytes=decodeImplicit(bytes,k)
					#write(log,[magic,k],0)	
					#write(log,bytes,0)			
					i=[magic]	
					bytes,magic=decodeWatermark(bytes,bytes,i)
					#write(log,[magic],0)	
					#write(log,bytes,0)	
				
		elif mode=='"TRIANGLES"':
					k=0
					bytes=decodeDelta(bytes,k)
					#write(log,[magic],0)	
					#write(log,bytes,0)			
					i=[magic]	
					bytes,magic=decodeWatermark(bytes,bytes,i)
					#write(log,[magic],0)	
					#write(log,bytes,0)	
		
		
		
		
	return magic,bytes
	
edit
Propably it depends on vertex encoding .
If vertices are not varint encoding we can skip indices decoding, MissingCondition=None.

If vertices are varint encoding we can not skip indices decoding, MissingCondition=1 (not None)
You do not have the required permissions to view the files attached to this post.
Last edited by Szkaradek123 on Wed Feb 13, 2019 5:56 pm, edited 1 time in total.

User avatar
M0fD
mega-veteran
mega-veteran
Posts: 173
Joined: Sat Nov 01, 2014 10:05 pm
Location: Les Gobelins
Has thanked: 17 times
Been thanked: 26 times

Re: WebGL ripping of a model from sketchfab

Post by M0fD » Sun Feb 10, 2019 2:25 pm

Line 314 of the script if people ask.
1. Ninja Ripper doesn't work with Sketchfab

2. Don't distribute the extraction method

3. Three models only

4. No buyable models


ImageImage

foxus
ultra-n00b
Posts: 6
Joined: Thu Jan 24, 2019 7:03 am

Re: WebGL ripping of a model from sketchfab

Post by foxus » Sun Feb 10, 2019 10:52 pm

any tip on how to decompress model_file.bin.gz and model_file_wireframe.bin.gz?
i have tried winrar, winzip, and some other shady tools but none seems to work, not even gunzip in ubuntu

Image
Image

or, if i just remove the gz file extension, blender wont load the model...

Image

pls help

User avatar
M0fD
mega-veteran
mega-veteran
Posts: 173
Joined: Sat Nov 01, 2014 10:05 pm
Location: Les Gobelins
Has thanked: 17 times
Been thanked: 26 times

Re: WebGL ripping of a model from sketchfab

Post by M0fD » Sun Feb 10, 2019 11:50 pm

Try this

Code: Select all

import os
import re
import json
import requests
from urllib.parse import urlparse
from html import unescape
from bs4 import BeautifulSoup

SCRIPT_VERSION = '1.0'

HEADERS = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.25 Safari/537.36',
    'accept-encoding': 'gzip, deflate, br',
    'accept-language': 'zh-CN,zh-TW;q=0.8,zh;q=0.6,en;q=0.4,ja;q=0.2',
    'cache-control': 'max-age=0'
}

def main():
    url = input('input url:')
    parse(url)

def parse(url):
    try:
        print('Parsing...')
        page = requests.get(url, headers=HEADERS, timeout=10).text
        soup = BeautifulSoup(page, 'html.parser')

        modelId = urlparse(url).path.split('/')[2]
        data = unescape(soup.find(id='js-dom-data-prefetched-data').string)
        data = json.loads(data)
        name = validateTitle(data['/i/models/'+modelId]['name'])
        thumbnailData = data['/i/models/'+modelId]['thumbnails']['images']
        thumbnail = getBiggestImage(thumbnailData)
        osgjsUrl = data['/i/models/'+modelId]['files'][0]['osgjsUrl']
        modelFile = osgjsUrl.replace('file.osgjs.gz', 'model_file.bin.gz')
        texturesData = data['/i/models/'+modelId+'/textures?optimized=1']['results']
        textures = []

        print('Model Id:', modelId)
        print('Name:', name)
        print('Thumbnail URL:',thumbnail)
        print('osgjs URL:', osgjsUrl)
        print('Model File:', modelFile)
        print('Textures:', len(texturesData))

        download(thumbnail, os.path.join(name, 'thumbnail.jpg'))
        download(osgjsUrl, os.path.join(name, 'file.osgjs'))
        download(modelFile, os.path.join(name, 'model_file.bin.gz'))

        for texture in texturesData:
            textureUrl = getBiggestImage(texture['images'])
            download(textureUrl, os.path.join(name, 'texture', validateTitle(texture['name'])))

    except AttributeError:
        raise
        return False

def getBiggestImage(images):
    size = 0
    for img in images:
        if img['size'] != None and img['size'] > size:
            size = img['size']
            imgUrl = img['url']
    return imgUrl

def validateTitle(title):
    pattern = r'[\\/:*?"<>|\r\n]+'
    newTitle = re.sub(pattern, "_", title)
    return newTitle

def download(url, filename):
    print('Downloading:', filename)
    try:
        os.makedirs(os.path.dirname(filename), exist_ok=True)
        if os.path.exists(filename):
            if os.path.getsize(filename) > 0:
                print('file exists.')
            else:
                with open(filename, 'wb') as file:
                    file.write(requests.get(url, headers=HEADERS, timeout=30).content)
        else:
            with open(filename, 'wb') as file:
                file.write(requests.get(url, headers=HEADERS, timeout=30).content)
    except Exception:
        pass

if __name__ == '__main__':
    main()
If it doesn't work, use Firefox (it can open the files in a text format then right click and download then it's extractable).
1. Ninja Ripper doesn't work with Sketchfab

2. Don't distribute the extraction method

3. Three models only

4. No buyable models


ImageImage

foxus
ultra-n00b
Posts: 6
Joined: Thu Jan 24, 2019 7:03 am

Re: WebGL ripping of a model from sketchfab

Post by foxus » Mon Feb 11, 2019 12:53 am

M0fD wrote:Try this

Code: Select all

import os
import re
import json
import requests
from urllib.parse import urlparse
from html import unescape
from bs4 import BeautifulSoup

SCRIPT_VERSION = '1.0'

HEADERS = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.25 Safari/537.36',
    'accept-encoding': 'gzip, deflate, br',
    'accept-language': 'zh-CN,zh-TW;q=0.8,zh;q=0.6,en;q=0.4,ja;q=0.2',
    'cache-control': 'max-age=0'
}

def main():
    url = input('input url:')
    parse(url)

def parse(url):
    try:
        print('Parsing...')
        page = requests.get(url, headers=HEADERS, timeout=10).text
        soup = BeautifulSoup(page, 'html.parser')

        modelId = urlparse(url).path.split('/')[2]
        data = unescape(soup.find(id='js-dom-data-prefetched-data').string)
        data = json.loads(data)
        name = validateTitle(data['/i/models/'+modelId]['name'])
        thumbnailData = data['/i/models/'+modelId]['thumbnails']['images']
        thumbnail = getBiggestImage(thumbnailData)
        osgjsUrl = data['/i/models/'+modelId]['files'][0]['osgjsUrl']
        modelFile = osgjsUrl.replace('file.osgjs.gz', 'model_file.bin.gz')
        texturesData = data['/i/models/'+modelId+'/textures?optimized=1']['results']
        textures = []

        print('Model Id:', modelId)
        print('Name:', name)
        print('Thumbnail URL:',thumbnail)
        print('osgjs URL:', osgjsUrl)
        print('Model File:', modelFile)
        print('Textures:', len(texturesData))

        download(thumbnail, os.path.join(name, 'thumbnail.jpg'))
        download(osgjsUrl, os.path.join(name, 'file.osgjs'))
        download(modelFile, os.path.join(name, 'model_file.bin.gz'))

        for texture in texturesData:
            textureUrl = getBiggestImage(texture['images'])
            download(textureUrl, os.path.join(name, 'texture', validateTitle(texture['name'])))

    except AttributeError:
        raise
        return False

def getBiggestImage(images):
    size = 0
    for img in images:
        if img['size'] != None and img['size'] > size:
            size = img['size']
            imgUrl = img['url']
    return imgUrl

def validateTitle(title):
    pattern = r'[\\/:*?"<>|\r\n]+'
    newTitle = re.sub(pattern, "_", title)
    return newTitle

def download(url, filename):
    print('Downloading:', filename)
    try:
        os.makedirs(os.path.dirname(filename), exist_ok=True)
        if os.path.exists(filename):
            if os.path.getsize(filename) > 0:
                print('file exists.')
            else:
                with open(filename, 'wb') as file:
                    file.write(requests.get(url, headers=HEADERS, timeout=30).content)
        else:
            with open(filename, 'wb') as file:
                file.write(requests.get(url, headers=HEADERS, timeout=30).content)
    except Exception:
        pass

if __name__ == '__main__':
    main()
If it doesn't work, use Firefox (it can open the files in a text format then right click and download then it's extractable).
not sure how to use this, is a javascript code that i need to run using a scrpiter or do i need to append this to the blender importer script?

User avatar
M0fD
mega-veteran
mega-veteran
Posts: 173
Joined: Sat Nov 01, 2014 10:05 pm
Location: Les Gobelins
Has thanked: 17 times
Been thanked: 26 times

Re: WebGL ripping of a model from sketchfab

Post by M0fD » Mon Feb 11, 2019 1:07 am

Run it in the webpage console of the model.
1. Ninja Ripper doesn't work with Sketchfab

2. Don't distribute the extraction method

3. Three models only

4. No buyable models


ImageImage

foxus
ultra-n00b
Posts: 6
Joined: Thu Jan 24, 2019 7:03 am

Re: WebGL ripping of a model from sketchfab

Post by foxus » Mon Feb 11, 2019 1:50 am

M0fD wrote:Run it in the webpage console of the model.
import json fails, Blender reports it is a python script error. console says import error, no module named json. Do i need to tweak blender? i read it is supposed to run as the xtrctor comes with its own python installation :(

i tried this on blender in the text editor, the same it is used to run the xtrctor

plus the only file you can open with firefox is file.osgjs. not the other two, when you open those files in a new tab, they automatically download and they are not plain files. Im doing this on firefox lattest version btw

Image

Magowicc
ultra-n00b
Posts: 2
Joined: Fri Feb 08, 2019 8:34 am

Re: WebGL ripping of a model from sketchfab

Post by Magowicc » Mon Feb 11, 2019 4:31 am

Szkaradek123 wrote:the code lacks one unknow condition to skip indices decoding
Replace this code for models only with indices problem

Code: Select all

def getIndices(itemsize,size,offset,type,g,mode,magic):
	if type!="Uint8Array":bytes=decodeVarint(g,offset,size*itemsize,type)
	else:
		g.seek(offset)
		bytes=list(g.B(size*itemsize))		
	write(log,[magic,mode,type],0)
	write(log,bytes,0)		
	
	IMPLICIT_HEADER_LENGTH=3
	IMPLICIT_HEADER_MASK_LENGTH=1
	IMPLICIT_HEADER_PRIMITIVE_LENGTH=0
	IMPLICIT_HEADER_EXPECTED_INDEX=2
	highWatermark=2
	
	MissingCondition=None
	if MissingCondition!=None:
		
		if mode=='"TRIANGLE_STRIP"':
					k=IMPLICIT_HEADER_LENGTH+bytes[IMPLICIT_HEADER_MASK_LENGTH]
					bytes=decodeDelta(bytes,k)	
					#write(log,[magic,k],0)	
					#write(log,bytes,0)		
					bytes=decodeImplicit(bytes,k)
					#write(log,[magic,k],0)	
					#write(log,bytes,0)			
					i=[magic]	
					bytes,magic=decodeWatermark(bytes,bytes,i)
					#write(log,[magic],0)	
					#write(log,bytes,0)	
				
		elif mode=='"TRIANGLES"':
					k=0
					bytes=decodeDelta(bytes,k)
					#write(log,[magic],0)	
					#write(log,bytes,0)			
					i=[magic]	
					bytes,magic=decodeWatermark(bytes,bytes,i)
					#write(log,[magic],0)	
					#write(log,bytes,0)	
		
		
		
		
	return magic,bytes
	

Man, you are a genius!!! Thank you so much!!

User avatar
M0fD
mega-veteran
mega-veteran
Posts: 173
Joined: Sat Nov 01, 2014 10:05 pm
Location: Les Gobelins
Has thanked: 17 times
Been thanked: 26 times

Re: WebGL ripping of a model from sketchfab

Post by M0fD » Mon Feb 11, 2019 2:48 pm

foxus wrote:
M0fD wrote:Run it in the webpage console of the model.
import json fails, Blender reports it is a python script error. console says import error, no module named json. Do i need to tweak blender? i read it is supposed to run as the xtrctor comes with its own python installation :(

i tried this on blender in the text editor, the same it is used to run the xtrctor

plus the only file you can open with firefox is file.osgjs. not the other two, when you open those files in a new tab, they automatically download and they are not plain files. Im doing this on firefox lattest version btw

Image
Not a blender script, it's a chrome/firefox script lol
1. Ninja Ripper doesn't work with Sketchfab

2. Don't distribute the extraction method

3. Three models only

4. No buyable models


ImageImage

Post Reply