Offtopic: Into Commodore 64 (6502) coding, pixeling or music?
Xentax is looking for new members for the C64 activities!
Just drop us a message at forum@xentax.com and join the Scene Team!

Forum rules: Click here

Unity Assets Explorer [v 1.2]

The Original Forum. Game archives, full of resources. How to open them? Get help here.
User avatar
ikskoks
mega-veteran
mega-veteran
Posts: 299
Joined: Thu Jul 26, 2012 5:06 pm
Location: Poland, Łódź
Has thanked: 385 times
Been thanked: 92 times
Contact:

Re: Unity Assets Explorer [v 1.2]

Post by ikskoks » Sat Jun 29, 2013 4:30 pm

I'm working on an Unity asset extractor by myself
Nice. I'm keeping my fingers crossed ^^

User avatar
Chipicao
ultra-veteran
ultra-veteran
Posts: 476
Joined: Thu Feb 03, 2011 11:18 am
Has thanked: 42 times
Been thanked: 303 times
Contact:

Re: Unity Assets Explorer [v 1.2]

Post by Chipicao » Sat Jun 29, 2013 4:51 pm

barracuda wrote:Which Unity engine version was this tool written for? I tried to load asset files for 2.0, 3.3, 3.4, 3.5 and 4.1, but none of them are read correctly.

I'm working on an Unity asset extractor by myself and I identified at least three completely different asset file formats. So far, it can read files for 3.5 and 4.1. I wonder if there has been any research on the older formats?
I don't mean to be rude, but an asset extractor is pretty much pointless.

First of all, depending on the version and game platform, you'll have different data structures for meshes, materials, textures or other assets.
Second, all assets are linked together via their indices from the preload table (which I assume you're using to get offsets and sizes for "files"). Furthermore, assets can be linked across multiple .asset files too.

By extracting assets as individual files, you lose all these pieces of information, making it hard to import 3D meshes and impossible to link them to materials, transformation matrices and scene hierarchy.
In order to import assets properly, you must keep them as they are in .asset files.

I have been working for some time on a maxscript for importing unity games, which you can download here. For now it's fully compatible with Web and iOS games made with Unity 3.3 and 4.0.1 (possibly some older versions too).
I'm looking to test and add compatibility for other versions and platforms. Would you mind sending me some of those games you tested?
Please post any requests or issues with my tools in the appropriate topics.
I'm sorry if I don't reply or if I ignore PMs. My time is very limited.

barracuda
beginner
Posts: 30
Joined: Sat Nov 24, 2012 2:15 pm
Has thanked: 3 times
Been thanked: 27 times

Re: Unity Assets Explorer [v 1.2]

Post by barracuda » Sat Jun 29, 2013 6:42 pm

Chipicao wrote: I don't mean to be rude, but an asset extractor is pretty much pointless.

First of all, depending on the version and game platform, you'll have different data structures for meshes, materials, textures or other assets.
I'm aware of that. But while the surrounding asset file format is indeed pretty variable, the raw serialized object data seems to be very similar across versions. And asset files for Unity 3.5 even contain information about their own data structure, which makes it pretty simple to deserialize them.
Chipicao wrote: Second, all assets are linked together via their indices from the preload table (which I assume you're using to get offsets and sizes for "files"). Furthermore, assets can be linked across multiple .asset files too.

By extracting assets as individual files, you lose all these pieces of information, making it hard to import 3D meshes and impossible to link them to materials, transformation matrices and scene hierarchy.
In order to import assets properly, you must keep them as they are in .asset files.
My tool currently supports Texture2D, AudioClip, Font and Shader, which store their data in single unity objects with no dependencies. It also has a mode to export the raw serialized object data.

A SkinnedMeshRenderer, for example, is indeed much more complex. But I'm currently focused on the content that is relatively easy to extract or inaccessible from within the Unity editor.
Chipicao wrote: I have been working for some time on a maxscript for importing unity games, which you can download here. For now it's fully compatible with Web and iOS games made with Unity 3.3 and 4.0.1 (possibly some older versions too).
I already tried to view it before, but the thread link doesn't seem to work as a guest.
Chipicao wrote: I'm looking to test and add compatibility for other versions and platforms. Would you mind sending me some of those games you tested?
Well, I found some free UnityWeb games with Google.

2.x.x:
http://www.idancerecords.com/games/MoonDragon.unity3d

3.3.x/3.4.x:
http://unity3d.com/gallery/demos/live-demos

3.5.x:
http://85playgames.eval.hwcdn.net/games ... un.unity3d
http://85playgames.eval.hwcdn.net/games ... ay.unity3d
https://w8f8a7t9.ssl.hwcdn.net/dragons/ ... 32.unity3d

4.1.x:
http://media.jumpstart.com/JSW3DUnity/JSMain.unity3d
http://media.mathblaster.com/MBUnity/MBMain.unity3d

These files are just LZMA containers with some metadata in the header, btw. They can be extracted with Unity 3D Obfuscator, my tool will support that as well.

User avatar
Chipicao
ultra-veteran
ultra-veteran
Posts: 476
Joined: Thu Feb 03, 2011 11:18 am
Has thanked: 42 times
Been thanked: 303 times
Contact:

Re: Unity Assets Explorer [v 1.2]

Post by Chipicao » Sat Jun 29, 2013 7:20 pm

barracuda wrote: I'm aware of that. But while the surrounding asset file format is indeed pretty variable, the raw serialized object data seems to be very similar across versions. And asset files for Unity 3.5 even contain information about their own data structure, which makes it pretty simple to deserialize them.
Sorry, but that's not true at all. On the few versions I've tested, I have seen differences in the raw data for the following asset types: Mesh, Texture2D, Renderer, SkinnedMeshRenderer. There are probably others too.
You may not have noticed it because from what I see you're only testing Web games. Try an iOS or PC game and you'll see what I mean.

barracuda wrote:My tool currently supports Texture2D, AudioClip, Font and Shader, which store their data in single unity objects with no dependencies. It also has a mode to export the raw serialized object data.

A SkinnedMeshRenderer, for example, is indeed much more complex. But I'm currently focused on the content that is relatively easy to extract or inaccessible from within the Unity editor.
What you say is true, but that's pretty much as far as you're gonna get regarding independent assets.
Let me give you an example on how a 3D object would be properly loaded:

                    / Transform -> parent-Transform -> parent-GameObject
GameObject - MeshFilter -> Mesh
                    \ Renderer -> Material -> Textures

If you're planning to extract only textures/sounds then it doesn't matter. But otherwise you'll be losing any means of linking assets together.

barracuda wrote:I already tried to view it before, but the thread link doesn't seem to work as a guest.
Yeah you just have to register. My tool also extracts 12 types of textures, including PVR :)
Thanks, I'm gonna have a look. They all look like Web games, which would explain why they don't work with this Explorer.
P.S. You don't need that obfuscator tool, use aluigi's unity3d extraction script ;)
Please post any requests or issues with my tools in the appropriate topics.
I'm sorry if I don't reply or if I ignore PMs. My time is very limited.

barracuda
beginner
Posts: 30
Joined: Sat Nov 24, 2012 2:15 pm
Has thanked: 3 times
Been thanked: 27 times

Re: Unity Assets Explorer [v 1.2]

Post by barracuda » Sat Jun 29, 2013 8:51 pm

Chipicao wrote:Sorry, but that's not true at all. On the few versions I've tested, I have seen differences in the raw data for the following asset types: Mesh, Texture2D, Renderer, SkinnedMeshRenderer. There are probably others too.
You may not have noticed it because from what I see you're only testing Web games. Try an iOS or PC game and you'll see what I mean.
Well yes, I mostly tested some web games. But I also tested two PC standalone games without problems. Even if there are differences, it doesn't have to mean that they can't be figured out, right? 100% support for all Unity games that have ever been developed is a highly unrealistic target anyway. :wink:

But I'd like to test a mobile Unity game... do you have an example? Preferably not from official app stores.
Chipicao wrote:Yeah you just have to register. My tool also extracts 12 types of textures, including PVR :)
Ah, mine only supports 7 right now. These DDS color bitmasks for exotic RGB formats are a bit tricky. :D
Do you maybe also know how to fix the vertical flip and convert AG to RGB normal maps?
Chipicao wrote:Thanks, I'm gonna have a look. They all look like Web games, which would explain why they don't work with this Explorer.
P.S. You don't need that obfuscator tool, use aluigi's unity3d extraction script ;)
That script could've saved me some time to figure out the archive format, but it somehow slipped through my Google research. :(

User avatar
Chipicao
ultra-veteran
ultra-veteran
Posts: 476
Joined: Thu Feb 03, 2011 11:18 am
Has thanked: 42 times
Been thanked: 303 times
Contact:

Re: Unity Assets Explorer [v 1.2]

Post by Chipicao » Sat Jun 29, 2013 10:13 pm

barracuda wrote:Even if there are differences, it doesn't have to mean that they can't be figured out, right? 100% support for all Unity games that have ever been developed is a highly unrealistic target anyway. :wink:
It's not about figuring out the differences, it's about knowing when there are differences. And the key to that are the file version and file "type", both of which would be discarded after extraction.
100% support for all Unity games is not unrealistic, it's my goal ;) Ok, maybe except xbox and PS3. Those use big endian data and would be better off with a separate script.
barracuda wrote:But I'd like to test a mobile Unity game... do you have an example? Preferably not from official app stores.
Fast & Furious 6: The game would be one example. Or CSR Racing and Drift Mania. You can find them at your local "pirate shop" :D
But here's a free app from Opel promoting one of their cars: https://mega.co.nz/#!2F9VRByY!HPu2GjBsj ... qmp9IxQlmM

barracuda wrote:These DDS color bitmasks for exotic RGB formats are a bit tricky. :D
Do you maybe also know how to fix the vertical flip and convert AG to RGB normal maps?
Yeah, I've been through hell figuring out which texture type is ARGB, which is RGBA or BGRA and so on. But in the end I didn't use custom bitmasks because for some reason the nvidia plugin for PS ignores them. So for the sake of compatibility I just switch image bytes instead :D
There's nothing to fix, they are supposed to be flipped and they look as they should once applied to the models.

What do you mean by "AG normal maps", Alpha-Green? Maybe you just have the wrong channel order...
Please post any requests or issues with my tools in the appropriate topics.
I'm sorry if I don't reply or if I ignore PMs. My time is very limited.

barracuda
beginner
Posts: 30
Joined: Sat Nov 24, 2012 2:15 pm
Has thanked: 3 times
Been thanked: 27 times

Re: Unity Assets Explorer [v 1.2]

Post by barracuda » Sun Jun 30, 2013 11:29 am

Chipicao wrote:It's not about figuring out the differences, it's about knowing when there are differences. And the key to that are the file version and file "type", both of which would be discarded after extraction.
100% support for all Unity games is not unrealistic, it's my goal ;) Ok, maybe except xbox and PS3. Those use big endian data and would be better off with a separate script.
Well, I'll know when there are differences, because my program will start throwing exceptions. :D And it's written in Java, so the endianness isn't a big issue in my case. It can easily switch between both during runtime without manual byte swapping.
Chipicao wrote:Fast & Furious 6: The game would be one example. Or CSR Racing and Drift Mania. You can find them at your local "pirate shop" :D
But here's a free app from Opel promoting one of their cars: https://mega.co.nz/#!2F9VRByY!HPu2GjBsj ... qmp9IxQlmM
Thanks! It mostly seems to work, just the PVR textures aren't supported yet. Hell, I don't even know how to read or convert these and the Wikipedia page about the format is a stub. :?
Chipicao wrote: Yeah, I've been through hell figuring out which texture type is ARGB, which is RGBA or BGRA and so on. But in the end I didn't use custom bitmasks because for some reason the nvidia plugin for PS ignores them. So for the sake of compatibility I just switch image bytes instead :D
Oh, so I'm not the only one with that problem. :D

The NVIDIA Texture Tools 2 can read the DDS files correctly, unlike the Photoshop plugin or ImageMagick. Really weird, since both the tools and the PS plugin are made by Nvidia...
Chipicao wrote: There's nothing to fix, they are supposed to be flipped and they look as they should once applied to the models.
Oh, ok then. I was a bit confused by the Unity editor, which flips the textures in the preview as well.
Chipicao wrote:What do you mean by "AG normal maps", Alpha-Green? Maybe you just have the wrong channel order...
No, the color channels are fine. Unity 3 and higher encodes the normals into alpha and green for DXT compressed textures. Red and blue are simply low-quality copies of green for better compression, so the texture appears to be grayscale even though it's not a bump map. It's explained in detail here.

Oh, and do you maybe have a mirror for your maxscript? I know, I could simply register on the forum, but I'm not a big fan of throwaway accounts...

User avatar
Chipicao
ultra-veteran
ultra-veteran
Posts: 476
Joined: Thu Feb 03, 2011 11:18 am
Has thanked: 42 times
Been thanked: 303 times
Contact:

Re: Unity Assets Explorer [v 1.2]

Post by Chipicao » Sun Jun 30, 2013 7:01 pm

barracuda wrote:Well, I'll know when there are differences, because my program will start throwing exceptions. :D
Yes but that's still not my point. It doesn't matter if your extractor knows about the differences, because whoever looks at the extracted data won't know. I'm talking about differences in the raw data structure (see below)
barracuda wrote:Thanks! It mostly seems to work, just the PVR textures aren't supported yet. Hell, I don't even know how to read or convert these and the Wikipedia page about the format is a stub. :?
Here's all you need to know about the PVR format. http://www.imgtec.com/powervr/insider/d ... ternal.pdf

I can see why it mostly works. The Web versions you've been working on are very similar to the iOS formats I linked. But have a look at these Web assets: http://www.mediafire.com/download/a684a ... 378/tc.rar
The version is not listed, but I think it's 3.3.0. You're gonna notice some differences in the position of the preload table, and also the data structure of Textures, Renderers, SkinnedMeshRenderers and meshes.
barracuda wrote:Oh, ok then. I was a bit confused by the Unity editor, which flips the textures in the preview as well.
I guess that's just an option, Nvidia's PS plugin has it as well. Most games I've worked with have flipped texture coordinates and "regular" textures are flipped in the shaders. So I'm guessing unity takes it a step forward by flipping textures from the get go.
barracuda wrote:No, the color channels are fine. Unity 3 and higher encodes the normals into alpha and green for DXT compressed textures. Red and blue are simply low-quality copies of green for better compression, so the texture appears to be grayscale even though it's not a bump map. It's explained in detail here..
Ah, so it's the old trick used in Doom 3. If that's the case, then the solution is indeed moving AG image data in RG channels, and discarding the rest. However, you would have to know when the texture is a normal map and when it isn't. And the only way to do that is to go through material slots.

After inspecting some of the games you linked, I realize a universal and automatic script might be impossible. I'll have to do some more tests to see if I need to differentiate it by platform, and I';; post a link when I\m done.
Please post any requests or issues with my tools in the appropriate topics.
I'm sorry if I don't reply or if I ignore PMs. My time is very limited.

barracuda
beginner
Posts: 30
Joined: Sat Nov 24, 2012 2:15 pm
Has thanked: 3 times
Been thanked: 27 times

Re: Unity Assets Explorer [v 1.2]

Post by barracuda » Mon Jul 01, 2013 6:41 pm

Chipicao wrote: I can see why it mostly works. The Web versions you've been working on are very similar to the iOS formats I linked. But have a look at these Web assets: http://www.mediafire.com/download/a684a ... 378/tc.rar
The version is not listed, but I think it's 3.3.0. You're gonna notice some differences in the position of the preload table, and also the data structure of Textures, Renderers, SkinnedMeshRenderers and meshes.
Seems to be the Scion car builder. It uses version 2.6.1f3 according to the .unity3d file. And yes, my tool can't read these older 2.x and 3.3 formats yet.

User avatar
Chipicao
ultra-veteran
ultra-veteran
Posts: 476
Joined: Thu Feb 03, 2011 11:18 am
Has thanked: 42 times
Been thanked: 303 times
Contact:

Re: Unity Assets Explorer [v 1.2]

Post by Chipicao » Mon Jul 01, 2013 11:21 pm

Indeed it is :)
Image
Please post any requests or issues with my tools in the appropriate topics.
I'm sorry if I don't reply or if I ignore PMs. My time is very limited.

rmezatang
advanced
Posts: 40
Joined: Mon Jun 21, 2010 3:32 am
Has thanked: 2 times
Been thanked: 4 times

Re: Unity Assets Explorer [v 1.2]

Post by rmezatang » Thu Jul 04, 2013 5:12 pm

Assets explorer works ok but errors out on some larger assets files.

Aluigi's BMS script apparently doesn't work on what it identifies as "Unity3d Raw" files, only works on "Unity3d Web"

anyone know if the BMS script will be updated to open these "Raw" files?

barracuda
beginner
Posts: 30
Joined: Sat Nov 24, 2012 2:15 pm
Has thanked: 3 times
Been thanked: 27 times

Re: Unity Assets Explorer [v 1.2]

Post by barracuda » Thu Jul 04, 2013 11:35 pm

Do you have a sample for a Unity3d Raw file? Then I could add support for it for my tool, which will be released in a few weeks.

rmezatang
advanced
Posts: 40
Joined: Mon Jun 21, 2010 3:32 am
Has thanked: 2 times
Been thanked: 4 times

Re: Unity Assets Explorer [v 1.2]

Post by rmezatang » Fri Jul 05, 2013 8:41 am

barracuda wrote:Do you have a sample for a Unity3d Raw file? Then I could add support for it for my tool, which will be released in a few weeks.
use your username to open the file

barracuda
beginner
Posts: 30
Joined: Sat Nov 24, 2012 2:15 pm
Has thanked: 3 times
Been thanked: 27 times

Re: Unity Assets Explorer [v 1.2]

Post by barracuda » Sat Jul 06, 2013 4:12 am

UnityRaw is essentially the same as UnityWeb, just without the LZMA compression. Here's the fixed BMS script:

Code: Select all

# Unity Web Player (unity3d files)
# script for QuickBMS http://quickbms.aluigi.org

endian big
idstring "UnityRaw"
get DUMMY long
get VER byte
get VER string
get VER string
get SIZE long
get DATAOFFSET long

goto DATAOFFSET

get NAME filename
string NAME += "_unpacked"
set PATH string NAME

get FILES long
for i = 0 < FILES
    get NAME string
    get OFFSET long
    get SIZE long
    set FNAME string PATH
    string FNAME += /
    string FNAME += NAME
    math OFFSET += DATAOFFSET
    log FNAME OFFSET SIZE
next i

rmezatang
advanced
Posts: 40
Joined: Mon Jun 21, 2010 3:32 am
Has thanked: 2 times
Been thanked: 4 times

Re: Unity Assets Explorer [v 1.2]

Post by rmezatang » Sat Jul 06, 2013 6:19 am

hmmm, something wrong here. tried the new script on some of the Unity3d Raw files kicked out by the earlier script. It kicked out a single file with a wacky name
CAB-62d024b8560c18343a45fc4dff7f5562 and no file extension.

Post Reply