XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Mon May 29, 2017 6:26 pm

All times are UTC + 1 hour


Forum rules


Please click here to view the forum rules



Post new topic Reply to topic  [ 76 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next
Author Message
 Post subject: Re: How to Get Ben 10 Ultimate Alien: Cosmic Destruction Mod
PostPosted: Fri Jan 27, 2017 1:02 pm 
Offline
ultra-n00b

Joined: Fri Jan 27, 2017 12:31 pm
Posts: 2
Has thanked: 0 time
Have thanks: 0 time
hi there im a big ben10 fan myself so it would really be cool if you shared with us the models like zip/rar thanks everybody for contributing in this :)


Top
 Profile  
 
 Post subject: Re: How to Get Ben 10 Ultimate Alien: Cosmic Destruction Mod
PostPosted: Sat Feb 18, 2017 1:47 am 
Offline
beginner

Joined: Fri Jun 26, 2015 11:01 pm
Posts: 37
Has thanked: 3 times
Have thanks: 1 time
is there any way to get these models with bones?


Top
 Profile  
 
 Post subject: Re: How to Get Ben 10 Ultimate Alien: Cosmic Destruction Mod
PostPosted: Thu May 04, 2017 9:23 am 
Offline
advanced
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 48
Has thanked: 10 times
Have thanks: 9 times
AceWell wrote:
i don't know, if there is models and textures in these the data makes no sense to me


Hi!
I'd done some research on the Ben 10 Omniverse files several months ago after I was told that the data
were compressed. But not untill recently did I figure out the way to decompress them. The size of the
umcompressed data recorded in the header helps me determine what algorithm it uses, and I got some
texts perfectly making sense. The model data is sequential,with notes in front of each type of data.
They're in order of $position, $normal, $uvatlas, $texcoord0, $tangent0, $binormal, $blendweight
and $blendindex. I tried to load the mesh with Hex2Obj but the point clouds were messed up like this:



It seems that the mesh is overlapped with several same meshes whose vertices were flattened
into orthogonal planes.
Tried with different face and vertice start addresses but nothing changed.
Could you take a look at them?

Here're some samples of the umcompressed data:

uncompressed data samples

Files prefixed with "SKIN" should contain only textures otherwise they contain both textures
and meshes. The hexadecimal sequence is the original file name which is still in the umcompressed data.

Thanks in advance!


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.



Top
 Profile  
 
 Post subject: Re: How to Get Ben 10 Ultimate Alien: Cosmic Destruction Mod
PostPosted: Thu May 04, 2017 12:47 pm 
Offline
M-M-M-Monster veteran
M-M-M-Monster veteran
User avatar

Joined: Fri Apr 20, 2012 9:24 am
Posts: 2025
Has thanked: 372 times
Have thanks: 1011 times
point cloud looks ok to me (if you use one at a time), problem is how to get suiting face indices
edit: got it


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.


_________________
Extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-jm9.zip

"We are Microsoft. You will be assimilated. Resistance is Futile."


Top
 Profile  
 
 Post subject: Re: How to Get Ben 10 Ultimate Alien: Cosmic Destruction Mod
PostPosted: Thu May 04, 2017 11:09 pm 
Offline
VIP member
VIP member
User avatar

Joined: Wed Nov 05, 2008 12:16 pm
Posts: 772
Has thanked: 1693 times
Have thanks: 382 times
Bigchillghost wrote:
I'd done some research on the Ben 10 Omniverse files several months ago after I was told that the data were compressed. But not untill recently did I figure out the way to decompress them. The size of the umcompressed data recorded in the header helps me determine what algorithm it uses, and I got some texts perfectly making sense.

care to share how you decompressed your samples? :D

_________________
Noesis
Hex2obj
QuickBMS
TextureFinder
Unity Assets Bundle Extractor


Top
 Profile  
 
 Post subject: Re: How to Get Ben 10 Ultimate Alien: Cosmic Destruction Mod
PostPosted: Fri May 05, 2017 9:38 am 
Offline
advanced
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 48
Has thanked: 10 times
Have thanks: 9 times
shakotay2 wrote:
point cloud looks ok to me (if you use one at a time), problem is how to get suiting face indices
edit: got it


I'm so impressed that you still follow with interest this thread. :D

I was using the same FI startaddr as you did in the pic, but using 12 bytes as FVFSize and 0x2330 as vertices startaddr.
And I used seq mode since the vertices and UVs & normals are at separated blocks.(Yeah, why is it a blocked model?)

So I changed it to VB mode and I got this:


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.



Top
 Profile  
 
 Post subject: Re: How to Get Ben 10 Ultimate Alien: Cosmic Destruction Mod
PostPosted: Fri May 05, 2017 9:57 am 
Offline
advanced
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 48
Has thanked: 10 times
Have thanks: 9 times
This is what I got from my lastest research:



The face indices counts is always 0x2B bytes after the string "texcoord0" at its first occurence, which also appears in the texcoord
block. However the vertices counts is not always accurate which should be fixed by Hex2Obj.

So the texcoord0 block is its UVs, then what is the uvatlas for?


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.



Last edited by Bigchillghost on Fri May 05, 2017 1:34 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: How to Get Ben 10 Ultimate Alien: Cosmic Destruction Mod
PostPosted: Fri May 05, 2017 12:05 pm 
Offline
advanced
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 48
Has thanked: 10 times
Have thanks: 9 times
AceWell wrote:
care to share how you decompressed your samples? :D


I'm glad that you're interested in it. Well, it's gonna be a "long" story, which took me several months... (Athough I put it aside for most of the time after I'd finished the analysis of the header. :D )
Few months ago...
Chrrox said that the data were compressed. So I did some research about this format and this was what I'd discovered:



At first I searched the size of this file which is 0x1CEB6B. To avoid confusion I search for 001CEB6B and it led me to 0x178. That means these four bytes is an offset. And looked back for every four bytes I found that they had some same features that their first byte was always ZERO byte and they're decreasing backwards. So they might also be offsets, and what I needed to do was to find out where the first offset was stored. So just continuely moved backwards and you can see 00 00 01 7C, and another four bytes before is 00 00 00 5B. Since 0x5B is smaller than 0x178 (the offset of the end of this file), it can't be any offsets. So what is it for? I compared different files and found that the header was 8 bytes long and the values of the four bytes at 0xC differed from file to file.
The I picked the smallest file, which had only 2 Kb, to understand this four bytes more easily. Then I discovered that there were only two offsets from 0x10 to 0x17, and it chanced to be 0x00 00 00 02 at 0xC. They I come back to check this file and found it matched the number of these offsets. So it was the counts of the offsets, and the four bytes at 0x10 should be the start of the compressed data. So just jumped to 0x17C to see what would happen. It read 08 FA AF FA AF 00 00 00, same as other files. This comfirmed that it was the start of the compressed data since 0x8FAAFFA won't be an offset. At offset 0x8 these four bytes is larger than the archive so it should be the uncompressed size of the data. (I didn't notice that chrrox had reminded me about it later. :oops: )
But I didn't know how to decompress them and chrrox didn't reply...

Few monthes went by...

One day I checked aluigi's site again and found a tutorial on the comtype scanner 2. (I didn't know how to get that script work before and was using only the bms script without the .bat file)
Then I tested it on a 2 Kb file which I removed the header & the stored offset. After the batch script finished its job, I filtered the files I didn't want with the assistance of the the uncompressed size stored inside the raw data. And then checked the results in HexEdit one by one to find the correct output. Therefore I discovered that it used LZF/LZFX compression, either result is the same. You can see that the decompressed file name like "Art\Characters\c_wildvine.max" which was uncompleted in the original file. I believed the data were decompressed correctly especially when I load it into hex2obj and saw the point cloud made sense. But what's interesting is that if I decompress it with BLOSC algorithm, the texts will still look correct while the model data would be totally different (with larger size), but the point cloud didn't change at all!
This is the BMS script I wrote to decompress the data:

Code:
#QuickBMS generic script to decompress data from Ben 10 Omniverse for XBox 360
#http://aluigi.altervista.org/quickbms.htm

idstring "\x2F\xA9\xC0\x56"
goto 0x8
get SIZE long
findloc OFFSET binary "\x08\xFA\xAF\xFA"
get ZSIZE asize
math ZSIZE - OFFSET
goto OFFSET
get NAME basename
string NAME + "_"
string NAME + OFFSET
string NAME + ".ucp"
comtype lzf
clog NAME OFFSET ZSIZE SIZE


It was written for the files I renamed manually. I can't get the getDstring command work. Can you help me improve it?
The original name (hexadecimal sequence) is four bytes before the name string, and the length of the name is
one byte (the symbol $) before it. The string doesn't end with null byte.
For example, the original name of the file CHAR_Wildvine is 91 FC 4C 91 5B 0D BE 44, while in the hex editor there is content like

Code:
hexadecimal sequence           Unknown       Length      $       C  H  A  R  _  W  i  l  d  v  i  n   e     End
   
91 FC 4C 91 5B 0D BE 44         20 0A         0D         24      43 48 41 52 5F 57 69 6C 64 76 69 6E 65      20


I added this piece of code to the begining of the script but it didn't work.

Code:
get ORIG_NAME basename
findloc OFFSET1 string ORIG_NAME
goto OFFSET1
get SKIP1 short
get LENGTH byte
get SKIP2 byte
getDstring NAME LENGTH


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.



Top
 Profile  
 
 Post subject: Re: How to Get Ben 10 Ultimate Alien: Cosmic Destruction Mod
PostPosted: Sat May 06, 2017 3:57 am 
Offline
VIP member
VIP member
User avatar

Joined: Wed Nov 05, 2008 12:16 pm
Posts: 772
Has thanked: 1693 times
Have thanks: 382 times
Bigchillghost wrote:
Can you help me improve it?

certainly :D
Code:
endian big
comtype lzf //lzf,lzfx,BLOSC
get ZSIZE asize
idstring "\x2F\xA9\xC0\x56\xCD\x33\x2E\x57"
get SIZE long
get NUM_OFFSETS long
get OFFSET long
math ZSIZE - OFFSET
xmath NAME_OFF "OFFSET + 0x33"
goto NAME_OFF
get NAME_LENGTH byte
get SKIP byte
getdstring NAME NAME_LENGTH 
string NAME + ".et_decomp"
clog NAME OFFSET ZSIZE SIZE


i think that offset table at the beginning is relevant for proper splitting though,
might need to research that a little more because the decompressed files look
like they contain multiple, like the texture files may have many textures etc.

_________________
Noesis
Hex2obj
QuickBMS
TextureFinder
Unity Assets Bundle Extractor


Top
 Profile  
 
 Post subject: Re: How to Get Ben 10 Ultimate Alien: Cosmic Destruction Mod
PostPosted: Sat May 06, 2017 10:37 am 
Offline
advanced
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 48
Has thanked: 10 times
Have thanks: 9 times
AceWell wrote:
certainly :D
Code:
endian big
comtype lzf //lzf,lzfx,BLOSC
get ZSIZE asize
idstring "\x2F\xA9\xC0\x56\xCD\x33\x2E\x57"
get SIZE long
get NUM_OFFSETS long
get OFFSET long
math ZSIZE - OFFSET
xmath NAME_OFF "OFFSET + 0x33"
goto NAME_OFF
get NAME_LENGTH byte
get SKIP byte
getdstring NAME NAME_LENGTH 
string NAME + ".et_decomp"
clog NAME OFFSET ZSIZE SIZE

Thanks! You'd just made it much more simple!

AceWell wrote:
i think that offset table at the beginning is relevant for proper splitting though,
might need to research that a little more because the decompressed files look
like they contain multiple, like the texture files may have many textures etc.


Yeah, I'd considered that possibility too. I'd also split few files from a texture archive before decompressing
them. But I didn't see any useful info at the header of these split files, same as in their decompressed data.

And I'd just written a script to split the file according to the offset table, and then decompressed them.
It turned out that all the separated files except the last one have a fixed size 0x8000 (exactly 32 Kb) after
being decompressed.
Seems it uses multi-volume compression.

Here the split textures if you want to take a look at:
SKIN_Arctiguana_Teen & SKIN_Wildvine_Young


Top
 Profile  
 
 Post subject: Re: How to Get Ben 10 Ultimate Alien: Cosmic Destruction Mod
PostPosted: Tue May 16, 2017 11:06 am 
Offline
advanced
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 48
Has thanked: 10 times
Have thanks: 9 times
AceWell wrote:
the decompressed files look like they contain multiple, like the texture files may have many textures etc.


The texture files contain diffuse and normal maps, as well as other smaller textures. I can't find a particular strings or sequences in front of the texture data to locate where it start, so instead I have to use the strings right after its name to locate the textures especially in meshes files(might need to change _d to _n to locate normal maps). The texture data look like dds format so I just add a 0x80 header to it and it looks like this:



At least its colors match with the low-resolution map from Wii. I've tried your previous script for untile 360 DXT(after some necessary modifications), but the converted texture still looked the same. Really have no idea what to do next.


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.



Top
 Profile  
 
 Post subject: Re: How to Get Ben 10 Ultimate Alien: Cosmic Destruction Mod
PostPosted: Thu May 25, 2017 3:18 pm 
Offline
advanced
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 48
Has thanked: 10 times
Have thanks: 9 times
@AceWell

I guess either you're too busy or you havn't seen my post? :( But I'm getting closer...

I extracted some textures from a Ben 10 iOS game some time ago, and got some large maps which I guess were taken directly from the Omniverse X360 game.

Since the textures in the archives looks like DDS, I converted one of them to the target format and compared it with the same texture part of the model archive. I tried different dds formats untill I found them highly consistent, except that I noticed the data were swapped. So I used Noesis to swap the array again and end up with this:



The results of the normal maps were even worse.

I guess it's the best I can do. I really really need your help right now... (:

Here're all the textures of the playable characters which I dumped from the archive. Their lengths should be enough to accommodate the actual image data as I split them according to the pointer inside the archives.

Omniverse_Texture

Oh, one more thing, the texture format is dxt1 with only RGB color, no Alpha channel.

Thank you for your kindness in advance!


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.



Top
 Profile  
 
 Post subject: Re: How to Get Ben 10 Ultimate Alien: Cosmic Destruction Mod
PostPosted: Thu May 25, 2017 4:01 pm 
Offline
advanced
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 48
Has thanked: 10 times
Have thanks: 9 times
This is the script I used:

Code:
from inc_noesis import *

def registerNoesisTypes():
    handle = noesis.register("Ben 10 Omniverse (X360)", ".omni")
    noesis.setHandlerTypeCheck(handle, noepyCheckType)
    noesis.setHandlerLoadRGBA(handle, noepyLoadRGB)
    #noesis.logPopup()
    return 1

#check if it's this type based on the data
def noepyCheckType(data):
    bs = NoeBitStream(data)
    Magic = bs.readBytes(8)
    if Magic != b'\x00\x00\x00\x01\x00\x00\x00\x24':
        return 0
    return 1
   
def noepyLoadRGB(data, texList):
    datasize = len(data) - 0x70       
    bs = NoeBitStream(data, NOE_BIGENDIAN)
    bs.seek(0x3E, NOESEEK_ABS)
    imgWidth = bs.readUShort()           
    imgHeight = bs.readUShort()           
    bs.seek(0x70, NOESEEK_ABS)
    data = bs.readBytes(datasize)     
    data = rapi.imageUntile360DXT(rapi.swapEndianArray(data, 2), imgWidth, imgHeight, 8)
    texFmt = noesis.NOESISTEX_DXT1
    texList.append(NoeTexture(rapi.getInputName(), imgWidth, imgHeight, data, texFmt))
    return 1


The four bytes after the 0x BB BB BB BB in the file is the original end offset of the texture data, which originally ended
with 0x BE BE BE BE. Since there's an id string 0x 00 00 00 01 00 00 00 24 in front of every name string, I assumed it
as the start of every file. This explanation should help avoid some confusions.


Top
 Profile  
 
 Post subject: Re: How to Get Ben 10 Ultimate Alien: Cosmic Destruction Mod
PostPosted: Thu May 25, 2017 7:23 pm 
Offline
VIP member
VIP member
User avatar

Joined: Wed Nov 05, 2008 12:16 pm
Posts: 772
Has thanked: 1693 times
Have thanks: 382 times
Bigchillghost wrote:
@AceWell
I guess either you're too busy or you havn't seen my post? :( But I'm getting closer...

i don't know what you were asking for there :]


Bigchillghost wrote:

it looks like you got this under control but here is a Noesis python script to open those samples :D

always specify full game name and platform from which the sample are taken.
you have samples coming from all over the place and from various games so it is a little confusing. :)


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.


_________________
Noesis
Hex2obj
QuickBMS
TextureFinder
Unity Assets Bundle Extractor


Top
 Profile  
 
 Post subject: Re: How to Get Ben 10 Ultimate Alien: Cosmic Destruction Mod
PostPosted: Fri May 26, 2017 6:56 am 
Offline
advanced
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 48
Has thanked: 10 times
Have thanks: 9 times
AceWell wrote:
i don't know what you were asking for there :]


I didn't ask for anything at the post on the split files according to the offset table, but to see if my finding is correct or not from your experience, so that I can dig deeper. But obviously this finding is worthless, the offset table might just be used by the game to decompress data faster.

In the other post I compared the two textures from XBox 360 and Wii, to show I was close to locate the texture data block, and the result looks like those from Cartoon Network Punch Time Explosion on XBox 360, which you've handled queerly. So I was hopping you can take a closer look at those samples, which are at the eighth post previous to that one, but still at this page. Well, it's indeed miles away especially since I've written that 'long story'. I should have attached the link there, my bad... I thought you would remember... :D


AceWell wrote:
it looks like you got this under control but here is a Noesis python script to open those samples :D


Wow, the size of the header is 0x6d? I’d also tried with 0x6c before but nothing change, can’t believe it. :o But thanks, it works fine!
Now just need to find a way to split the textures from all the archives of this game.

So in Noesis there’s no difference between RGB and ARGB pixel formats of DDS according to your script, right? :]


AceWell wrote:
always specify full game name and platform from which the sample are taken.
you have samples coming from all over the place and from various games so it is a little confusing. :)


I've never uploaded samples from this game of other platforms, but just use those smaller or same-resolution textures I grabbed from other games for a comparison on the sight. Sometimes I do compare the Wii data with those dumped by Dolphin, the Wii emulator, hoping to get a easy start. But I've never published the result, because it was totally futile efforts. I can't even extract one mesh from one of the Wii version of those game this thread has mentioned previously. I will always specify if the samples come from other platform or games, either in the text between the link, or the line before it (the link).
As for the full name, since this whole thread is about Ben 10, I guess there's no need to write down Ben 10 Omniverse AGAIN rather than using Omniverse which we all can know what it refers to.

But just as you said, since I work with archives of the same game from all over the platforms (those "other game" are Unity games), it may indeed cause confusion. Actually sometime I just skip those details to avoid repeated statements. Anyway I will adopt your advice so there won't be any confusion next time. :)


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 76 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next

All times are UTC + 1 hour


Who is online

Users browsing this forum: Darko, Tosyk and 12 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group