XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Tue Aug 22, 2017 1:57 pm

All times are UTC + 1 hour


Forum rules


Please click here to view the forum rules



Post new topic Reply to topic  [ 143 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6, 7, 8 ... 10  Next
Author Message
 Post subject: Re: Battlefield 3: Sounds recording technique
PostPosted: Wed May 01, 2013 1:11 am 
Offline
double-veteran
double-veteran

Joined: Sun Apr 20, 2008 2:58 am
Posts: 824
Has thanked: 5 times
Have thanks: 32 times

shouldn't be too hard, match the ebx with the id. Although it would be cool to have a linker script as not to rename manually one by one but this will do for now. then we can add our own xma header. and convert.

You can make the ads go away by registering



Top
 Profile  
 
 Post subject: Re: Battlefield 3: Sounds recording technique
PostPosted: Wed May 01, 2013 1:17 am 
Offline
veteran
User avatar

Joined: Fri Feb 18, 2011 4:58 pm
Posts: 125
Has thanked: 15 times
Have thanks: 14 times
OrangeC wrote:
shouldn't be too hard, match the ebx with the id. Although it would be cool to have a linker script as not to rename manually one by one but this will do for now. then we can add our own xma header. and convert.


I don't know anything about .xma format. So, I can't help you for that...

_________________
Vosvoy


Top
 Profile  
 
 Post subject: Re: Battlefield 3: Sounds recording technique
PostPosted: Wed May 01, 2013 2:46 am 
Offline
veteran
User avatar

Joined: Tue Jul 17, 2012 3:52 am
Posts: 87
Has thanked: 15 times
Have thanks: 10 times
Image

OMG I'm so dumb....... The PC chunks are different from the console chunks. They don't match up. Why didn't I think of that.

As an example the AEK-71 long reload CHUNKID for BF3 X360 is 34ab8c16913d8e37e7c05f76457d83a4 for BF3 PC it is 191FBE10D72A6AE3333401D966E8CBAE

I'm so sorry I didn't look at it sooner.


Last edited by durandal217 on Thu May 09, 2013 8:22 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Battlefield 3: Sounds recording technique
PostPosted: Wed May 01, 2013 6:23 am 
Offline
double-veteran
double-veteran

Joined: Sun Apr 20, 2008 2:58 am
Posts: 824
Has thanked: 5 times
Have thanks: 32 times
That's unfortunate. All the more reason to implement console ebx conversion.


Top
 Profile  
 
 Post subject: Re: Battlefield 3: Sounds recording technique
PostPosted: Wed May 01, 2013 2:32 pm 
Offline
veteran
User avatar

Joined: Fri Feb 18, 2011 4:58 pm
Posts: 125
Has thanked: 15 times
Have thanks: 14 times
Yeah, we need to find a solution for that. I'll investigate scrupulously the python script for my part. Otherwise, maybe the script creator could help us.

Anyway, we now know that .ebx files from PC are not compliant with Xbox360 version. It's a small step forward.

_________________
Vosvoy


Top
 Profile  
 
 Post subject: Re: Battlefield 3: Sounds recording technique
PostPosted: Sun May 05, 2013 11:39 pm 
Offline
veteran
User avatar

Joined: Tue Jul 17, 2012 3:52 am
Posts: 87
Has thanked: 15 times
Have thanks: 10 times
Just wanted to give you guys a heads up and let you know Frankelstner updated his sb/toc dumper for MOH warfighter.
http://www.bfeditor.org/forums/index.ph ... opic=15731

No update yet on audio conversion though.


Top
 Profile  
 
 Post subject: Re: Battlefield 3: Sounds recording technique
PostPosted: Tue May 07, 2013 4:58 pm 
Offline
veteran
User avatar

Joined: Fri Feb 18, 2011 4:58 pm
Posts: 125
Has thanked: 15 times
Have thanks: 14 times
We've got some useful details from the script creator:

Quote:
From Frankelstner:

The dll does exactly one thing: It converts a encoded block of 76 bytes into a decoded block of 128 floats. So the dll already does everything audio specific already. The task of the Python script is to feed the right bytes to the dll and eventually create a wav file from the floats it receives. It's not about audio at anymore, the problem is to have the script ensure that the file in question is actually XAS encoded, and to jump to the right places while reading an encoded file. The jumping part is already taken care of. And ensuring the file is actually XAS is just looking at the file with the strange length error and comparing it with other files that work correctly. Somewhere in the beginning there must be a byte different to tell the engine how to decode the format.

_________________
Vosvoy


Top
 Profile  
 
 Post subject: Re: Battlefield 3: Sounds recording technique
PostPosted: Tue May 07, 2013 10:28 pm 
Offline
double-veteran
double-veteran

Joined: Sun Apr 20, 2008 2:58 am
Posts: 824
Has thanked: 5 times
Have thanks: 32 times
hmm strange it seems that the console chunks are weird when trying to match them up. Whats the difference here?

had to find the file by using the size because the chunk ID is not mall the same except the last numbers.

Code:
000008c0    SoundWaveAsset 4a8e4be51666e0f748c9e60fc57f66a5 #primary instance
000008d0        $::SoundDataAsset
000008d0            $::Asset
000008d0                $::DataContainer
000008d8                Name AO4_Assets/Sound/Music/02_Outskirts/Action/Mus_Loop_OS02_Action_MMG1_134_Multi
000008dc                BudgetCategory Bc_Unknown
000008e0            NameHash 2538223577
000008e4            Chunks::array
00000944                member(0)::SoundDataChunk
00000944                    ChunkId B17600B855D3AC5D4D9090FF8B2B17BE
00000954                    ChunkSize 2083084
000008e8        Variations::array

Now the actual file is:b80076b1d3555dac4d9090ff8b2b17be.chunk


Some numbers are similiar but others not.


Top
 Profile  
 
 Post subject: Re: Battlefield 3: Sounds recording technique
PostPosted: Thu May 09, 2013 12:06 am 
Offline
veteran
User avatar

Joined: Fri Feb 18, 2011 4:58 pm
Posts: 125
Has thanked: 15 times
Have thanks: 14 times
Hey mates! Some news here:

Quote:
From Frankelstner:

Alright, here's a version that can deal with mohw (and hopefully big-endian files too, though I haven't tested that). For some reason the codec sometimes uses very strange values for the number of channels. E.g. the byte usually is 00 for 1 channel and 04 for 2 channels. However, some files use 01 and 05. I have played around with one sample file of each type, and think that 01 is 1 channel too and 05 is 2 channels. However, there's a chance some files are messed up, so give feedback if you find garbled audio.

Code:
#needs python 2.7
import string
import sys
from binascii import hexlify
from struct import unpack,pack
import os
from cStringIO import StringIO
import cProfile
import cPickle

ebxFolder=   r"D:\hexing\mohw dump\bundles\ebx" #audio is found in several places
chunkFolder= r"D:\hexing\mohw dump\chunks" #grab audio from here
chunkFolder2=r"D:\hexing\mohw dump\bundles\chunks" #if the chunk is not found in the first folder, use this one
outputFolder=r"D:\decoded mohw123123"


#Run through the sound ebx files, find fields with chunk Guids and fieldName = ChunkId.
#The script will overwrite existing files.

#The filename of the ebx file importing an audio chunk becomes the name of the wav file.
#As for the indices, there are three of them in the following order.
#1: Variation.ChunkIndex: Some ebx files import several completely independent audio chunk files. This index differentiates between them.
#2: Variation.Index: An ebx may use the same audio chunk for several sound variations, this index keeps them apart.
#3: Segment.Index: A variation may contain several segments, so there is another index.


##############################################################
##############################################################
def unpackBE(typ,data): return unpack(">"+typ,data)

def open2(path,mode="rb"):
    if mode=="wb":
        #create folders if necessary and return the file handle
        #first of all, create one folder level manully because makedirs might fail
        path=os.path.normpath(path)
        pathParts=path.split("\\")
        manualPart="\\".join(pathParts[:2])
        if not os.path.isdir(manualPart): os.makedirs(manualPart)

        #now handle the rest, including extra long path names
        folderPath=lp(os.path.dirname(path))
        if not os.path.isdir(folderPath): os.makedirs(folderPath)
    return open(lp(path),mode)

def lp(path): #long pathnames
    if len(path)<=247 or path=="" or path[:4]=='\\\\?\\': return path
    return unicode('\\\\?\\' + os.path.normpath(path))

def decodeAudio():
    for dir0, dirs, ff in os.walk(ebxFolder):
        for fname in ff:
            f=open2(dir0+"\\"+fname)
            magic=f.read(4)
            if magic=="\xCE\xD1\xB2\x0F":
                dbx=Dbx(f,unpack)
            elif magic=="\x0F\xB2\xD1\xCE":
                dbx=Dbx(f,unpackBE)
            else:
                f.close()
                continue
            dbx.decode()
           
try:
    from ctypes import *
    floatlib = cdll.LoadLibrary("floattostring")
    def formatfloat(num):
        bufType = c_char * 100
        buf = bufType()
        bufpointer = pointer(buf)
        floatlib.convertNum(c_double(num), bufpointer, 100)
        rawstring=(buf.raw)[:buf.raw.find("\x00")]
        if rawstring[:2]=="-.": return "-0."+rawstring[2:]
        elif rawstring[0]==".": return "0."+rawstring[1:]
        elif "e" not in rawstring and "." not in rawstring: return rawstring+".0"
        return rawstring
except:
    def formatfloat(num):
        return str(num)
def hasher(keyword): #32bit FNV-1 hash with FNV_offset_basis = 5381 and FNV_prime = 33
    hash = 5381
    for byte in keyword:
        hash = (hash*33) ^ ord(byte)
    return hash & 0xffffffff # use & because Python promotes the num instead of intended overflow
class Header:
    def __init__(self,varList): ##all 4byte unsigned integers
        self.absStringOffset     = varList[0]  ## absolute offset for string section start
        self.lenStringToEOF      = varList[1]  ## length from string section start to EOF
        self.numGUID             = varList[2]  ## number of external GUIDs
        self.null                = varList[3]  ## 00000000
        self.numInstanceRepeater = varList[4]
        self.numComplex          = varList[5]  ## number of complex entries
        self.numField            = varList[6]  ## number of field entries
        self.lenName             = varList[7]  ## length of name section including padding
        self.lenString           = varList[8]  ## length of string section including padding
        self.numArrayRepeater    = varList[9]
        self.lenPayload          = varList[10] ## length of normal payload section; the start of the array payload section is absStringOffset+lenString+lenPayload
class FieldDescriptor:
    def __init__(self,varList,keywordDict):
        self.name            = keywordDict[varList[0]]
        self.type            = varList[1]
        self.ref             = varList[2] #the field may contain another complex
        self.offset          = varList[3] #offset in payload section; relative to the complex containing it
        self.secondaryOffset = varList[4]
class ComplexDescriptor:
    def __init__(self,varList,keywordDict):
        self.name            = keywordDict[varList[0]]
        self.fieldStartIndex = varList[1] #the index of the first field belonging to the complex
        self.numField        = varList[2] #the total number of fields belonging to the complex
        self.alignment       = varList[3]
        self.type            = varList[4]
        self.size            = varList[5] #total length of the complex in the payload section
        self.secondarySize   = varList[6] #seems deprecated
class InstanceRepeater:
    def __init__(self,varList):
        self.null            = varList[0] #called "internalCount", seems to be always null
        self.repetitions     = varList[1] #number of instance repetitions
        self.complexIndex    = varList[2] #index of complex used as the instance
class arrayRepeater:
    def __init__(self,varList):
        self.offset          = varList[0] #offset in array payload section
        self.repetitions     = varList[1] #number of array repetitions
        self.complexIndex    = varList[2] #not necessary for extraction
class Complex:
    def __init__(self,desc,dbxhandle):
        self.desc=desc
        self.dbx=dbxhandle #lazy
    def get(self,name):
        pathElems=name.split("/")
        curPos=self
        if pathElems[-1].find("::")!=-1: #grab a complex
            for elem in pathElems:
                try:
                    curPos=curPos.go1(elem)
                except Exception,e:
                    raise Exception("Could not find complex with name: "+str(e)+"\nFull path: "+name+"\nFilename: "+self.dbx.trueFilename)
            return curPos
        #grab a field instead
        for elem in pathElems[:-1]:
            try:
                curPos=curPos.go1(elem)
            except Exception,e:
                raise Exception("Could not find complex with name: "+str(e)+"\nFull path: "+name+"\nFilename: "+self.dbx.trueFilename)
        for field in curPos.fields:
            if field.desc.name==pathElems[-1]:
                return field
           
        raise Exception("Could not find field with name: "+name+"\nFilename: "+self.dbx.trueFilename)

    def go1(self,name): #go once
        for field in self.fields:
            if field.desc.type in (0x0029, 0xd029,0x0000,0x0041):
                if field.desc.name+"::"+field.value.desc.name == name:
                    return field.value
        raise Exception(name)


class Field:
    def __init__(self,desc,dbx):
        self.desc=desc
        self.dbx=dbx
    def link(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)
       
        if self.value>>31:
            extguid=self.dbx.externalGUIDs[self.value&0x7fffffff]
           
            for existingDbx in dbxArray:
                if existingDbx.fileGUID==extguid[0]:
                    for guid, instance in existingDbx.instances:
                        if guid==extguid[1]:
                            return instance
                   

            f=valid(inputFolder+guidTable[extguid[0]]+".ebx")
##            print guidTable[extguid[0]]
            dbx=Dbx(f)
            dbxArray.append(dbx)
            for guid, instance in dbx.instances:
                if guid==extguid[1]:
                    return instance
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
        elif self.value!=0:
            for guid, instance in self.dbx.instances:
                if guid==self.dbx.internalGUIDs[self.value-1]:
                    return instance
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)

        raise Exception("Invalid link, could not find target.")

    def getlinkguid(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)

        if self.value>>31:
            return "".join(self.dbx.externalGUIDs[self.value&0x7fffffff])
        elif self.value!=0:
            return self.dbx.fileGUID+self.dbx.internalGUIDs[self.value-1]
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
    def getlinkname(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)

        if self.value>>31:
            return guidTable[self.dbx.externalGUIDs[self.value&0x7fffffff][0]]+"/"+self.dbx.externalGUIDs[self.value&0x7fffffff][1]
        elif self.value!=0:
            return self.dbx.trueFilename+"/"+self.dbx.internalGUIDs[self.value-1]
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
   

         
def valid(fname):
    f=open2(fname,"rb")
    if f.read(4) not in ("\xCE\xD1\xB2\x0F","\x0F\xB2\xD1\xCE"):
        f.close()
        raise Exception("nope")
    return f

class nullguid(Exception):
    def __init__(self, value):
        self.value = value
    def __str__(self):
        return repr(self.value)


numDict={0x0035:("I",4),0xc10d:("I",4),0xc14d:("d",8),0xc0ad:("?",1),0xc0fd:("i",4),0xc0bd:("B",1),0xc0ed:("h",2), 0xc0dd:("H",2), 0xc13d:("f",4)}


class Stub:
    pass



class Dbx:
    def __init__(self, f,unpacker):
        self.unpack=unpacker
        self.trueFilename=""
        self.header=Header(self.unpack("11I",f.read(44)))
        self.arraySectionstart=self.header.absStringOffset+self.header.lenString+self.header.lenPayload
        self.fileGUID, self.primaryInstanceGUID = f.read(16), f.read(16)   
        self.externalGUIDs=[(f.read(16),f.read(16)) for i in xrange(self.header.numGUID)]
        self.keywords=str.split(f.read(self.header.lenName),"\x00")
        self.keywordDict=dict((hasher(keyword),keyword) for keyword in self.keywords)
        self.fieldDescriptors=[FieldDescriptor(self.unpack("IHHII",f.read(16)), self.keywordDict) for i in xrange(self.header.numField)]
        self.complexDescriptors=[ComplexDescriptor(self.unpack("IIBBHHH",f.read(16)), self.keywordDict) for i in xrange(self.header.numComplex)]
        self.instanceRepeaters=[InstanceRepeater(self.unpack("3I",f.read(12))) for i in xrange(self.header.numInstanceRepeater)]
        while f.tell()%16!=0: f.seek(1,1) #padding
        self.arrayRepeaters=[arrayRepeater(self.unpack("3I",f.read(12))) for i in xrange(self.header.numArrayRepeater)]

        #payload
        f.seek(self.header.absStringOffset+self.header.lenString)
        self.internalGUIDs=[]
        self.instances=[] # (guid, complex)
        for instanceRepeater in self.instanceRepeaters:
            for repetition in xrange(instanceRepeater.repetitions):
                instanceGUID=f.read(16)
                self.internalGUIDs.append(instanceGUID)
                if instanceGUID==self.primaryInstanceGUID:
                    self.isPrimaryInstance=True
                else:
                    self.isPrimaryInstance=False
                inst=self.readComplex(instanceRepeater.complexIndex,f)
                inst.guid=instanceGUID
               
                if self.isPrimaryInstance: self.prim=inst
                self.instances.append((instanceGUID,inst))
   
        f.close()
       

    def readComplex(self, complexIndex,f):
        complexDesc=self.complexDescriptors[complexIndex]
        cmplx=Complex(complexDesc,self)
       
        startPos=f.tell()                 
        cmplx.fields=[]
        for fieldIndex in xrange(complexDesc.fieldStartIndex,complexDesc.fieldStartIndex+complexDesc.numField):
            f.seek(startPos+self.fieldDescriptors[fieldIndex].offset)
            cmplx.fields.append(self.readField(fieldIndex,f))
       
        f.seek(startPos+complexDesc.size)
        return cmplx


    def readField(self,fieldIndex,f):
        fieldDesc = self.fieldDescriptors[fieldIndex]
        field=Field(fieldDesc,self)
       
        if fieldDesc.type in (0x0029, 0xd029,0x0000):
            field.value=self.readComplex(fieldDesc.ref,f)
        elif fieldDesc.type==0x0041:
            arrayRepeater=self.arrayRepeaters[self.unpack("I",f.read(4))[0]]
            arrayComplexDesc=self.complexDescriptors[fieldDesc.ref]

##            if arrayRepeater.repetitions==0: field.value = "*nullArray*"
            f.seek(self.arraySectionstart+arrayRepeater.offset)
            arrayComplex=Complex(arrayComplexDesc,self)
            arrayComplex.fields=[self.readField(arrayComplexDesc.fieldStartIndex,f) for repetition in xrange(arrayRepeater.repetitions)]
            field.value=arrayComplex
           
        elif fieldDesc.type in (0x407d, 0x409d):
            startPos=f.tell()
            f.seek(self.header.absStringOffset+self.unpack("I",f.read(4))[0])
            string=""
            while 1:
                a=f.read(1)
                if a=="\x00": break
                else: string+=a
            f.seek(startPos+4)
           
            if len(string)==0: field.value="*nullString*" #actually the string is ""
            else: field.value=string
           
            if self.isPrimaryInstance and self.trueFilename=="" and fieldDesc.name=="Name": self.trueFilename=string
           
                   
        elif fieldDesc.type==0x0089: #incomplete implementation, only gives back the selected string
            compareValue=self.unpack("I",f.read(4))[0]
            enumComplex=self.complexDescriptors[fieldDesc.ref]

            if enumComplex.numField==0:
                field.value="*nullEnum*"
            for fieldIndex in xrange(enumComplex.fieldStartIndex,enumComplex.fieldStartIndex+enumComplex.numField):
                if self.fieldDescriptors[fieldIndex].offset==compareValue:
                    field.value=self.fieldDescriptors[fieldIndex].name
                    break
        elif fieldDesc.type==0xc15d:
            field.value=f.read(16)
        elif fieldDesc.type == 0xc13d: ################################
            field.value=formatfloat(self.unpack("f",f.read(4))[0])
        else:
            (typ,length)=numDict[fieldDesc.type]
            num=self.unpack(typ,f.read(length))[0]
            field.value=num
       
        return field
       

    def dump(self,outputFolder):
        dirName=os.path.dirname(outputFolder+self.trueFilename)
        if not os.path.isdir(dirName): os.makedirs(dirName)
        f2=open2(outputFolder+self.trueFilename+EXTENSION,"wb")
        print self.trueFilename
       
        for (guid,instance) in self.instances:
            if guid==self.primaryInstanceGUID: f2.write(instance.desc.name+" "+hexlify(guid)+ " #primary instance\r\n")
            else: f2.write(instance.desc.name+" "+hexlify(guid)+ "\r\n")
            self.recurse(instance.fields,f2,0)
        f2.close()

    def recurse(self, fields,f2, lvl): #over fields
        lvl+=1
        for field in fields:
            if field.desc.type in (0xc14d, 0xc0fd, 0xc10d, 0xc0ed, 0xc0dd, 0xc0bd, 0xc0ad, 0x407d, 0x409d, 0x0089):
                f2.write(lvl*SEP+field.desc.name+" "+str(field.value)+"\r\n")
            elif field.desc.type == 0xc13d:
                f2.write(lvl*SEP+field.desc.name+" "+formatfloat(field.value)+"\r\n")
            elif field.desc.type == 0xc15d:
                f2.write(lvl*SEP+field.desc.name+" "+hexlify(field.value)+"\r\n")
            elif field.desc.type == 0x0035:
                towrite=""
                if field.value>>31:
                    extguid=self.externalGUIDs[field.value&0x7fffffff]
                    try: towrite=guidTable[extguid[0]]+"/"+hexlify(extguid[1])
                    except: towrite=hexlify(extguid[0])+"/"+hexlify(extguid[1])
                elif field.value==0: towrite="*nullGuid*"
                else: towrite=hexlify(self.internalGUIDs[field.value-1])
                f2.write(lvl*SEP+field.desc.name+" "+towrite+"\r\n")
            elif field.desc.type==0x0041 and len(field.value.fields)==0:
                f2.write(lvl*SEP+field.desc.name+" "+"*nullArray*"+"\r\n")
            else:
                f2.write(lvl*SEP+field.desc.name+"::"+field.value.desc.name+"\r\n")
                self.recurse(field.value.fields,f2,lvl)


    def decode(self):
        if not self.prim.desc.name=="SoundWaveAsset": return

        histogram=dict() #count the number of times each chunk is used by a variation to obtain the right index

        Chunks=[]
        for i in self.prim.get("$::SoundDataAsset/Chunks::array").fields:
            chnk=Stub()
            Chunks.append(chnk)
            chnk.ChunkId=i.value.get("ChunkId").value
            chnk.ChunkSize=i.value.get("ChunkSize").value

           
        variations=[i.link() for i in self.prim.get("Variations::array").fields]

        Variations=[]
       
        for var in variations:
            Variation=Stub()
            Variations.append(Variation)
            Variation.ChunkIndex=var.get("ChunkIndex").value
            Variation.SeekTablesSize=var.get("SeekTablesSize").value
            Variation.FirstLoopSegmentIndex=var.get("FirstLoopSegmentIndex").value
            Variation.LastLoopSegmentIndex=var.get("LastLoopSegmentIndex").value


            Variation.Segments=[]
            segs=var.get("Segments::array").fields
            for seg in segs:
                Segment=Stub()
                Variation.Segments.append(Segment)
                Segment.SamplesOffset = seg.value.get("SamplesOffset").value
                Segment.SeekTableOffset = seg.value.get("SeekTableOffset").value
                Segment.SegmentLength = seg.value.get("SegmentLength").value

            Variation.ChunkId=hexlify(Chunks[Variation.ChunkIndex].ChunkId)
            Variation.ChunkSize=Chunks[Variation.ChunkIndex].ChunkSize
       

            #find the appropriate index
            if Variation.ChunkIndex in histogram: #has been used previously already
                Variation.Index=histogram[Variation.ChunkIndex]
                histogram[Variation.ChunkIndex]+=1
            else:
                Variation.Index=0
                histogram[Variation.ChunkIndex]=1

               
        #everything is laid out neatly now
        #ChunkId, ChunkSize, Index, ChunkIndex, SeekTablesSize, FirstLoopSegmentIndex, LastLoopSegmentIndex
        #Segments with SamplesOffset, SeekTableOffset, SegmentLength

        ChunkHandles=dict() #should speed things up

        for Variation in Variations:
            try:
                f=ChunkHandles[Variation.ChunkId]
            except:
                try:
                    f=open2(chunkFolder+Variation.ChunkId+".chunk")
                except IOError:
                    try:
                        f=open2(chunkFolder2+Variation.ChunkId+".chunk")
                    except:
                        print "Chunk does not exist: "+Variation.ChunkId+" "+self.trueFilename
                        return
                ChunkHandles[Variation.ChunkId]=f


            for ijk in xrange(len(Variation.Segments)):
                Segment=Variation.Segments[ijk]
                f.seek(Segment.SamplesOffset)
                magic=f.read(4)
               
                if magic!="\x48\x00\x00\x0c":
                    raise Exception("Wrong XAS magic.")

                audioType=f.read(1) #0x14 is DICE XAS
                if audioType!="\x14":
                    print "Non XAS audio segment (type "+hexlify(audioType)+"): "+Variation.ChunkId+" "+self.trueFilename
                    continue
                channelRaw=f.read(1)
                if channelRaw not in channelDict:
                    print self.trueFilename
               
                   
                numChannels=channelDict[channelRaw]
                samplingRate=unpack(">H",f.read(2))[0] #[48000, 11025, 16000, 44100, 24000]
                f.read(4) #the first byte is always 0x40, the second takes one of 82 values, third and fourth can each take any value (i.e. 00 to ff)
                #82 values: ['\x01', '\x00', '\x03', '\x02', '\x05', '\x04', '\x07', '\x06', '\t', '\x08', '\x0b', '\n', '\r', '\x0c', '\x0f', '\x0e', '\x11', '\x10', '\x13', '\x12', '\x15', '\x14', '\x17', '\x16', '\x19', '\x18', '\x1b', '\x1a', '\x1d', '\x1c', '\x9f', '\x1e', '!', ' ', '#', '"', '%', '$', "'", '&', ')', '(', '+', '*', '-', ',', '/', '.', '1', '0', '3', '2', '5', '\x1f', '7', '6', '9', ';', '<', '?', '>', '\xc1', 'a', 'C', 'E', 'D', '\xa0', 'I', 'H', 'K', 'J', 'L', 'Y', 'X', '\x8e', '\x87', '4', 'A', 'd', 'F', '\x95', '\x7f']
               
                payload=""
                target= os.path.join(outputFolder,self.trueFilename)+" "+str(Variation.ChunkIndex)+" "+str(Variation.Index)+" "+str(ijk)+".wav"
                targetFolder=os.path.dirname(target)
                if not os.path.isdir(targetFolder): os.makedirs(targetFolder)

                f2=StringIO()
               
                write_header(f2, samplingRate, numChannels)
               
                while 1:
                    v1,length=unpack(">HH",f.read(4))

                    if length<=4: break
                    length-=8
                   
                    v2,v3=unpack(">HH",f.read(4))

                    if length%76!=0:
                        raise Exception("Strange length: "+str(length))
                       
                    atoms=length/76

                    blocks=[None]*numChannels
                   
                    for atom in xrange(0,atoms,numChannels):
                        for i in xrange(numChannels):
                            blocks[i]=decode(f.read(76))
                           
                        atomData=""
                        for i in xrange(128):
                            for j in xrange(numChannels):
                                atomData+=pack("f",blocks[j][i])
               
                        f2.write(atomData)

                finalPos=f2.tell()
                f2.seek(4)   
                f2.write(pack("I",finalPos-8))
                f2.seek(76)
                f2.write(pack("I",finalPos-76))

                f3=open2(target,"wb")
                f3.write(f2.getvalue())
               
                f2.close()

        for key in ChunkHandles:
            ChunkHandles[key].close()



def write_header(fp, samplingRate, nchannels):
    out = pack('4sl4s','RIFF',0,'WAVE')
    floatSize = 4
    fmt_size = 40
    tag = 0xFFFE
    etag = 3

    out += pack('4slHHllHH', 'fmt ', fmt_size, tag, nchannels, samplingRate,
                nchannels * samplingRate * floatSize, nchannels * floatSize, floatSize * 8)

    out += pack('HHlH14s', 22, floatSize * 8, (1 << nchannels) - 1, etag,
                '\x00\x00\x00\x00\x10\x00\x80\x00\x00\xaa\x008\x9b\x71')

    out += pack("<4sll","fact",4,floatSize)
    out += pack("4sl","data",0)

    fp.seek(0)
    fp.write(out)

   
xaslib = cdll.LoadLibrary("xas")
def decode(inputChars):
    charType = c_char * 76
    charBuf = charType(*inputChars)
    charbufpointer = pointer(charBuf)
   
    floatType = c_float * 128
    floatBuf = floatType()
    floatbufpointer = pointer(floatBuf)
   
    xaslib.decode(charbufpointer, floatbufpointer)
   
    return floatBuf


channelDict={"\x00":1,"\x04":2,"\x0c":4,"\x14":6,"\x1c":8,"\x05":2,"\x01":1}


if outputFolder[-1] not in ("/","\\"): outputFolder+="/"
if ebxFolder[-1] not in ("/","\\"): ebxFolder+="/"
if chunkFolder[-1] not in ("/","\\"): chunkFolder+="/"
if chunkFolder2[-1] not in ("/","\\"): chunkFolder2+="/"


   
decodeAudio()


Remember one thing guys: The script only works on BF3 PC version ( 100% ) and MOHW PC version ( 90% > Have to use Zench EALayer3 tool for some sounds ), for the moment. There's some troubles with console version as durandal217 and OrangeC said.

_________________
Vosvoy


Last edited by Vosvoy on Fri May 10, 2013 1:31 am, edited 4 times in total.

Top
 Profile  
 
 Post subject: Re: Frostbite 2 sound extraction research
PostPosted: Thu May 09, 2013 7:17 am 
Offline
double-veteran
double-veteran

Joined: Sun Apr 20, 2008 2:58 am
Posts: 824
Has thanked: 5 times
Have thanks: 32 times
To Convert the non-xas use the ealayer3 tool from zench.


Top
 Profile  
 
 Post subject: Re: Frostbite 2 sound extraction research
PostPosted: Thu May 09, 2013 8:25 pm 
Offline
veteran
User avatar

Joined: Tue Jul 17, 2012 3:52 am
Posts: 87
Has thanked: 15 times
Have thanks: 10 times
Frankelstner has updated the XAS script again to properly handle EALayer3 files for MOHW. Music now properly extracts.

Code:
#needs python 2.7
import string
import sys
from binascii import hexlify
from struct import unpack,pack
import os
from cStringIO import StringIO
import cProfile
import cPickle
import subprocess



ebxFolder=   r"D:\hexing\mohw dump\bundles\ebx" #audio is found in several places
chunkFolder= r"D:\hexing\mohw dump\chunks" #grab audio from here
chunkFolder2=r"D:\hexing\mohw dump\bundles\chunks" #if the chunk is not found in the first folder, use this one
outputFolder=r"D:\decoded mohw123123123"
ealayer3Path=r"D:\ealayer3-0.7.0-win32\ealayer3.exe" #https://bitbucket.org/Zenchreal/ealayer3/downloads


#Run through the sound ebx files, find fields with chunk Guids and fieldName = ChunkId.
#The script will overwrite existing files.

#The filename of the ebx file importing an audio chunk becomes the name of the wav file.
#As for the indices, there are three of them in the following order.
#1: Variation.ChunkIndex: Some ebx files import several completely independent audio chunk files. This index differentiates between them.
#2: Variation.Index: An ebx may use the same audio chunk for several sound variations, this index keeps them apart.
#3: Segment.Index: A variation may contain several segments, so there is another index.


##############################################################
##############################################################
def unpackBE(typ,data): return unpack(">"+typ,data)

def makeLongDirs(path):
    #create folders if necessary and return the file handle
    #first of all, create one folder level manully because makedirs might fail
    path=os.path.normpath(path)
    pathParts=path.split("\\")
    manualPart="\\".join(pathParts[:2])
    if not os.path.isdir(manualPart): os.makedirs(manualPart)
   
    #now handle the rest, including extra long path names
    folderPath=lp(os.path.dirname(path))
    if not os.path.isdir(folderPath): os.makedirs(folderPath)
   

def open2(path,mode="rb"):
    if mode=="wb": makeLongDirs(path)
    return open(lp(path),mode)

def lp(path): #long pathnames
    if len(path)<=247 or path=="" or path[:4]=='\\\\?\\': return path
    return unicode('\\\\?\\' + os.path.normpath(path))

def decodeAudio():
    for dir0, dirs, ff in os.walk(ebxFolder):
        for fname in ff:
            f=open2(dir0+"\\"+fname)
            magic=f.read(4)
            if magic=="\xCE\xD1\xB2\x0F":
                dbx=Dbx(f,unpack)
            elif magic=="\x0F\xB2\xD1\xCE":
                dbx=Dbx(f,unpackBE)
            else:
                f.close()
                continue
            dbx.decode()
           
try:
    from ctypes import *
    floatlib = cdll.LoadLibrary("floattostring")
    def formatfloat(num):
        bufType = c_char * 100
        buf = bufType()
        bufpointer = pointer(buf)
        floatlib.convertNum(c_double(num), bufpointer, 100)
        rawstring=(buf.raw)[:buf.raw.find("\x00")]
        if rawstring[:2]=="-.": return "-0."+rawstring[2:]
        elif rawstring[0]==".": return "0."+rawstring[1:]
        elif "e" not in rawstring and "." not in rawstring: return rawstring+".0"
        return rawstring
except:
    def formatfloat(num):
        return str(num)
def hasher(keyword): #32bit FNV-1 hash with FNV_offset_basis = 5381 and FNV_prime = 33
    hash = 5381
    for byte in keyword:
        hash = (hash*33) ^ ord(byte)
    return hash & 0xffffffff # use & because Python promotes the num instead of intended overflow
class Header:
    def __init__(self,varList): ##all 4byte unsigned integers
        self.absStringOffset     = varList[0]  ## absolute offset for string section start
        self.lenStringToEOF      = varList[1]  ## length from string section start to EOF
        self.numGUID             = varList[2]  ## number of external GUIDs
        self.null                = varList[3]  ## 00000000
        self.numInstanceRepeater = varList[4]
        self.numComplex          = varList[5]  ## number of complex entries
        self.numField            = varList[6]  ## number of field entries
        self.lenName             = varList[7]  ## length of name section including padding
        self.lenString           = varList[8]  ## length of string section including padding
        self.numArrayRepeater    = varList[9]
        self.lenPayload          = varList[10] ## length of normal payload section; the start of the array payload section is absStringOffset+lenString+lenPayload
class FieldDescriptor:
    def __init__(self,varList,keywordDict):
        self.name            = keywordDict[varList[0]]
        self.type            = varList[1]
        self.ref             = varList[2] #the field may contain another complex
        self.offset          = varList[3] #offset in payload section; relative to the complex containing it
        self.secondaryOffset = varList[4]
class ComplexDescriptor:
    def __init__(self,varList,keywordDict):
        self.name            = keywordDict[varList[0]]
        self.fieldStartIndex = varList[1] #the index of the first field belonging to the complex
        self.numField        = varList[2] #the total number of fields belonging to the complex
        self.alignment       = varList[3]
        self.type            = varList[4]
        self.size            = varList[5] #total length of the complex in the payload section
        self.secondarySize   = varList[6] #seems deprecated
class InstanceRepeater:
    def __init__(self,varList):
        self.null            = varList[0] #called "internalCount", seems to be always null
        self.repetitions     = varList[1] #number of instance repetitions
        self.complexIndex    = varList[2] #index of complex used as the instance
class arrayRepeater:
    def __init__(self,varList):
        self.offset          = varList[0] #offset in array payload section
        self.repetitions     = varList[1] #number of array repetitions
        self.complexIndex    = varList[2] #not necessary for extraction
class Complex:
    def __init__(self,desc,dbxhandle):
        self.desc=desc
        self.dbx=dbxhandle #lazy
    def get(self,name):
        pathElems=name.split("/")
        curPos=self
        if pathElems[-1].find("::")!=-1: #grab a complex
            for elem in pathElems:
                try:
                    curPos=curPos.go1(elem)
                except Exception,e:
                    raise Exception("Could not find complex with name: "+str(e)+"\nFull path: "+name+"\nFilename: "+self.dbx.trueFilename)
            return curPos
        #grab a field instead
        for elem in pathElems[:-1]:
            try:
                curPos=curPos.go1(elem)
            except Exception,e:
                raise Exception("Could not find complex with name: "+str(e)+"\nFull path: "+name+"\nFilename: "+self.dbx.trueFilename)
        for field in curPos.fields:
            if field.desc.name==pathElems[-1]:
                return field
           
        raise Exception("Could not find field with name: "+name+"\nFilename: "+self.dbx.trueFilename)

    def go1(self,name): #go once
        for field in self.fields:
            if field.desc.type in (0x0029, 0xd029,0x0000,0x0041):
                if field.desc.name+"::"+field.value.desc.name == name:
                    return field.value
        raise Exception(name)


class Field:
    def __init__(self,desc,dbx):
        self.desc=desc
        self.dbx=dbx
    def link(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)
       
        if self.value>>31:
            extguid=self.dbx.externalGUIDs[self.value&0x7fffffff]
           
            for existingDbx in dbxArray:
                if existingDbx.fileGUID==extguid[0]:
                    for guid, instance in existingDbx.instances:
                        if guid==extguid[1]:
                            return instance
                   

            f=valid(inputFolder+guidTable[extguid[0]]+".ebx")
##            print guidTable[extguid[0]]
            dbx=Dbx(f)
            dbxArray.append(dbx)
            for guid, instance in dbx.instances:
                if guid==extguid[1]:
                    return instance
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
        elif self.value!=0:
            for guid, instance in self.dbx.instances:
                if guid==self.dbx.internalGUIDs[self.value-1]:
                    return instance
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)

        raise Exception("Invalid link, could not find target.")

    def getlinkguid(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)

        if self.value>>31:
            return "".join(self.dbx.externalGUIDs[self.value&0x7fffffff])
        elif self.value!=0:
            return self.dbx.fileGUID+self.dbx.internalGUIDs[self.value-1]
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
    def getlinkname(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)

        if self.value>>31:
            return guidTable[self.dbx.externalGUIDs[self.value&0x7fffffff][0]]+"/"+self.dbx.externalGUIDs[self.value&0x7fffffff][1]
        elif self.value!=0:
            return self.dbx.trueFilename+"/"+self.dbx.internalGUIDs[self.value-1]
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
   

         
def valid(fname):
    f=open2(fname,"rb")
    if f.read(4) not in ("\xCE\xD1\xB2\x0F","\x0F\xB2\xD1\xCE"):
        f.close()
        raise Exception("nope")
    return f

class nullguid(Exception):
    def __init__(self, value):
        self.value = value
    def __str__(self):
        return repr(self.value)


numDict={0x0035:("I",4),0xc10d:("I",4),0xc14d:("d",8),0xc0ad:("?",1),0xc0fd:("i",4),0xc0bd:("B",1),0xc0ed:("h",2), 0xc0dd:("H",2), 0xc13d:("f",4)}


class Stub:
    pass



class Dbx:
    def __init__(self, f,unpacker):
        self.unpack=unpacker
        self.trueFilename=""
        self.header=Header(self.unpack("11I",f.read(44)))
        self.arraySectionstart=self.header.absStringOffset+self.header.lenString+self.header.lenPayload
        self.fileGUID, self.primaryInstanceGUID = f.read(16), f.read(16)   
        self.externalGUIDs=[(f.read(16),f.read(16)) for i in xrange(self.header.numGUID)]
        self.keywords=str.split(f.read(self.header.lenName),"\x00")
        self.keywordDict=dict((hasher(keyword),keyword) for keyword in self.keywords)
        self.fieldDescriptors=[FieldDescriptor(self.unpack("IHHII",f.read(16)), self.keywordDict) for i in xrange(self.header.numField)]
        self.complexDescriptors=[ComplexDescriptor(self.unpack("IIBBHHH",f.read(16)), self.keywordDict) for i in xrange(self.header.numComplex)]
        self.instanceRepeaters=[InstanceRepeater(self.unpack("3I",f.read(12))) for i in xrange(self.header.numInstanceRepeater)]
        while f.tell()%16!=0: f.seek(1,1) #padding
        self.arrayRepeaters=[arrayRepeater(self.unpack("3I",f.read(12))) for i in xrange(self.header.numArrayRepeater)]

        #payload
        f.seek(self.header.absStringOffset+self.header.lenString)
        self.internalGUIDs=[]
        self.instances=[] # (guid, complex)
        for instanceRepeater in self.instanceRepeaters:
            for repetition in xrange(instanceRepeater.repetitions):
                instanceGUID=f.read(16)
                self.internalGUIDs.append(instanceGUID)
                if instanceGUID==self.primaryInstanceGUID:
                    self.isPrimaryInstance=True
                else:
                    self.isPrimaryInstance=False
                inst=self.readComplex(instanceRepeater.complexIndex,f)
                inst.guid=instanceGUID
               
                if self.isPrimaryInstance: self.prim=inst
                self.instances.append((instanceGUID,inst))
   
        f.close()
       

    def readComplex(self, complexIndex,f):
        complexDesc=self.complexDescriptors[complexIndex]
        cmplx=Complex(complexDesc,self)
       
        startPos=f.tell()                 
        cmplx.fields=[]
        for fieldIndex in xrange(complexDesc.fieldStartIndex,complexDesc.fieldStartIndex+complexDesc.numField):
            f.seek(startPos+self.fieldDescriptors[fieldIndex].offset)
            cmplx.fields.append(self.readField(fieldIndex,f))
       
        f.seek(startPos+complexDesc.size)
        return cmplx


    def readField(self,fieldIndex,f):
        fieldDesc = self.fieldDescriptors[fieldIndex]
        field=Field(fieldDesc,self)
       
        if fieldDesc.type in (0x0029, 0xd029,0x0000):
            field.value=self.readComplex(fieldDesc.ref,f)
        elif fieldDesc.type==0x0041:
            arrayRepeater=self.arrayRepeaters[self.unpack("I",f.read(4))[0]]
            arrayComplexDesc=self.complexDescriptors[fieldDesc.ref]

##            if arrayRepeater.repetitions==0: field.value = "*nullArray*"
            f.seek(self.arraySectionstart+arrayRepeater.offset)
            arrayComplex=Complex(arrayComplexDesc,self)
            arrayComplex.fields=[self.readField(arrayComplexDesc.fieldStartIndex,f) for repetition in xrange(arrayRepeater.repetitions)]
            field.value=arrayComplex
           
        elif fieldDesc.type in (0x407d, 0x409d):
            startPos=f.tell()
            f.seek(self.header.absStringOffset+self.unpack("I",f.read(4))[0])
            string=""
            while 1:
                a=f.read(1)
                if a=="\x00": break
                else: string+=a
            f.seek(startPos+4)
           
            if len(string)==0: field.value="*nullString*" #actually the string is ""
            else: field.value=string
           
            if self.isPrimaryInstance and self.trueFilename=="" and fieldDesc.name=="Name": self.trueFilename=string
           
                   
        elif fieldDesc.type==0x0089: #incomplete implementation, only gives back the selected string
            compareValue=self.unpack("I",f.read(4))[0]
            enumComplex=self.complexDescriptors[fieldDesc.ref]

            if enumComplex.numField==0:
                field.value="*nullEnum*"
            for fieldIndex in xrange(enumComplex.fieldStartIndex,enumComplex.fieldStartIndex+enumComplex.numField):
                if self.fieldDescriptors[fieldIndex].offset==compareValue:
                    field.value=self.fieldDescriptors[fieldIndex].name
                    break
        elif fieldDesc.type==0xc15d:
            field.value=f.read(16)
        elif fieldDesc.type == 0xc13d: ################################
            field.value=formatfloat(self.unpack("f",f.read(4))[0])
        else:
            (typ,length)=numDict[fieldDesc.type]
            num=self.unpack(typ,f.read(length))[0]
            field.value=num
       
        return field
       

    def dump(self,outputFolder):
        dirName=os.path.dirname(outputFolder+self.trueFilename)
        if not os.path.isdir(dirName): os.makedirs(dirName)
        f2=open2(outputFolder+self.trueFilename+EXTENSION,"wb")
        print self.trueFilename
       
        for (guid,instance) in self.instances:
            if guid==self.primaryInstanceGUID: f2.write(instance.desc.name+" "+hexlify(guid)+ " #primary instance\r\n")
            else: f2.write(instance.desc.name+" "+hexlify(guid)+ "\r\n")
            self.recurse(instance.fields,f2,0)
        f2.close()

    def recurse(self, fields,f2, lvl): #over fields
        lvl+=1
        for field in fields:
            if field.desc.type in (0xc14d, 0xc0fd, 0xc10d, 0xc0ed, 0xc0dd, 0xc0bd, 0xc0ad, 0x407d, 0x409d, 0x0089):
                f2.write(lvl*SEP+field.desc.name+" "+str(field.value)+"\r\n")
            elif field.desc.type == 0xc13d:
                f2.write(lvl*SEP+field.desc.name+" "+formatfloat(field.value)+"\r\n")
            elif field.desc.type == 0xc15d:
                f2.write(lvl*SEP+field.desc.name+" "+hexlify(field.value)+"\r\n")
            elif field.desc.type == 0x0035:
                towrite=""
                if field.value>>31:
                    extguid=self.externalGUIDs[field.value&0x7fffffff]
                    try: towrite=guidTable[extguid[0]]+"/"+hexlify(extguid[1])
                    except: towrite=hexlify(extguid[0])+"/"+hexlify(extguid[1])
                elif field.value==0: towrite="*nullGuid*"
                else: towrite=hexlify(self.internalGUIDs[field.value-1])
                f2.write(lvl*SEP+field.desc.name+" "+towrite+"\r\n")
            elif field.desc.type==0x0041 and len(field.value.fields)==0:
                f2.write(lvl*SEP+field.desc.name+" "+"*nullArray*"+"\r\n")
            else:
                f2.write(lvl*SEP+field.desc.name+"::"+field.value.desc.name+"\r\n")
                self.recurse(field.value.fields,f2,lvl)


    def decode(self):
        if not self.prim.desc.name=="SoundWaveAsset": return

        histogram=dict() #count the number of times each chunk is used by a variation to obtain the right index

        Chunks=[]
        for i in self.prim.get("$::SoundDataAsset/Chunks::array").fields:
            chnk=Stub()
            Chunks.append(chnk)
            chnk.ChunkId=i.value.get("ChunkId").value
            chnk.ChunkSize=i.value.get("ChunkSize").value

           
        variations=[i.link() for i in self.prim.get("Variations::array").fields]

        Variations=[]
       
        for var in variations:
            Variation=Stub()
            Variations.append(Variation)
            Variation.ChunkIndex=var.get("ChunkIndex").value
            Variation.SeekTablesSize=var.get("SeekTablesSize").value
            Variation.FirstLoopSegmentIndex=var.get("FirstLoopSegmentIndex").value
            Variation.LastLoopSegmentIndex=var.get("LastLoopSegmentIndex").value


            Variation.Segments=[]
            segs=var.get("Segments::array").fields
            for seg in segs:
                Segment=Stub()
                Variation.Segments.append(Segment)
                Segment.SamplesOffset = seg.value.get("SamplesOffset").value
                Segment.SeekTableOffset = seg.value.get("SeekTableOffset").value
                Segment.SegmentLength = seg.value.get("SegmentLength").value

            Variation.ChunkId=hexlify(Chunks[Variation.ChunkIndex].ChunkId)
            Variation.ChunkSize=Chunks[Variation.ChunkIndex].ChunkSize
       

            #find the appropriate index
            if Variation.ChunkIndex in histogram: #has been used previously already
                Variation.Index=histogram[Variation.ChunkIndex]
                histogram[Variation.ChunkIndex]+=1
            else:
                Variation.Index=0
                histogram[Variation.ChunkIndex]=1

               
        #everything is laid out neatly now
        #ChunkId, ChunkSize, Index, ChunkIndex, SeekTablesSize, FirstLoopSegmentIndex, LastLoopSegmentIndex
        #Segments with SamplesOffset, SeekTableOffset, SegmentLength

        ChunkHandles=dict() #for each ebx, keep track of all file handles

        for Variation in Variations:
            try:
                f=ChunkHandles[Variation.ChunkId]
            except:
                try:
                    f=open2(chunkFolder+Variation.ChunkId+".chunk")
                    currentChunkName=chunkFolder+Variation.ChunkId+".chunk"
                except IOError:
                    try:
                        f=open2(chunkFolder2+Variation.ChunkId+".chunk")
                        currentChunkName=chunkFolder2+Variation.ChunkId+".chunk"
                    except:
                        print "Chunk does not exist: "+Variation.ChunkId+" "+self.trueFilename
                        return
                ChunkHandles[Variation.ChunkId]=f


            for ijk in xrange(len(Variation.Segments)):
                Segment=Variation.Segments[ijk]
                f.seek(Segment.SamplesOffset)
                magic=f.read(4)
               
                if magic!="\x48\x00\x00\x0c":
                    raise Exception("Wrong XAS magic.")

                audioType=f.read(1) #0x14 is DICE XAS, 0x16 is ealayer3, 0x12 is unknown
                if audioType=="\x16":
                    if not ealayerSupport:
                        print "EALayer3 segment not converted as the tool is missing: "+Variation.ChunkId+" "+self.trueFilename
                        continue
                    target=os.path.join(outputFolder,self.trueFilename)+" "+str(Variation.ChunkIndex)+" "+str(Variation.Index)+" "+str(ijk)+".mp3"
                    makeLongDirs(target) #prepare the folder structure

                    process = subprocess.Popen([ealayer3Path,currentChunkName,"-i",str(Segment.SamplesOffset),"-o",target], stderr=subprocess.PIPE,startupinfo=startupinfo)
                   
                    process.communicate() #this should set the returncode
                    if process.returncode:
                        print process.stderr.readlines()

                    continue
                elif audioType!="\x14":
                    print "Unknown audio segment (type "+hexlify(audioType)+"): "+Variation.ChunkId+" "+self.trueFilename
                    continue

               

                channelRaw=f.read(1)
                if channelRaw not in channelDict:
                    print self.trueFilename
               
                   
                numChannels=channelDict[channelRaw]
                samplingRate=unpack(">H",f.read(2))[0] #[48000, 11025, 16000, 44100, 24000]
                f.read(4) #the first byte is always 0x40, the second takes one of 82 values, third and fourth can each take any value (i.e. 00 to ff)
                #82 values: ['\x01', '\x00', '\x03', '\x02', '\x05', '\x04', '\x07', '\x06', '\t', '\x08', '\x0b', '\n', '\r', '\x0c', '\x0f', '\x0e', '\x11', '\x10', '\x13', '\x12', '\x15', '\x14', '\x17', '\x16', '\x19', '\x18', '\x1b', '\x1a', '\x1d', '\x1c', '\x9f', '\x1e', '!', ' ', '#', '"', '%', '$', "'", '&', ')', '(', '+', '*', '-', ',', '/', '.', '1', '0', '3', '2', '5', '\x1f', '7', '6', '9', ';', '<', '?', '>', '\xc1', 'a', 'C', 'E', 'D', '\xa0', 'I', 'H', 'K', 'J', 'L', 'Y', 'X', '\x8e', '\x87', '4', 'A', 'd', 'F', '\x95', '\x7f']
               
                payload=""
                target= os.path.join(outputFolder,self.trueFilename)+" "+str(Variation.ChunkIndex)+" "+str(Variation.Index)+" "+str(ijk)+".wav"
                targetFolder=os.path.dirname(target)
                if not os.path.isdir(targetFolder): os.makedirs(targetFolder)

                f2=StringIO()
               
                write_header(f2, samplingRate, numChannels)
               
                while 1:
                    v1,length=unpack(">HH",f.read(4))

                    if length<=4: break
                    length-=8
                   
                    v2,v3=unpack(">HH",f.read(4))

                    if length%76!=0:
                        raise Exception("Strange length: "+str(length))
                       
                    atoms=length/76

                    blocks=[None]*numChannels
                   
                    for atom in xrange(0,atoms,numChannels):
                        for i in xrange(numChannels):
                            blocks[i]=decode(f.read(76))
                           
                        atomData=""
                        for i in xrange(128):
                            for j in xrange(numChannels):
                                atomData+=pack("f",blocks[j][i])
               
                        f2.write(atomData)

                finalPos=f2.tell()
                f2.seek(4)   
                f2.write(pack("I",finalPos-8))
                f2.seek(76)
                f2.write(pack("I",finalPos-76))

                f3=open2(target,"wb")
                f3.write(f2.getvalue())
               
                f2.close()

        for key in ChunkHandles:
            ChunkHandles[key].close()



def write_header(fp, samplingRate, nchannels):
    out = pack('4sl4s','RIFF',0,'WAVE')
    floatSize = 4
    fmt_size = 40
    tag = 0xFFFE
    etag = 3

    out += pack('4slHHllHH', 'fmt ', fmt_size, tag, nchannels, samplingRate,
                nchannels * samplingRate * floatSize, nchannels * floatSize, floatSize * 8)

    out += pack('HHlH14s', 22, floatSize * 8, (1 << nchannels) - 1, etag,
                '\x00\x00\x00\x00\x10\x00\x80\x00\x00\xaa\x008\x9b\x71')

    out += pack("<4sll","fact",4,floatSize)
    out += pack("4sl","data",0)

    fp.seek(0)
    fp.write(out)

   
xaslib = cdll.LoadLibrary("xas")
def decode(inputChars):
    charType = c_char * 76
    charBuf = charType(*inputChars)
    charbufpointer = pointer(charBuf)
   
    floatType = c_float * 128
    floatBuf = floatType()
    floatbufpointer = pointer(floatBuf)
   
    xaslib.decode(charbufpointer, floatbufpointer)
   
    return floatBuf


channelDict={"\x00":1,"\x04":2,"\x0c":4,"\x14":6,"\x1c":8,"\x05":2,"\x01":1}


if outputFolder[-1] not in ("/","\\"): outputFolder+="/"
if ebxFolder[-1] not in ("/","\\"): ebxFolder+="/"
if chunkFolder[-1] not in ("/","\\"): chunkFolder+="/"
if chunkFolder2[-1] not in ("/","\\"): chunkFolder2+="/"


#for ealayer. By default Python opens a new window for a split second and puts focus on it. This info makes no window show up at all.
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
startupinfo.wShowWindow = subprocess.SW_HIDE


try:
    subprocess.Popen([ealayer3Path],startupinfo=startupinfo)
    ealayerSupport=True
    print "EALayer3 tool detected."
except WindowsError:
    ealayerSupport=False
    print "EALayer3 tool not detected."

   
decodeAudio()


Top
 Profile  
 
 Post subject: Re: Frostbite 2 sound extraction research
PostPosted: Thu May 09, 2013 9:01 pm 
Offline
veteran
User avatar

Joined: Fri Feb 18, 2011 4:58 pm
Posts: 125
Has thanked: 15 times
Have thanks: 14 times
That's god damn good.

Still have issues with Xbox360 version by the way?

_________________
Vosvoy


Top
 Profile  
 
 Post subject: Re: Frostbite 2 sound extraction research
PostPosted: Fri May 10, 2013 12:03 am 
Offline
veteran
User avatar

Joined: Tue Jul 17, 2012 3:52 am
Posts: 87
Has thanked: 15 times
Have thanks: 10 times
Vosvoy wrote:
That's god damn good.

Still have issues with Xbox360 version by the way?


I decided to skip the 360 version, nothing was matching up. I just asked a friend if I could copy his files from the pc version and use that, he was cool with it, if not perplexed as to why I wanted just the files for a game we didn't even play anymore.

But he was cool with it, and I got what I needed.


Top
 Profile  
 
 Post subject: Re: Frostbite 2 sound extraction research
PostPosted: Fri May 10, 2013 12:31 am 
Offline
double-veteran
double-veteran

Joined: Sun Apr 20, 2008 2:58 am
Posts: 824
Has thanked: 5 times
Have thanks: 32 times
Update

Testing now Army of two devils cartel the PS3 version as it uses EAlayer3. Works for the most part but has problems handling multichannel files.

EDIT: Okay seems i was using wrong arguments. The proper argument from ealayer3 to use is -S all" but i have no way of putting that properly into the python script in this line.

Code:
[ealayer3Path,currentChunkName,"-i",str(Segment.SamplesOffset),"-o",target], stderr=subprocess.PIPE,startupinfo=startupinfo)


any help?


Top
 Profile  
 
 Post subject: Re: Frostbite 2 sound extraction research
PostPosted: Fri May 10, 2013 2:57 am 
Offline
veteran
User avatar

Joined: Fri Feb 18, 2011 4:58 pm
Posts: 125
Has thanked: 15 times
Have thanks: 14 times
All right. I believe that this time is the right one for console users. As OrangeC said: Frankelstner updated his script that can handle console FB2 games ( Army of Two: Devil's Cartel at least ) and seems to be functional. Here is the last script:

Code:
#needs python 2.7
import string
import sys
from binascii import hexlify
from struct import unpack,pack
import os
from cStringIO import StringIO
import cProfile
import cPickle
import subprocess



ebxFolder=   r"D:\MOHWDUMP\bundles\ebx\mohw\audio" #audio is found in several places
chunkFolder= r"D:\MOHWDUMP\chunks" #grab audio from here
chunkFolder2=r"D:\MOHWDUMP\bundles\chunks" #if the chunk is not found in the first folder, use this one
outputFolder=r"D:\MOHWSOUNDS"
ealayer3Path=r"D:\ealayer3-0.7.0-win32\ealayer3.exe" #https://bitbucket.org/Zenchreal/ealayer3/downloads

ChunkIdObfuscation=False #set to True for some console games when the script does not find any chunks at all
obfuscationPermutation=[3,2,1,0,5,4,7,6,8,9,10,11,12,13,14,15]

#Run through the sound ebx files, find fields with chunk Guids and fieldName = ChunkId.
#The script will overwrite existing files.

#The filename of the ebx file importing an audio chunk becomes the name of the wav file.
#As for the indices, there are three of them in the following order.
#1: Variation.ChunkIndex: Some ebx files import several completely independent audio chunk files. This index differentiates between them.
#2: Variation.Index: An ebx may use the same audio chunk for several sound variations, this index keeps them apart.
#3: Segment.Index: A variation may contain several segments, so there is another index.


##############################################################
##############################################################
def unpackBE(typ,data): return unpack(">"+typ,data)

def makeLongDirs(path):
    #create folders if necessary and return the file handle
    #first of all, create one folder level manully because makedirs might fail
    path=os.path.normpath(path)
    pathParts=path.split("\\")
    manualPart="\\".join(pathParts[:2])
    if not os.path.isdir(manualPart): os.makedirs(manualPart)
   
    #now handle the rest, including extra long path names
    folderPath=lp(os.path.dirname(path))
    if not os.path.isdir(folderPath): os.makedirs(folderPath)
   

def open2(path,mode="rb"):
    if mode=="wb": makeLongDirs(path)
    return open(lp(path),mode)

def lp(path): #long pathnames
    if len(path)<=247 or path=="" or path[:4]=='\\\\?\\': return path
    return unicode('\\\\?\\' + os.path.normpath(path))

def decodeAudio():
    for dir0, dirs, ff in os.walk(ebxFolder):
        for fname in ff:
            f=open2(dir0+"\\"+fname)
            magic=f.read(4)
            if magic=="\xCE\xD1\xB2\x0F":
                dbx=Dbx(f,unpack)
            elif magic=="\x0F\xB2\xD1\xCE":
                dbx=Dbx(f,unpackBE)
            else:
                f.close()
                continue
            dbx.decode()
           
try:
    from ctypes import *
    floatlib = cdll.LoadLibrary("floattostring")
    def formatfloat(num):
        bufType = c_char * 100
        buf = bufType()
        bufpointer = pointer(buf)
        floatlib.convertNum(c_double(num), bufpointer, 100)
        rawstring=(buf.raw)[:buf.raw.find("\x00")]
        if rawstring[:2]=="-.": return "-0."+rawstring[2:]
        elif rawstring[0]==".": return "0."+rawstring[1:]
        elif "e" not in rawstring and "." not in rawstring: return rawstring+".0"
        return rawstring
except:
    def formatfloat(num):
        return str(num)
def hasher(keyword): #32bit FNV-1 hash with FNV_offset_basis = 5381 and FNV_prime = 33
    hash = 5381
    for byte in keyword:
        hash = (hash*33) ^ ord(byte)
    return hash & 0xffffffff # use & because Python promotes the num instead of intended overflow
class Header:
    def __init__(self,varList): ##all 4byte unsigned integers
        self.absStringOffset     = varList[0]  ## absolute offset for string section start
        self.lenStringToEOF      = varList[1]  ## length from string section start to EOF
        self.numGUID             = varList[2]  ## number of external GUIDs
        self.null                = varList[3]  ## 00000000
        self.numInstanceRepeater = varList[4]
        self.numComplex          = varList[5]  ## number of complex entries
        self.numField            = varList[6]  ## number of field entries
        self.lenName             = varList[7]  ## length of name section including padding
        self.lenString           = varList[8]  ## length of string section including padding
        self.numArrayRepeater    = varList[9]
        self.lenPayload          = varList[10] ## length of normal payload section; the start of the array payload section is absStringOffset+lenString+lenPayload
class FieldDescriptor:
    def __init__(self,varList,keywordDict):
        self.name            = keywordDict[varList[0]]
        self.type            = varList[1]
        self.ref             = varList[2] #the field may contain another complex
        self.offset          = varList[3] #offset in payload section; relative to the complex containing it
        self.secondaryOffset = varList[4]
class ComplexDescriptor:
    def __init__(self,varList,keywordDict):
        self.name            = keywordDict[varList[0]]
        self.fieldStartIndex = varList[1] #the index of the first field belonging to the complex
        self.numField        = varList[2] #the total number of fields belonging to the complex
        self.alignment       = varList[3]
        self.type            = varList[4]
        self.size            = varList[5] #total length of the complex in the payload section
        self.secondarySize   = varList[6] #seems deprecated
class InstanceRepeater:
    def __init__(self,varList):
        self.null            = varList[0] #called "internalCount", seems to be always null
        self.repetitions     = varList[1] #number of instance repetitions
        self.complexIndex    = varList[2] #index of complex used as the instance
class arrayRepeater:
    def __init__(self,varList):
        self.offset          = varList[0] #offset in array payload section
        self.repetitions     = varList[1] #number of array repetitions
        self.complexIndex    = varList[2] #not necessary for extraction
class Complex:
    def __init__(self,desc,dbxhandle):
        self.desc=desc
        self.dbx=dbxhandle #lazy
    def get(self,name):
        pathElems=name.split("/")
        curPos=self
        if pathElems[-1].find("::")!=-1: #grab a complex
            for elem in pathElems:
                try:
                    curPos=curPos.go1(elem)
                except Exception,e:
                    raise Exception("Could not find complex with name: "+str(e)+"\nFull path: "+name+"\nFilename: "+self.dbx.trueFilename)
            return curPos
        #grab a field instead
        for elem in pathElems[:-1]:
            try:
                curPos=curPos.go1(elem)
            except Exception,e:
                raise Exception("Could not find complex with name: "+str(e)+"\nFull path: "+name+"\nFilename: "+self.dbx.trueFilename)
        for field in curPos.fields:
            if field.desc.name==pathElems[-1]:
                return field
           
        raise Exception("Could not find field with name: "+name+"\nFilename: "+self.dbx.trueFilename)

    def go1(self,name): #go once
        for field in self.fields:
            if field.desc.type in (0x0029, 0xd029,0x0000,0x0041):
                if field.desc.name+"::"+field.value.desc.name == name:
                    return field.value
        raise Exception(name)


class Field:
    def __init__(self,desc,dbx):
        self.desc=desc
        self.dbx=dbx
    def link(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)
       
        if self.value>>31:
            extguid=self.dbx.externalGUIDs[self.value&0x7fffffff]
           
            for existingDbx in dbxArray:
                if existingDbx.fileGUID==extguid[0]:
                    for guid, instance in existingDbx.instances:
                        if guid==extguid[1]:
                            return instance
                   

            f=valid(inputFolder+guidTable[extguid[0]]+".ebx")
##            print guidTable[extguid[0]]
            dbx=Dbx(f)
            dbxArray.append(dbx)
            for guid, instance in dbx.instances:
                if guid==extguid[1]:
                    return instance
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
        elif self.value!=0:
            for guid, instance in self.dbx.instances:
                if guid==self.dbx.internalGUIDs[self.value-1]:
                    return instance
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)

        raise Exception("Invalid link, could not find target.")

    def getlinkguid(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)

        if self.value>>31:
            return "".join(self.dbx.externalGUIDs[self.value&0x7fffffff])
        elif self.value!=0:
            return self.dbx.fileGUID+self.dbx.internalGUIDs[self.value-1]
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
    def getlinkname(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)

        if self.value>>31:
            return guidTable[self.dbx.externalGUIDs[self.value&0x7fffffff][0]]+"/"+self.dbx.externalGUIDs[self.value&0x7fffffff][1]
        elif self.value!=0:
            return self.dbx.trueFilename+"/"+self.dbx.internalGUIDs[self.value-1]
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
   

         
def valid(fname):
    f=open2(fname,"rb")
    if f.read(4) not in ("\xCE\xD1\xB2\x0F","\x0F\xB2\xD1\xCE"):
        f.close()
        raise Exception("nope")
    return f

class nullguid(Exception):
    def __init__(self, value):
        self.value = value
    def __str__(self):
        return repr(self.value)


numDict={0x0035:("I",4),0xc10d:("I",4),0xc14d:("d",8),0xc0ad:("?",1),0xc0fd:("i",4),0xc0bd:("B",1),0xc0ed:("h",2), 0xc0dd:("H",2), 0xc13d:("f",4)}


class Stub:
    pass



class Dbx:
    def __init__(self, f,unpacker):
        self.unpack=unpacker
        self.trueFilename=""
        self.header=Header(self.unpack("11I",f.read(44)))
        self.arraySectionstart=self.header.absStringOffset+self.header.lenString+self.header.lenPayload
        self.fileGUID, self.primaryInstanceGUID = f.read(16), f.read(16)   
        self.externalGUIDs=[(f.read(16),f.read(16)) for i in xrange(self.header.numGUID)]
        self.keywords=str.split(f.read(self.header.lenName),"\x00")
        self.keywordDict=dict((hasher(keyword),keyword) for keyword in self.keywords)
        self.fieldDescriptors=[FieldDescriptor(self.unpack("IHHII",f.read(16)), self.keywordDict) for i in xrange(self.header.numField)]
        self.complexDescriptors=[ComplexDescriptor(self.unpack("IIBBHHH",f.read(16)), self.keywordDict) for i in xrange(self.header.numComplex)]
        self.instanceRepeaters=[InstanceRepeater(self.unpack("3I",f.read(12))) for i in xrange(self.header.numInstanceRepeater)]
        while f.tell()%16!=0: f.seek(1,1) #padding
        self.arrayRepeaters=[arrayRepeater(self.unpack("3I",f.read(12))) for i in xrange(self.header.numArrayRepeater)]

        #payload
        f.seek(self.header.absStringOffset+self.header.lenString)
        self.internalGUIDs=[]
        self.instances=[] # (guid, complex)
        for instanceRepeater in self.instanceRepeaters:
            for repetition in xrange(instanceRepeater.repetitions):
                instanceGUID=f.read(16)
                self.internalGUIDs.append(instanceGUID)
                if instanceGUID==self.primaryInstanceGUID:
                    self.isPrimaryInstance=True
                else:
                    self.isPrimaryInstance=False
                inst=self.readComplex(instanceRepeater.complexIndex,f)
                inst.guid=instanceGUID
               
                if self.isPrimaryInstance: self.prim=inst
                self.instances.append((instanceGUID,inst))
   
        f.close()
       

    def readComplex(self, complexIndex,f):
        complexDesc=self.complexDescriptors[complexIndex]
        cmplx=Complex(complexDesc,self)
       
        startPos=f.tell()                 
        cmplx.fields=[]
        for fieldIndex in xrange(complexDesc.fieldStartIndex,complexDesc.fieldStartIndex+complexDesc.numField):
            f.seek(startPos+self.fieldDescriptors[fieldIndex].offset)
            cmplx.fields.append(self.readField(fieldIndex,f))
       
        f.seek(startPos+complexDesc.size)
        return cmplx


    def readField(self,fieldIndex,f):
        fieldDesc = self.fieldDescriptors[fieldIndex]
        field=Field(fieldDesc,self)
       
        if fieldDesc.type in (0x0029, 0xd029,0x0000):
            field.value=self.readComplex(fieldDesc.ref,f)
        elif fieldDesc.type==0x0041:
            arrayRepeater=self.arrayRepeaters[self.unpack("I",f.read(4))[0]]
            arrayComplexDesc=self.complexDescriptors[fieldDesc.ref]

##            if arrayRepeater.repetitions==0: field.value = "*nullArray*"
            f.seek(self.arraySectionstart+arrayRepeater.offset)
            arrayComplex=Complex(arrayComplexDesc,self)
            arrayComplex.fields=[self.readField(arrayComplexDesc.fieldStartIndex,f) for repetition in xrange(arrayRepeater.repetitions)]
            field.value=arrayComplex
           
        elif fieldDesc.type in (0x407d, 0x409d):
            startPos=f.tell()
            f.seek(self.header.absStringOffset+self.unpack("I",f.read(4))[0])
            string=""
            while 1:
                a=f.read(1)
                if a=="\x00": break
                else: string+=a
            f.seek(startPos+4)
           
            if len(string)==0: field.value="*nullString*" #actually the string is ""
            else: field.value=string
           
            if self.isPrimaryInstance and self.trueFilename=="" and fieldDesc.name=="Name": self.trueFilename=string
           
                   
        elif fieldDesc.type==0x0089: #incomplete implementation, only gives back the selected string
            compareValue=self.unpack("I",f.read(4))[0]
            enumComplex=self.complexDescriptors[fieldDesc.ref]

            if enumComplex.numField==0:
                field.value="*nullEnum*"
            for fieldIndex in xrange(enumComplex.fieldStartIndex,enumComplex.fieldStartIndex+enumComplex.numField):
                if self.fieldDescriptors[fieldIndex].offset==compareValue:
                    field.value=self.fieldDescriptors[fieldIndex].name
                    break
        elif fieldDesc.type==0xc15d:
            field.value=f.read(16)
        elif fieldDesc.type == 0xc13d: ################################
            field.value=formatfloat(self.unpack("f",f.read(4))[0])
        else:
            (typ,length)=numDict[fieldDesc.type]
            num=self.unpack(typ,f.read(length))[0]
            field.value=num
       
        return field
       

    def dump(self,outputFolder):
        dirName=os.path.dirname(outputFolder+self.trueFilename)
        if not os.path.isdir(dirName): os.makedirs(dirName)
        f2=open2(outputFolder+self.trueFilename+EXTENSION,"wb")
        print self.trueFilename
       
        for (guid,instance) in self.instances:
            if guid==self.primaryInstanceGUID: f2.write(instance.desc.name+" "+hexlify(guid)+ " #primary instance\r\n")
            else: f2.write(instance.desc.name+" "+hexlify(guid)+ "\r\n")
            self.recurse(instance.fields,f2,0)
        f2.close()

    def recurse(self, fields,f2, lvl): #over fields
        lvl+=1
        for field in fields:
            if field.desc.type in (0xc14d, 0xc0fd, 0xc10d, 0xc0ed, 0xc0dd, 0xc0bd, 0xc0ad, 0x407d, 0x409d, 0x0089):
                f2.write(lvl*SEP+field.desc.name+" "+str(field.value)+"\r\n")
            elif field.desc.type == 0xc13d:
                f2.write(lvl*SEP+field.desc.name+" "+formatfloat(field.value)+"\r\n")
            elif field.desc.type == 0xc15d:
                f2.write(lvl*SEP+field.desc.name+" "+hexlify(field.value)+"\r\n")
            elif field.desc.type == 0x0035:
                towrite=""
                if field.value>>31:
                    extguid=self.externalGUIDs[field.value&0x7fffffff]
                    try: towrite=guidTable[extguid[0]]+"/"+hexlify(extguid[1])
                    except: towrite=hexlify(extguid[0])+"/"+hexlify(extguid[1])
                elif field.value==0: towrite="*nullGuid*"
                else: towrite=hexlify(self.internalGUIDs[field.value-1])
                f2.write(lvl*SEP+field.desc.name+" "+towrite+"\r\n")
            elif field.desc.type==0x0041 and len(field.value.fields)==0:
                f2.write(lvl*SEP+field.desc.name+" "+"*nullArray*"+"\r\n")
            else:
                f2.write(lvl*SEP+field.desc.name+"::"+field.value.desc.name+"\r\n")
                self.recurse(field.value.fields,f2,lvl)


    def decode(self):
        if not self.prim.desc.name=="SoundWaveAsset": return

        histogram=dict() #count the number of times each chunk is used by a variation to obtain the right index

        Chunks=[]
        for i in self.prim.get("$::SoundDataAsset/Chunks::array").fields:
            chnk=Stub()
            Chunks.append(chnk)
            chnk.ChunkId=i.value.get("ChunkId").value
            chnk.ChunkSize=i.value.get("ChunkSize").value

           
        variations=[i.link() for i in self.prim.get("Variations::array").fields]

        Variations=[]
       
        for var in variations:
            Variation=Stub()
            Variations.append(Variation)
            Variation.ChunkIndex=var.get("ChunkIndex").value
            Variation.SeekTablesSize=var.get("SeekTablesSize").value
            Variation.FirstLoopSegmentIndex=var.get("FirstLoopSegmentIndex").value
            Variation.LastLoopSegmentIndex=var.get("LastLoopSegmentIndex").value


            Variation.Segments=[]
            segs=var.get("Segments::array").fields
            for seg in segs:
                Segment=Stub()
                Variation.Segments.append(Segment)
                Segment.SamplesOffset = seg.value.get("SamplesOffset").value
                Segment.SeekTableOffset = seg.value.get("SeekTableOffset").value
                Segment.SegmentLength = seg.value.get("SegmentLength").value

            Variation.ChunkId=hexlify(Chunks[Variation.ChunkIndex].ChunkId)
            Variation.ChunkSize=Chunks[Variation.ChunkIndex].ChunkSize
       

            #find the appropriate index
            if Variation.ChunkIndex in histogram: #has been used previously already
                Variation.Index=histogram[Variation.ChunkIndex]
                histogram[Variation.ChunkIndex]+=1
            else:
                Variation.Index=0
                histogram[Variation.ChunkIndex]=1

               
        #everything is laid out neatly now
        #ChunkId, ChunkSize, Index, ChunkIndex, SeekTablesSize, FirstLoopSegmentIndex, LastLoopSegmentIndex
        #Segments with SamplesOffset, SeekTableOffset, SegmentLength

        ChunkHandles=dict() #for each ebx, keep track of all file handles

        for Variation in Variations:
            try:
                f=ChunkHandles[Variation.ChunkId]
            except:
                try:
                    f=open2(chunkFolder+Variation.ChunkId+".chunk")
                    currentChunkName=chunkFolder+Variation.ChunkId+".chunk"
                except IOError:
                    try:
                        f=open2(chunkFolder2+Variation.ChunkId+".chunk")
                        currentChunkName=chunkFolder2+Variation.ChunkId+".chunk"
                    except:
                        print "Chunk does not exist: "+Variation.ChunkId+" "+self.trueFilename
                        return
                ChunkHandles[Variation.ChunkId]=f


            for ijk in xrange(len(Variation.Segments)):
                Segment=Variation.Segments[ijk]
                f.seek(Segment.SamplesOffset)
                magic=f.read(4)
               
                if magic!="\x48\x00\x00\x0c":
                    raise Exception("Wrong XAS magic.")

                audioType=f.read(1) #0x14 is DICE XAS, 0x16 is ealayer3, 0x12 is unknown
                if audioType=="\x16":
                    if not ealayerSupport:
                        print "EALayer3 segment not converted as the tool is missing: "+Variation.ChunkId+" "+self.trueFilename
                        continue
                    target=os.path.join(outputFolder,self.trueFilename)+" "+str(Variation.ChunkIndex)+" "+str(Variation.Index)+" "+str(ijk)+".wav"
                    makeLongDirs(target) #prepare the folder structure

                    process = subprocess.Popen([ealayer3Path,currentChunkName,"-i",str(Segment.SamplesOffset),"-o",target,"-s","-w"], stderr=subprocess.PIPE,startupinfo=startupinfo)
                   
                    process.communicate() #this should set the returncode
                    if process.returncode:
                        print process.stderr.readlines()

                    continue
                elif audioType!="\x14":
                    print "Unknown audio segment (type "+hexlify(audioType)+"): "+Variation.ChunkId+" "+self.trueFilename
                    continue

               

                channelRaw=f.read(1)
                if channelRaw not in channelDict:
                    print self.trueFilename
               
                   
                numChannels=channelDict[channelRaw]
                samplingRate=unpack(">H",f.read(2))[0] #[48000, 11025, 16000, 44100, 24000]
                f.read(4) #the first byte is always 0x40, the second takes one of 82 values, third and fourth can each take any value (i.e. 00 to ff)
                #82 values: ['\x01', '\x00', '\x03', '\x02', '\x05', '\x04', '\x07', '\x06', '\t', '\x08', '\x0b', '\n', '\r', '\x0c', '\x0f', '\x0e', '\x11', '\x10', '\x13', '\x12', '\x15', '\x14', '\x17', '\x16', '\x19', '\x18', '\x1b', '\x1a', '\x1d', '\x1c', '\x9f', '\x1e', '!', ' ', '#', '"', '%', '$', "'", '&', ')', '(', '+', '*', '-', ',', '/', '.', '1', '0', '3', '2', '5', '\x1f', '7', '6', '9', ';', '<', '?', '>', '\xc1', 'a', 'C', 'E', 'D', '\xa0', 'I', 'H', 'K', 'J', 'L', 'Y', 'X', '\x8e', '\x87', '4', 'A', 'd', 'F', '\x95', '\x7f']
               
                payload=""
                target= os.path.join(outputFolder,self.trueFilename)+" "+str(Variation.ChunkIndex)+" "+str(Variation.Index)+" "+str(ijk)+".wav"
                targetFolder=os.path.dirname(target)
                if not os.path.isdir(targetFolder): os.makedirs(targetFolder)

                f2=StringIO()
               
                write_header(f2, samplingRate, numChannels)
               
                while 1:
                    v1,length=unpack(">HH",f.read(4))

                    if length<=4: break
                    length-=8
                   
                    v2,v3=unpack(">HH",f.read(4))

                    if length%76!=0:
                        raise Exception("Strange length: "+str(length))
                       
                    atoms=length/76

                    blocks=[None]*numChannels
                   
                    for atom in xrange(0,atoms,numChannels):
                        for i in xrange(numChannels):
                            blocks[i]=decode(f.read(76))
                           
                        atomData=""
                        for i in xrange(128):
                            for j in xrange(numChannels):
                                atomData+=pack("f",blocks[j][i])
               
                        f2.write(atomData)

                finalPos=f2.tell()
                f2.seek(4)   
                f2.write(pack("I",finalPos-8))
                f2.seek(76)
                f2.write(pack("I",finalPos-76))

                f3=open2(target,"wb")
                f3.write(f2.getvalue())
               
                f2.close()

        for key in ChunkHandles:
            ChunkHandles[key].close()



def write_header(fp, samplingRate, nchannels):
    out = pack('4sl4s','RIFF',0,'WAVE')
    floatSize = 4
    fmt_size = 40
    tag = 0xFFFE
    etag = 3

    out += pack('4slHHllHH', 'fmt ', fmt_size, tag, nchannels, samplingRate,
                nchannels * samplingRate * floatSize, nchannels * floatSize, floatSize * 8)

    out += pack('HHlH14s', 22, floatSize * 8, (1 << nchannels) - 1, etag,
                '\x00\x00\x00\x00\x10\x00\x80\x00\x00\xaa\x008\x9b\x71')

    out += pack("<4sll","fact",4,floatSize)
    out += pack("4sl","data",0)

    fp.seek(0)
    fp.write(out)

   
xaslib = cdll.LoadLibrary("xas")
def decode(inputChars):
    charType = c_char * 76
    charBuf = charType(*inputChars)
    charbufpointer = pointer(charBuf)
   
    floatType = c_float * 128
    floatBuf = floatType()
    floatbufpointer = pointer(floatBuf)
   
    xaslib.decode(charbufpointer, floatbufpointer)
   
    return floatBuf


channelDict={"\x00":1,"\x04":2,"\x0c":4,"\x14":6,"\x1c":8,"\x05":2,"\x01":1}


if outputFolder[-1] not in ("/","\\"): outputFolder+="/"
if ebxFolder[-1] not in ("/","\\"): ebxFolder+="/"
if chunkFolder[-1] not in ("/","\\"): chunkFolder+="/"
if chunkFolder2[-1] not in ("/","\\"): chunkFolder2+="/"


#for ealayer. By default Python opens a new window for a split second and puts focus on it. This info makes no window show up at all.
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
startupinfo.wShowWindow = subprocess.SW_HIDE


try:
    subprocess.Popen([ealayer3Path,"-s","-w"],startupinfo=startupinfo)
    ealayerSupport=True
    print "EALayer3 tool detected."
except WindowsError:
    ealayerSupport=False
    print "EALayer3 tool not detected."

   
decodeAudio()


For PS3 users:

Quote:
From Frankelstner:

ChunkId FECAAFC2309C055D7E7D61C4C669F9C0 in EBX

Chunk ID In the filechunk name: c2afcafe9c305d057e7d61c4c669f9c0

lol

It's simple as pie.

FE CA AF C2 30 9C 05 5D 7E7D61C4C669F9C0
C2 CA AF FE 30 9C 05 5D 7E7D61C4C669F9C0 #0 with 3
C2 AF CA FE 30 9C 05 5D 7E7D61C4C669F9C0 #1 with 2
C2 AF CA FE 9C 30 05 5D 7E7D61C4C669F9C0 #4 with 5
C2 AF CA FE 9C 30 5D 05 7E7D61C4C669F9C0 #7 with 6

Ergo: 3 2 1 0 5 4 7 6 8 9 10 11 12 13 14 15

So, look at line 20 in the script and change False to True to allow the obfuscation permutation.

This one:

Code:
ChunkIdObfuscation=False #set to True for some console games when the script does not find any chunks at all
obfuscationPermutation=[3,2,1,0,5,4,7,6,8,9,10,11,12,13,14,15]

As OrangeC mentionned, it works for PS3 version as they are EALayer3 encoded.


_________________
Vosvoy


Last edited by Vosvoy on Fri May 10, 2013 10:59 pm, edited 3 times in total.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 143 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6, 7, 8 ... 10  Next

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group