Join also our Discord channel! Click here.

Frostbite 2 sound extraction research

Get help on any and all audio formats, or chip in and help others!
Post Reply
OrangeC
double-veteran
double-veteran
Posts: 864
Joined: Sun Apr 20, 2008 2:58 am
Has thanked: 5 times
Been thanked: 34 times

Re: Battlefield 3: Sounds recording technique

Post by OrangeC » Wed May 01, 2013 1:11 am

shouldn't be too hard, match the ebx with the id. Although it would be cool to have a linker script as not to rename manually one by one but this will do for now. then we can add our own xma header. and convert.

User avatar
Vosvoy
veteran
Posts: 127
Joined: Fri Feb 18, 2011 4:58 pm
Has thanked: 15 times
Been thanked: 15 times

Re: Battlefield 3: Sounds recording technique

Post by Vosvoy » Wed May 01, 2013 1:17 am

OrangeC wrote:shouldn't be too hard, match the ebx with the id. Although it would be cool to have a linker script as not to rename manually one by one but this will do for now. then we can add our own xma header. and convert.
I don't know anything about .xma format. So, I can't help you for that...
Vosvoy

User avatar
durandal217
veteran
Posts: 93
Joined: Tue Jul 17, 2012 3:52 am
Has thanked: 16 times
Been thanked: 11 times

Re: Battlefield 3: Sounds recording technique

Post by durandal217 » Wed May 01, 2013 2:46 am

Image

OMG I'm so dumb....... The PC chunks are different from the console chunks. They don't match up. Why didn't I think of that.

As an example the AEK-71 long reload CHUNKID for BF3 X360 is 34ab8c16913d8e37e7c05f76457d83a4 for BF3 PC it is 191FBE10D72A6AE3333401D966E8CBAE

I'm so sorry I didn't look at it sooner.
Last edited by durandal217 on Thu May 09, 2013 8:22 pm, edited 1 time in total.

OrangeC
double-veteran
double-veteran
Posts: 864
Joined: Sun Apr 20, 2008 2:58 am
Has thanked: 5 times
Been thanked: 34 times

Re: Battlefield 3: Sounds recording technique

Post by OrangeC » Wed May 01, 2013 6:23 am

That's unfortunate. All the more reason to implement console ebx conversion.

User avatar
Vosvoy
veteran
Posts: 127
Joined: Fri Feb 18, 2011 4:58 pm
Has thanked: 15 times
Been thanked: 15 times

Re: Battlefield 3: Sounds recording technique

Post by Vosvoy » Wed May 01, 2013 2:32 pm

Yeah, we need to find a solution for that. I'll investigate scrupulously the python script for my part. Otherwise, maybe the script creator could help us.

Anyway, we now know that .ebx files from PC are not compliant with Xbox360 version. It's a small step forward.
Vosvoy

User avatar
durandal217
veteran
Posts: 93
Joined: Tue Jul 17, 2012 3:52 am
Has thanked: 16 times
Been thanked: 11 times

Re: Battlefield 3: Sounds recording technique

Post by durandal217 » Sun May 05, 2013 11:39 pm

Just wanted to give you guys a heads up and let you know Frankelstner updated his sb/toc dumper for MOH warfighter.
http://www.bfeditor.org/forums/index.ph ... opic=15731

No update yet on audio conversion though.

User avatar
Vosvoy
veteran
Posts: 127
Joined: Fri Feb 18, 2011 4:58 pm
Has thanked: 15 times
Been thanked: 15 times

Re: Battlefield 3: Sounds recording technique

Post by Vosvoy » Tue May 07, 2013 4:58 pm

We've got some useful details from the script creator:
From Frankelstner:

The dll does exactly one thing: It converts a encoded block of 76 bytes into a decoded block of 128 floats. So the dll already does everything audio specific already. The task of the Python script is to feed the right bytes to the dll and eventually create a wav file from the floats it receives. It's not about audio at anymore, the problem is to have the script ensure that the file in question is actually XAS encoded, and to jump to the right places while reading an encoded file. The jumping part is already taken care of. And ensuring the file is actually XAS is just looking at the file with the strange length error and comparing it with other files that work correctly. Somewhere in the beginning there must be a byte different to tell the engine how to decode the format.
Vosvoy

OrangeC
double-veteran
double-veteran
Posts: 864
Joined: Sun Apr 20, 2008 2:58 am
Has thanked: 5 times
Been thanked: 34 times

Re: Battlefield 3: Sounds recording technique

Post by OrangeC » Tue May 07, 2013 10:28 pm

hmm strange it seems that the console chunks are weird when trying to match them up. Whats the difference here?

had to find the file by using the size because the chunk ID is not mall the same except the last numbers.

Code: Select all

000008c0    SoundWaveAsset 4a8e4be51666e0f748c9e60fc57f66a5 #primary instance
000008d0        $::SoundDataAsset
000008d0            $::Asset
000008d0                $::DataContainer
000008d8                Name AO4_Assets/Sound/Music/02_Outskirts/Action/Mus_Loop_OS02_Action_MMG1_134_Multi
000008dc                BudgetCategory Bc_Unknown
000008e0            NameHash 2538223577
000008e4            Chunks::array
00000944                member(0)::SoundDataChunk
00000944                    ChunkId B17600B855D3AC5D4D9090FF8B2B17BE
00000954                    ChunkSize 2083084
000008e8        Variations::array

Now the actual file is:b80076b1d3555dac4d9090ff8b2b17be.chunk
Some numbers are similiar but others not.

User avatar
Vosvoy
veteran
Posts: 127
Joined: Fri Feb 18, 2011 4:58 pm
Has thanked: 15 times
Been thanked: 15 times

Re: Battlefield 3: Sounds recording technique

Post by Vosvoy » Thu May 09, 2013 12:06 am

Hey mates! Some news here:
From Frankelstner:

Alright, here's a version that can deal with mohw (and hopefully big-endian files too, though I haven't tested that). For some reason the codec sometimes uses very strange values for the number of channels. E.g. the byte usually is 00 for 1 channel and 04 for 2 channels. However, some files use 01 and 05. I have played around with one sample file of each type, and think that 01 is 1 channel too and 05 is 2 channels. However, there's a chance some files are messed up, so give feedback if you find garbled audio.

Code: Select all

#needs python 2.7
import string
import sys
from binascii import hexlify
from struct import unpack,pack
import os
from cStringIO import StringIO
import cProfile
import cPickle

ebxFolder=   r"D:\hexing\mohw dump\bundles\ebx" #audio is found in several places
chunkFolder= r"D:\hexing\mohw dump\chunks" #grab audio from here
chunkFolder2=r"D:\hexing\mohw dump\bundles\chunks" #if the chunk is not found in the first folder, use this one
outputFolder=r"D:\decoded mohw123123"


#Run through the sound ebx files, find fields with chunk Guids and fieldName = ChunkId.
#The script will overwrite existing files.

#The filename of the ebx file importing an audio chunk becomes the name of the wav file.
#As for the indices, there are three of them in the following order.
#1: Variation.ChunkIndex: Some ebx files import several completely independent audio chunk files. This index differentiates between them. 
#2: Variation.Index: An ebx may use the same audio chunk for several sound variations, this index keeps them apart.
#3: Segment.Index: A variation may contain several segments, so there is another index.


##############################################################
##############################################################
def unpackBE(typ,data): return unpack(">"+typ,data)

def open2(path,mode="rb"):
    if mode=="wb":
        #create folders if necessary and return the file handle
        #first of all, create one folder level manully because makedirs might fail
        path=os.path.normpath(path)
        pathParts=path.split("\\")
        manualPart="\\".join(pathParts[:2])
        if not os.path.isdir(manualPart): os.makedirs(manualPart)

        #now handle the rest, including extra long path names
        folderPath=lp(os.path.dirname(path))
        if not os.path.isdir(folderPath): os.makedirs(folderPath)
    return open(lp(path),mode)

def lp(path): #long pathnames
    if len(path)<=247 or path=="" or path[:4]=='\\\\?\\': return path
    return unicode('\\\\?\\' + os.path.normpath(path))

def decodeAudio():
    for dir0, dirs, ff in os.walk(ebxFolder):
        for fname in ff:
            f=open2(dir0+"\\"+fname)
            magic=f.read(4)
            if magic=="\xCE\xD1\xB2\x0F":
                dbx=Dbx(f,unpack)
            elif magic=="\x0F\xB2\xD1\xCE":
                dbx=Dbx(f,unpackBE)
            else:
                f.close()
                continue
            dbx.decode()
            
try:
    from ctypes import *
    floatlib = cdll.LoadLibrary("floattostring")
    def formatfloat(num):
        bufType = c_char * 100
        buf = bufType()
        bufpointer = pointer(buf)
        floatlib.convertNum(c_double(num), bufpointer, 100)
        rawstring=(buf.raw)[:buf.raw.find("\x00")]
        if rawstring[:2]=="-.": return "-0."+rawstring[2:]
        elif rawstring[0]==".": return "0."+rawstring[1:]
        elif "e" not in rawstring and "." not in rawstring: return rawstring+".0"
        return rawstring
except:
    def formatfloat(num):
        return str(num)
def hasher(keyword): #32bit FNV-1 hash with FNV_offset_basis = 5381 and FNV_prime = 33
    hash = 5381
    for byte in keyword:
        hash = (hash*33) ^ ord(byte)
    return hash & 0xffffffff # use & because Python promotes the num instead of intended overflow
class Header:
    def __init__(self,varList): ##all 4byte unsigned integers
        self.absStringOffset     = varList[0]  ## absolute offset for string section start
        self.lenStringToEOF      = varList[1]  ## length from string section start to EOF
        self.numGUID             = varList[2]  ## number of external GUIDs
        self.null                = varList[3]  ## 00000000
        self.numInstanceRepeater = varList[4]
        self.numComplex          = varList[5]  ## number of complex entries
        self.numField            = varList[6]  ## number of field entries
        self.lenName             = varList[7]  ## length of name section including padding
        self.lenString           = varList[8]  ## length of string section including padding
        self.numArrayRepeater    = varList[9]
        self.lenPayload          = varList[10] ## length of normal payload section; the start of the array payload section is absStringOffset+lenString+lenPayload
class FieldDescriptor:
    def __init__(self,varList,keywordDict):
        self.name            = keywordDict[varList[0]]
        self.type            = varList[1]
        self.ref             = varList[2] #the field may contain another complex
        self.offset          = varList[3] #offset in payload section; relative to the complex containing it
        self.secondaryOffset = varList[4]
class ComplexDescriptor:
    def __init__(self,varList,keywordDict):
        self.name            = keywordDict[varList[0]]
        self.fieldStartIndex = varList[1] #the index of the first field belonging to the complex
        self.numField        = varList[2] #the total number of fields belonging to the complex
        self.alignment       = varList[3]
        self.type            = varList[4]
        self.size            = varList[5] #total length of the complex in the payload section
        self.secondarySize   = varList[6] #seems deprecated
class InstanceRepeater:
    def __init__(self,varList):
        self.null            = varList[0] #called "internalCount", seems to be always null
        self.repetitions     = varList[1] #number of instance repetitions
        self.complexIndex    = varList[2] #index of complex used as the instance
class arrayRepeater:
    def __init__(self,varList):
        self.offset          = varList[0] #offset in array payload section
        self.repetitions     = varList[1] #number of array repetitions
        self.complexIndex    = varList[2] #not necessary for extraction
class Complex:
    def __init__(self,desc,dbxhandle):
        self.desc=desc
        self.dbx=dbxhandle #lazy
    def get(self,name):
        pathElems=name.split("/")
        curPos=self
        if pathElems[-1].find("::")!=-1: #grab a complex
            for elem in pathElems:
                try:
                    curPos=curPos.go1(elem)
                except Exception,e:
                    raise Exception("Could not find complex with name: "+str(e)+"\nFull path: "+name+"\nFilename: "+self.dbx.trueFilename)
            return curPos
        #grab a field instead
        for elem in pathElems[:-1]:
            try:
                curPos=curPos.go1(elem)
            except Exception,e:
                raise Exception("Could not find complex with name: "+str(e)+"\nFull path: "+name+"\nFilename: "+self.dbx.trueFilename)
        for field in curPos.fields:
            if field.desc.name==pathElems[-1]:
                return field
            
        raise Exception("Could not find field with name: "+name+"\nFilename: "+self.dbx.trueFilename)

    def go1(self,name): #go once
        for field in self.fields:
            if field.desc.type in (0x0029, 0xd029,0x0000,0x0041):
                if field.desc.name+"::"+field.value.desc.name == name:
                    return field.value
        raise Exception(name)


class Field:
    def __init__(self,desc,dbx):
        self.desc=desc
        self.dbx=dbx
    def link(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)
        
        if self.value>>31:
            extguid=self.dbx.externalGUIDs[self.value&0x7fffffff]
            
            for existingDbx in dbxArray:
                if existingDbx.fileGUID==extguid[0]:
                    for guid, instance in existingDbx.instances:
                        if guid==extguid[1]:
                            return instance
                    

            f=valid(inputFolder+guidTable[extguid[0]]+".ebx")
##            print guidTable[extguid[0]]
            dbx=Dbx(f)
            dbxArray.append(dbx)
            for guid, instance in dbx.instances:
                if guid==extguid[1]:
                    return instance
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
        elif self.value!=0:
            for guid, instance in self.dbx.instances:
                if guid==self.dbx.internalGUIDs[self.value-1]:
                    return instance
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)

        raise Exception("Invalid link, could not find target.")

    def getlinkguid(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)

        if self.value>>31:
            return "".join(self.dbx.externalGUIDs[self.value&0x7fffffff])
        elif self.value!=0:
            return self.dbx.fileGUID+self.dbx.internalGUIDs[self.value-1]
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
    def getlinkname(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)

        if self.value>>31:
            return guidTable[self.dbx.externalGUIDs[self.value&0x7fffffff][0]]+"/"+self.dbx.externalGUIDs[self.value&0x7fffffff][1]
        elif self.value!=0:
            return self.dbx.trueFilename+"/"+self.dbx.internalGUIDs[self.value-1]
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
    

         
def valid(fname):
    f=open2(fname,"rb")
    if f.read(4) not in ("\xCE\xD1\xB2\x0F","\x0F\xB2\xD1\xCE"):
        f.close()
        raise Exception("nope")
    return f

class nullguid(Exception):
    def __init__(self, value):
        self.value = value
    def __str__(self):
        return repr(self.value)


numDict={0x0035:("I",4),0xc10d:("I",4),0xc14d:("d",8),0xc0ad:("?",1),0xc0fd:("i",4),0xc0bd:("B",1),0xc0ed:("h",2), 0xc0dd:("H",2), 0xc13d:("f",4)}


class Stub:
    pass



class Dbx:
    def __init__(self, f,unpacker):
        self.unpack=unpacker
        self.trueFilename=""
        self.header=Header(self.unpack("11I",f.read(44)))
        self.arraySectionstart=self.header.absStringOffset+self.header.lenString+self.header.lenPayload
        self.fileGUID, self.primaryInstanceGUID = f.read(16), f.read(16)    
        self.externalGUIDs=[(f.read(16),f.read(16)) for i in xrange(self.header.numGUID)]
        self.keywords=str.split(f.read(self.header.lenName),"\x00")
        self.keywordDict=dict((hasher(keyword),keyword) for keyword in self.keywords)
        self.fieldDescriptors=[FieldDescriptor(self.unpack("IHHII",f.read(16)), self.keywordDict) for i in xrange(self.header.numField)]
        self.complexDescriptors=[ComplexDescriptor(self.unpack("IIBBHHH",f.read(16)), self.keywordDict) for i in xrange(self.header.numComplex)]
        self.instanceRepeaters=[InstanceRepeater(self.unpack("3I",f.read(12))) for i in xrange(self.header.numInstanceRepeater)] 
        while f.tell()%16!=0: f.seek(1,1) #padding
        self.arrayRepeaters=[arrayRepeater(self.unpack("3I",f.read(12))) for i in xrange(self.header.numArrayRepeater)]

        #payload
        f.seek(self.header.absStringOffset+self.header.lenString)
        self.internalGUIDs=[]
        self.instances=[] # (guid, complex)
        for instanceRepeater in self.instanceRepeaters:
            for repetition in xrange(instanceRepeater.repetitions):
                instanceGUID=f.read(16)
                self.internalGUIDs.append(instanceGUID)
                if instanceGUID==self.primaryInstanceGUID:
                    self.isPrimaryInstance=True
                else:
                    self.isPrimaryInstance=False
                inst=self.readComplex(instanceRepeater.complexIndex,f)
                inst.guid=instanceGUID
                
                if self.isPrimaryInstance: self.prim=inst
                self.instances.append((instanceGUID,inst))
    
        f.close()
        

    def readComplex(self, complexIndex,f):
        complexDesc=self.complexDescriptors[complexIndex]
        cmplx=Complex(complexDesc,self)
        
        startPos=f.tell()                 
        cmplx.fields=[]
        for fieldIndex in xrange(complexDesc.fieldStartIndex,complexDesc.fieldStartIndex+complexDesc.numField):
            f.seek(startPos+self.fieldDescriptors[fieldIndex].offset)
            cmplx.fields.append(self.readField(fieldIndex,f))
        
        f.seek(startPos+complexDesc.size)
        return cmplx


    def readField(self,fieldIndex,f):
        fieldDesc = self.fieldDescriptors[fieldIndex]
        field=Field(fieldDesc,self)
        
        if fieldDesc.type in (0x0029, 0xd029,0x0000):
            field.value=self.readComplex(fieldDesc.ref,f)
        elif fieldDesc.type==0x0041:
            arrayRepeater=self.arrayRepeaters[self.unpack("I",f.read(4))[0]]
            arrayComplexDesc=self.complexDescriptors[fieldDesc.ref]

##            if arrayRepeater.repetitions==0: field.value = "*nullArray*"
            f.seek(self.arraySectionstart+arrayRepeater.offset)
            arrayComplex=Complex(arrayComplexDesc,self)
            arrayComplex.fields=[self.readField(arrayComplexDesc.fieldStartIndex,f) for repetition in xrange(arrayRepeater.repetitions)]
            field.value=arrayComplex
            
        elif fieldDesc.type in (0x407d, 0x409d):
            startPos=f.tell()
            f.seek(self.header.absStringOffset+self.unpack("I",f.read(4))[0])
            string=""
            while 1:
                a=f.read(1)
                if a=="\x00": break
                else: string+=a
            f.seek(startPos+4)
            
            if len(string)==0: field.value="*nullString*" #actually the string is ""
            else: field.value=string
            
            if self.isPrimaryInstance and self.trueFilename=="" and fieldDesc.name=="Name": self.trueFilename=string
            
                   
        elif fieldDesc.type==0x0089: #incomplete implementation, only gives back the selected string
            compareValue=self.unpack("I",f.read(4))[0] 
            enumComplex=self.complexDescriptors[fieldDesc.ref]

            if enumComplex.numField==0:
                field.value="*nullEnum*"
            for fieldIndex in xrange(enumComplex.fieldStartIndex,enumComplex.fieldStartIndex+enumComplex.numField):
                if self.fieldDescriptors[fieldIndex].offset==compareValue:
                    field.value=self.fieldDescriptors[fieldIndex].name
                    break
        elif fieldDesc.type==0xc15d:
            field.value=f.read(16)
        elif fieldDesc.type == 0xc13d: ################################
            field.value=formatfloat(self.unpack("f",f.read(4))[0])
        else:
            (typ,length)=numDict[fieldDesc.type]
            num=self.unpack(typ,f.read(length))[0]
            field.value=num
        
        return field
        

    def dump(self,outputFolder):
        dirName=os.path.dirname(outputFolder+self.trueFilename)
        if not os.path.isdir(dirName): os.makedirs(dirName)
        f2=open2(outputFolder+self.trueFilename+EXTENSION,"wb")
        print self.trueFilename
        
        for (guid,instance) in self.instances:
            if guid==self.primaryInstanceGUID: f2.write(instance.desc.name+" "+hexlify(guid)+ " #primary instance\r\n")
            else: f2.write(instance.desc.name+" "+hexlify(guid)+ "\r\n")
            self.recurse(instance.fields,f2,0)
        f2.close()

    def recurse(self, fields,f2, lvl): #over fields
        lvl+=1
        for field in fields:
            if field.desc.type in (0xc14d, 0xc0fd, 0xc10d, 0xc0ed, 0xc0dd, 0xc0bd, 0xc0ad, 0x407d, 0x409d, 0x0089):
                f2.write(lvl*SEP+field.desc.name+" "+str(field.value)+"\r\n")
            elif field.desc.type == 0xc13d:
                f2.write(lvl*SEP+field.desc.name+" "+formatfloat(field.value)+"\r\n")
            elif field.desc.type == 0xc15d:
                f2.write(lvl*SEP+field.desc.name+" "+hexlify(field.value)+"\r\n")
            elif field.desc.type == 0x0035:
                towrite=""
                if field.value>>31:
                    extguid=self.externalGUIDs[field.value&0x7fffffff]
                    try: towrite=guidTable[extguid[0]]+"/"+hexlify(extguid[1])
                    except: towrite=hexlify(extguid[0])+"/"+hexlify(extguid[1])
                elif field.value==0: towrite="*nullGuid*"
                else: towrite=hexlify(self.internalGUIDs[field.value-1])
                f2.write(lvl*SEP+field.desc.name+" "+towrite+"\r\n") 
            elif field.desc.type==0x0041 and len(field.value.fields)==0:
                f2.write(lvl*SEP+field.desc.name+" "+"*nullArray*"+"\r\n")
            else:
                f2.write(lvl*SEP+field.desc.name+"::"+field.value.desc.name+"\r\n")
                self.recurse(field.value.fields,f2,lvl)


    def decode(self):
        if not self.prim.desc.name=="SoundWaveAsset": return

        histogram=dict() #count the number of times each chunk is used by a variation to obtain the right index

        Chunks=[]
        for i in self.prim.get("$::SoundDataAsset/Chunks::array").fields:
            chnk=Stub()
            Chunks.append(chnk)
            chnk.ChunkId=i.value.get("ChunkId").value
            chnk.ChunkSize=i.value.get("ChunkSize").value

            
        variations=[i.link() for i in self.prim.get("Variations::array").fields]

        Variations=[]
        
        for var in variations:
            Variation=Stub()
            Variations.append(Variation)
            Variation.ChunkIndex=var.get("ChunkIndex").value
            Variation.SeekTablesSize=var.get("SeekTablesSize").value
            Variation.FirstLoopSegmentIndex=var.get("FirstLoopSegmentIndex").value
            Variation.LastLoopSegmentIndex=var.get("LastLoopSegmentIndex").value


            Variation.Segments=[]
            segs=var.get("Segments::array").fields
            for seg in segs:
                Segment=Stub()
                Variation.Segments.append(Segment)
                Segment.SamplesOffset = seg.value.get("SamplesOffset").value
                Segment.SeekTableOffset = seg.value.get("SeekTableOffset").value
                Segment.SegmentLength = seg.value.get("SegmentLength").value

            Variation.ChunkId=hexlify(Chunks[Variation.ChunkIndex].ChunkId)
            Variation.ChunkSize=Chunks[Variation.ChunkIndex].ChunkSize
        

            #find the appropriate index
            if Variation.ChunkIndex in histogram: #has been used previously already
                Variation.Index=histogram[Variation.ChunkIndex]
                histogram[Variation.ChunkIndex]+=1
            else:
                Variation.Index=0
                histogram[Variation.ChunkIndex]=1

                
        #everything is laid out neatly now
        #ChunkId, ChunkSize, Index, ChunkIndex, SeekTablesSize, FirstLoopSegmentIndex, LastLoopSegmentIndex
        #Segments with SamplesOffset, SeekTableOffset, SegmentLength

        ChunkHandles=dict() #should speed things up

        for Variation in Variations:
            try:
                f=ChunkHandles[Variation.ChunkId]
            except:
                try:
                    f=open2(chunkFolder+Variation.ChunkId+".chunk")
                except IOError:
                    try:
                        f=open2(chunkFolder2+Variation.ChunkId+".chunk")
                    except:
                        print "Chunk does not exist: "+Variation.ChunkId+" "+self.trueFilename
                        return
                ChunkHandles[Variation.ChunkId]=f


            for ijk in xrange(len(Variation.Segments)):
                Segment=Variation.Segments[ijk]
                f.seek(Segment.SamplesOffset)
                magic=f.read(4)
                
                if magic!="\x48\x00\x00\x0c":
                    raise Exception("Wrong XAS magic.")

                audioType=f.read(1) #0x14 is DICE XAS
                if audioType!="\x14":
                    print "Non XAS audio segment (type "+hexlify(audioType)+"): "+Variation.ChunkId+" "+self.trueFilename
                    continue
                channelRaw=f.read(1)
                if channelRaw not in channelDict:
                    print self.trueFilename
                
                    
                numChannels=channelDict[channelRaw]
                samplingRate=unpack(">H",f.read(2))[0] #[48000, 11025, 16000, 44100, 24000]
                f.read(4) #the first byte is always 0x40, the second takes one of 82 values, third and fourth can each take any value (i.e. 00 to ff)
                #82 values: ['\x01', '\x00', '\x03', '\x02', '\x05', '\x04', '\x07', '\x06', '\t', '\x08', '\x0b', '\n', '\r', '\x0c', '\x0f', '\x0e', '\x11', '\x10', '\x13', '\x12', '\x15', '\x14', '\x17', '\x16', '\x19', '\x18', '\x1b', '\x1a', '\x1d', '\x1c', '\x9f', '\x1e', '!', ' ', '#', '"', '%', '$', "'", '&', ')', '(', '+', '*', '-', ',', '/', '.', '1', '0', '3', '2', '5', '\x1f', '7', '6', '9', ';', '<', '?', '>', '\xc1', 'a', 'C', 'E', 'D', '\xa0', 'I', 'H', 'K', 'J', 'L', 'Y', 'X', '\x8e', '\x87', '4', 'A', 'd', 'F', '\x95', '\x7f']
                
                payload=""
                target= os.path.join(outputFolder,self.trueFilename)+" "+str(Variation.ChunkIndex)+" "+str(Variation.Index)+" "+str(ijk)+".wav"
                targetFolder=os.path.dirname(target)
                if not os.path.isdir(targetFolder): os.makedirs(targetFolder)

                f2=StringIO()
                
                write_header(f2, samplingRate, numChannels)
                
                while 1:
                    v1,length=unpack(">HH",f.read(4))

                    if length<=4: break
                    length-=8
                    
                    v2,v3=unpack(">HH",f.read(4))

                    if length%76!=0:
                        raise Exception("Strange length: "+str(length))
                        
                    atoms=length/76

                    blocks=[None]*numChannels
                    
                    for atom in xrange(0,atoms,numChannels):
                        for i in xrange(numChannels):
                            blocks[i]=decode(f.read(76))
                            
                        atomData=""
                        for i in xrange(128):
                            for j in xrange(numChannels):
                                atomData+=pack("f",blocks[j][i])
               
                        f2.write(atomData)

                finalPos=f2.tell()
                f2.seek(4)    
                f2.write(pack("I",finalPos-8))
                f2.seek(76)
                f2.write(pack("I",finalPos-76))

                f3=open2(target,"wb")
                f3.write(f2.getvalue())
                
                f2.close()

        for key in ChunkHandles:
            ChunkHandles[key].close()



def write_header(fp, samplingRate, nchannels):
    out = pack('4sl4s','RIFF',0,'WAVE')
    floatSize = 4
    fmt_size = 40
    tag = 0xFFFE
    etag = 3

    out += pack('4slHHllHH', 'fmt ', fmt_size, tag, nchannels, samplingRate,
                nchannels * samplingRate * floatSize, nchannels * floatSize, floatSize * 8)

    out += pack('HHlH14s', 22, floatSize * 8, (1 << nchannels) - 1, etag,
                '\x00\x00\x00\x00\x10\x00\x80\x00\x00\xaa\x008\x9b\x71')

    out += pack("<4sll","fact",4,floatSize)
    out += pack("4sl","data",0)

    fp.seek(0)
    fp.write(out)

    
xaslib = cdll.LoadLibrary("xas")
def decode(inputChars):
    charType = c_char * 76
    charBuf = charType(*inputChars)
    charbufpointer = pointer(charBuf)
    
    floatType = c_float * 128
    floatBuf = floatType()
    floatbufpointer = pointer(floatBuf)
    
    xaslib.decode(charbufpointer, floatbufpointer)
    
    return floatBuf


channelDict={"\x00":1,"\x04":2,"\x0c":4,"\x14":6,"\x1c":8,"\x05":2,"\x01":1}


if outputFolder[-1] not in ("/","\\"): outputFolder+="/"
if ebxFolder[-1] not in ("/","\\"): ebxFolder+="/"
if chunkFolder[-1] not in ("/","\\"): chunkFolder+="/"
if chunkFolder2[-1] not in ("/","\\"): chunkFolder2+="/"


    
decodeAudio()
Remember one thing guys: The script only works on BF3 PC version ( 100% ) and MOHW PC version ( 90% > Have to use Zench EALayer3 tool for some sounds ), for the moment. There's some troubles with console version as durandal217 and OrangeC said.
Last edited by Vosvoy on Fri May 10, 2013 1:31 am, edited 4 times in total.
Vosvoy

OrangeC
double-veteran
double-veteran
Posts: 864
Joined: Sun Apr 20, 2008 2:58 am
Has thanked: 5 times
Been thanked: 34 times

Re: Frostbite 2 sound extraction research

Post by OrangeC » Thu May 09, 2013 7:17 am

To Convert the non-xas use the ealayer3 tool from zench.

User avatar
durandal217
veteran
Posts: 93
Joined: Tue Jul 17, 2012 3:52 am
Has thanked: 16 times
Been thanked: 11 times

Re: Frostbite 2 sound extraction research

Post by durandal217 » Thu May 09, 2013 8:25 pm

Frankelstner has updated the XAS script again to properly handle EALayer3 files for MOHW. Music now properly extracts.

Code: Select all

#needs python 2.7
import string
import sys
from binascii import hexlify
from struct import unpack,pack
import os
from cStringIO import StringIO
import cProfile
import cPickle
import subprocess



ebxFolder=   r"D:\hexing\mohw dump\bundles\ebx" #audio is found in several places
chunkFolder= r"D:\hexing\mohw dump\chunks" #grab audio from here
chunkFolder2=r"D:\hexing\mohw dump\bundles\chunks" #if the chunk is not found in the first folder, use this one
outputFolder=r"D:\decoded mohw123123123"
ealayer3Path=r"D:\ealayer3-0.7.0-win32\ealayer3.exe" #https://bitbucket.org/Zenchreal/ealayer3/downloads


#Run through the sound ebx files, find fields with chunk Guids and fieldName = ChunkId.
#The script will overwrite existing files.

#The filename of the ebx file importing an audio chunk becomes the name of the wav file.
#As for the indices, there are three of them in the following order.
#1: Variation.ChunkIndex: Some ebx files import several completely independent audio chunk files. This index differentiates between them. 
#2: Variation.Index: An ebx may use the same audio chunk for several sound variations, this index keeps them apart.
#3: Segment.Index: A variation may contain several segments, so there is another index.


##############################################################
##############################################################
def unpackBE(typ,data): return unpack(">"+typ,data)

def makeLongDirs(path):
    #create folders if necessary and return the file handle
    #first of all, create one folder level manully because makedirs might fail
    path=os.path.normpath(path)
    pathParts=path.split("\\")
    manualPart="\\".join(pathParts[:2])
    if not os.path.isdir(manualPart): os.makedirs(manualPart)
    
    #now handle the rest, including extra long path names
    folderPath=lp(os.path.dirname(path))
    if not os.path.isdir(folderPath): os.makedirs(folderPath)
    

def open2(path,mode="rb"):
    if mode=="wb": makeLongDirs(path)
    return open(lp(path),mode)

def lp(path): #long pathnames
    if len(path)<=247 or path=="" or path[:4]=='\\\\?\\': return path
    return unicode('\\\\?\\' + os.path.normpath(path))

def decodeAudio():
    for dir0, dirs, ff in os.walk(ebxFolder):
        for fname in ff:
            f=open2(dir0+"\\"+fname)
            magic=f.read(4)
            if magic=="\xCE\xD1\xB2\x0F":
                dbx=Dbx(f,unpack)
            elif magic=="\x0F\xB2\xD1\xCE":
                dbx=Dbx(f,unpackBE)
            else:
                f.close()
                continue
            dbx.decode()
            
try:
    from ctypes import *
    floatlib = cdll.LoadLibrary("floattostring")
    def formatfloat(num):
        bufType = c_char * 100
        buf = bufType()
        bufpointer = pointer(buf)
        floatlib.convertNum(c_double(num), bufpointer, 100)
        rawstring=(buf.raw)[:buf.raw.find("\x00")]
        if rawstring[:2]=="-.": return "-0."+rawstring[2:]
        elif rawstring[0]==".": return "0."+rawstring[1:]
        elif "e" not in rawstring and "." not in rawstring: return rawstring+".0"
        return rawstring
except:
    def formatfloat(num):
        return str(num)
def hasher(keyword): #32bit FNV-1 hash with FNV_offset_basis = 5381 and FNV_prime = 33
    hash = 5381
    for byte in keyword:
        hash = (hash*33) ^ ord(byte)
    return hash & 0xffffffff # use & because Python promotes the num instead of intended overflow
class Header:
    def __init__(self,varList): ##all 4byte unsigned integers
        self.absStringOffset     = varList[0]  ## absolute offset for string section start
        self.lenStringToEOF      = varList[1]  ## length from string section start to EOF
        self.numGUID             = varList[2]  ## number of external GUIDs
        self.null                = varList[3]  ## 00000000
        self.numInstanceRepeater = varList[4]
        self.numComplex          = varList[5]  ## number of complex entries
        self.numField            = varList[6]  ## number of field entries
        self.lenName             = varList[7]  ## length of name section including padding
        self.lenString           = varList[8]  ## length of string section including padding
        self.numArrayRepeater    = varList[9]
        self.lenPayload          = varList[10] ## length of normal payload section; the start of the array payload section is absStringOffset+lenString+lenPayload
class FieldDescriptor:
    def __init__(self,varList,keywordDict):
        self.name            = keywordDict[varList[0]]
        self.type            = varList[1]
        self.ref             = varList[2] #the field may contain another complex
        self.offset          = varList[3] #offset in payload section; relative to the complex containing it
        self.secondaryOffset = varList[4]
class ComplexDescriptor:
    def __init__(self,varList,keywordDict):
        self.name            = keywordDict[varList[0]]
        self.fieldStartIndex = varList[1] #the index of the first field belonging to the complex
        self.numField        = varList[2] #the total number of fields belonging to the complex
        self.alignment       = varList[3]
        self.type            = varList[4]
        self.size            = varList[5] #total length of the complex in the payload section
        self.secondarySize   = varList[6] #seems deprecated
class InstanceRepeater:
    def __init__(self,varList):
        self.null            = varList[0] #called "internalCount", seems to be always null
        self.repetitions     = varList[1] #number of instance repetitions
        self.complexIndex    = varList[2] #index of complex used as the instance
class arrayRepeater:
    def __init__(self,varList):
        self.offset          = varList[0] #offset in array payload section
        self.repetitions     = varList[1] #number of array repetitions
        self.complexIndex    = varList[2] #not necessary for extraction
class Complex:
    def __init__(self,desc,dbxhandle):
        self.desc=desc
        self.dbx=dbxhandle #lazy
    def get(self,name):
        pathElems=name.split("/")
        curPos=self
        if pathElems[-1].find("::")!=-1: #grab a complex
            for elem in pathElems:
                try:
                    curPos=curPos.go1(elem)
                except Exception,e:
                    raise Exception("Could not find complex with name: "+str(e)+"\nFull path: "+name+"\nFilename: "+self.dbx.trueFilename)
            return curPos
        #grab a field instead
        for elem in pathElems[:-1]:
            try:
                curPos=curPos.go1(elem)
            except Exception,e:
                raise Exception("Could not find complex with name: "+str(e)+"\nFull path: "+name+"\nFilename: "+self.dbx.trueFilename)
        for field in curPos.fields:
            if field.desc.name==pathElems[-1]:
                return field
            
        raise Exception("Could not find field with name: "+name+"\nFilename: "+self.dbx.trueFilename)

    def go1(self,name): #go once
        for field in self.fields:
            if field.desc.type in (0x0029, 0xd029,0x0000,0x0041):
                if field.desc.name+"::"+field.value.desc.name == name:
                    return field.value
        raise Exception(name)


class Field:
    def __init__(self,desc,dbx):
        self.desc=desc
        self.dbx=dbx
    def link(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)
        
        if self.value>>31:
            extguid=self.dbx.externalGUIDs[self.value&0x7fffffff]
            
            for existingDbx in dbxArray:
                if existingDbx.fileGUID==extguid[0]:
                    for guid, instance in existingDbx.instances:
                        if guid==extguid[1]:
                            return instance
                    

            f=valid(inputFolder+guidTable[extguid[0]]+".ebx")
##            print guidTable[extguid[0]]
            dbx=Dbx(f)
            dbxArray.append(dbx)
            for guid, instance in dbx.instances:
                if guid==extguid[1]:
                    return instance
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
        elif self.value!=0:
            for guid, instance in self.dbx.instances:
                if guid==self.dbx.internalGUIDs[self.value-1]:
                    return instance
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)

        raise Exception("Invalid link, could not find target.")

    def getlinkguid(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)

        if self.value>>31:
            return "".join(self.dbx.externalGUIDs[self.value&0x7fffffff])
        elif self.value!=0:
            return self.dbx.fileGUID+self.dbx.internalGUIDs[self.value-1]
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
    def getlinkname(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)

        if self.value>>31:
            return guidTable[self.dbx.externalGUIDs[self.value&0x7fffffff][0]]+"/"+self.dbx.externalGUIDs[self.value&0x7fffffff][1]
        elif self.value!=0:
            return self.dbx.trueFilename+"/"+self.dbx.internalGUIDs[self.value-1]
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
    

         
def valid(fname):
    f=open2(fname,"rb")
    if f.read(4) not in ("\xCE\xD1\xB2\x0F","\x0F\xB2\xD1\xCE"):
        f.close()
        raise Exception("nope")
    return f

class nullguid(Exception):
    def __init__(self, value):
        self.value = value
    def __str__(self):
        return repr(self.value)


numDict={0x0035:("I",4),0xc10d:("I",4),0xc14d:("d",8),0xc0ad:("?",1),0xc0fd:("i",4),0xc0bd:("B",1),0xc0ed:("h",2), 0xc0dd:("H",2), 0xc13d:("f",4)}


class Stub:
    pass



class Dbx:
    def __init__(self, f,unpacker):
        self.unpack=unpacker
        self.trueFilename=""
        self.header=Header(self.unpack("11I",f.read(44)))
        self.arraySectionstart=self.header.absStringOffset+self.header.lenString+self.header.lenPayload
        self.fileGUID, self.primaryInstanceGUID = f.read(16), f.read(16)    
        self.externalGUIDs=[(f.read(16),f.read(16)) for i in xrange(self.header.numGUID)]
        self.keywords=str.split(f.read(self.header.lenName),"\x00")
        self.keywordDict=dict((hasher(keyword),keyword) for keyword in self.keywords)
        self.fieldDescriptors=[FieldDescriptor(self.unpack("IHHII",f.read(16)), self.keywordDict) for i in xrange(self.header.numField)]
        self.complexDescriptors=[ComplexDescriptor(self.unpack("IIBBHHH",f.read(16)), self.keywordDict) for i in xrange(self.header.numComplex)]
        self.instanceRepeaters=[InstanceRepeater(self.unpack("3I",f.read(12))) for i in xrange(self.header.numInstanceRepeater)] 
        while f.tell()%16!=0: f.seek(1,1) #padding
        self.arrayRepeaters=[arrayRepeater(self.unpack("3I",f.read(12))) for i in xrange(self.header.numArrayRepeater)]

        #payload
        f.seek(self.header.absStringOffset+self.header.lenString)
        self.internalGUIDs=[]
        self.instances=[] # (guid, complex)
        for instanceRepeater in self.instanceRepeaters:
            for repetition in xrange(instanceRepeater.repetitions):
                instanceGUID=f.read(16)
                self.internalGUIDs.append(instanceGUID)
                if instanceGUID==self.primaryInstanceGUID:
                    self.isPrimaryInstance=True
                else:
                    self.isPrimaryInstance=False
                inst=self.readComplex(instanceRepeater.complexIndex,f)
                inst.guid=instanceGUID
                
                if self.isPrimaryInstance: self.prim=inst
                self.instances.append((instanceGUID,inst))
    
        f.close()
        

    def readComplex(self, complexIndex,f):
        complexDesc=self.complexDescriptors[complexIndex]
        cmplx=Complex(complexDesc,self)
        
        startPos=f.tell()                 
        cmplx.fields=[]
        for fieldIndex in xrange(complexDesc.fieldStartIndex,complexDesc.fieldStartIndex+complexDesc.numField):
            f.seek(startPos+self.fieldDescriptors[fieldIndex].offset)
            cmplx.fields.append(self.readField(fieldIndex,f))
        
        f.seek(startPos+complexDesc.size)
        return cmplx


    def readField(self,fieldIndex,f):
        fieldDesc = self.fieldDescriptors[fieldIndex]
        field=Field(fieldDesc,self)
        
        if fieldDesc.type in (0x0029, 0xd029,0x0000):
            field.value=self.readComplex(fieldDesc.ref,f)
        elif fieldDesc.type==0x0041:
            arrayRepeater=self.arrayRepeaters[self.unpack("I",f.read(4))[0]]
            arrayComplexDesc=self.complexDescriptors[fieldDesc.ref]

##            if arrayRepeater.repetitions==0: field.value = "*nullArray*"
            f.seek(self.arraySectionstart+arrayRepeater.offset)
            arrayComplex=Complex(arrayComplexDesc,self)
            arrayComplex.fields=[self.readField(arrayComplexDesc.fieldStartIndex,f) for repetition in xrange(arrayRepeater.repetitions)]
            field.value=arrayComplex
            
        elif fieldDesc.type in (0x407d, 0x409d):
            startPos=f.tell()
            f.seek(self.header.absStringOffset+self.unpack("I",f.read(4))[0])
            string=""
            while 1:
                a=f.read(1)
                if a=="\x00": break
                else: string+=a
            f.seek(startPos+4)
            
            if len(string)==0: field.value="*nullString*" #actually the string is ""
            else: field.value=string
            
            if self.isPrimaryInstance and self.trueFilename=="" and fieldDesc.name=="Name": self.trueFilename=string
            
                   
        elif fieldDesc.type==0x0089: #incomplete implementation, only gives back the selected string
            compareValue=self.unpack("I",f.read(4))[0] 
            enumComplex=self.complexDescriptors[fieldDesc.ref]

            if enumComplex.numField==0:
                field.value="*nullEnum*"
            for fieldIndex in xrange(enumComplex.fieldStartIndex,enumComplex.fieldStartIndex+enumComplex.numField):
                if self.fieldDescriptors[fieldIndex].offset==compareValue:
                    field.value=self.fieldDescriptors[fieldIndex].name
                    break
        elif fieldDesc.type==0xc15d:
            field.value=f.read(16)
        elif fieldDesc.type == 0xc13d: ################################
            field.value=formatfloat(self.unpack("f",f.read(4))[0])
        else:
            (typ,length)=numDict[fieldDesc.type]
            num=self.unpack(typ,f.read(length))[0]
            field.value=num
        
        return field
        

    def dump(self,outputFolder):
        dirName=os.path.dirname(outputFolder+self.trueFilename)
        if not os.path.isdir(dirName): os.makedirs(dirName)
        f2=open2(outputFolder+self.trueFilename+EXTENSION,"wb")
        print self.trueFilename
        
        for (guid,instance) in self.instances:
            if guid==self.primaryInstanceGUID: f2.write(instance.desc.name+" "+hexlify(guid)+ " #primary instance\r\n")
            else: f2.write(instance.desc.name+" "+hexlify(guid)+ "\r\n")
            self.recurse(instance.fields,f2,0)
        f2.close()

    def recurse(self, fields,f2, lvl): #over fields
        lvl+=1
        for field in fields:
            if field.desc.type in (0xc14d, 0xc0fd, 0xc10d, 0xc0ed, 0xc0dd, 0xc0bd, 0xc0ad, 0x407d, 0x409d, 0x0089):
                f2.write(lvl*SEP+field.desc.name+" "+str(field.value)+"\r\n")
            elif field.desc.type == 0xc13d:
                f2.write(lvl*SEP+field.desc.name+" "+formatfloat(field.value)+"\r\n")
            elif field.desc.type == 0xc15d:
                f2.write(lvl*SEP+field.desc.name+" "+hexlify(field.value)+"\r\n")
            elif field.desc.type == 0x0035:
                towrite=""
                if field.value>>31:
                    extguid=self.externalGUIDs[field.value&0x7fffffff]
                    try: towrite=guidTable[extguid[0]]+"/"+hexlify(extguid[1])
                    except: towrite=hexlify(extguid[0])+"/"+hexlify(extguid[1])
                elif field.value==0: towrite="*nullGuid*"
                else: towrite=hexlify(self.internalGUIDs[field.value-1])
                f2.write(lvl*SEP+field.desc.name+" "+towrite+"\r\n") 
            elif field.desc.type==0x0041 and len(field.value.fields)==0:
                f2.write(lvl*SEP+field.desc.name+" "+"*nullArray*"+"\r\n")
            else:
                f2.write(lvl*SEP+field.desc.name+"::"+field.value.desc.name+"\r\n")
                self.recurse(field.value.fields,f2,lvl)


    def decode(self):
        if not self.prim.desc.name=="SoundWaveAsset": return

        histogram=dict() #count the number of times each chunk is used by a variation to obtain the right index

        Chunks=[]
        for i in self.prim.get("$::SoundDataAsset/Chunks::array").fields:
            chnk=Stub()
            Chunks.append(chnk)
            chnk.ChunkId=i.value.get("ChunkId").value
            chnk.ChunkSize=i.value.get("ChunkSize").value

            
        variations=[i.link() for i in self.prim.get("Variations::array").fields]

        Variations=[]
        
        for var in variations:
            Variation=Stub()
            Variations.append(Variation)
            Variation.ChunkIndex=var.get("ChunkIndex").value
            Variation.SeekTablesSize=var.get("SeekTablesSize").value
            Variation.FirstLoopSegmentIndex=var.get("FirstLoopSegmentIndex").value
            Variation.LastLoopSegmentIndex=var.get("LastLoopSegmentIndex").value


            Variation.Segments=[]
            segs=var.get("Segments::array").fields
            for seg in segs:
                Segment=Stub()
                Variation.Segments.append(Segment)
                Segment.SamplesOffset = seg.value.get("SamplesOffset").value
                Segment.SeekTableOffset = seg.value.get("SeekTableOffset").value
                Segment.SegmentLength = seg.value.get("SegmentLength").value

            Variation.ChunkId=hexlify(Chunks[Variation.ChunkIndex].ChunkId)
            Variation.ChunkSize=Chunks[Variation.ChunkIndex].ChunkSize
        

            #find the appropriate index
            if Variation.ChunkIndex in histogram: #has been used previously already
                Variation.Index=histogram[Variation.ChunkIndex]
                histogram[Variation.ChunkIndex]+=1
            else:
                Variation.Index=0
                histogram[Variation.ChunkIndex]=1

                
        #everything is laid out neatly now
        #ChunkId, ChunkSize, Index, ChunkIndex, SeekTablesSize, FirstLoopSegmentIndex, LastLoopSegmentIndex
        #Segments with SamplesOffset, SeekTableOffset, SegmentLength

        ChunkHandles=dict() #for each ebx, keep track of all file handles

        for Variation in Variations:
            try:
                f=ChunkHandles[Variation.ChunkId]
            except:
                try:
                    f=open2(chunkFolder+Variation.ChunkId+".chunk")
                    currentChunkName=chunkFolder+Variation.ChunkId+".chunk"
                except IOError:
                    try:
                        f=open2(chunkFolder2+Variation.ChunkId+".chunk")
                        currentChunkName=chunkFolder2+Variation.ChunkId+".chunk"
                    except:
                        print "Chunk does not exist: "+Variation.ChunkId+" "+self.trueFilename
                        return
                ChunkHandles[Variation.ChunkId]=f


            for ijk in xrange(len(Variation.Segments)):
                Segment=Variation.Segments[ijk]
                f.seek(Segment.SamplesOffset)
                magic=f.read(4)
                
                if magic!="\x48\x00\x00\x0c":
                    raise Exception("Wrong XAS magic.")

                audioType=f.read(1) #0x14 is DICE XAS, 0x16 is ealayer3, 0x12 is unknown
                if audioType=="\x16":
                    if not ealayerSupport:
                        print "EALayer3 segment not converted as the tool is missing: "+Variation.ChunkId+" "+self.trueFilename
                        continue
                    target=os.path.join(outputFolder,self.trueFilename)+" "+str(Variation.ChunkIndex)+" "+str(Variation.Index)+" "+str(ijk)+".mp3"
                    makeLongDirs(target) #prepare the folder structure

                    process = subprocess.Popen([ealayer3Path,currentChunkName,"-i",str(Segment.SamplesOffset),"-o",target], stderr=subprocess.PIPE,startupinfo=startupinfo)
                    
                    process.communicate() #this should set the returncode
                    if process.returncode:
                        print process.stderr.readlines()

                    continue
                elif audioType!="\x14":
                    print "Unknown audio segment (type "+hexlify(audioType)+"): "+Variation.ChunkId+" "+self.trueFilename
                    continue

                

                channelRaw=f.read(1)
                if channelRaw not in channelDict:
                    print self.trueFilename
                
                    
                numChannels=channelDict[channelRaw]
                samplingRate=unpack(">H",f.read(2))[0] #[48000, 11025, 16000, 44100, 24000]
                f.read(4) #the first byte is always 0x40, the second takes one of 82 values, third and fourth can each take any value (i.e. 00 to ff)
                #82 values: ['\x01', '\x00', '\x03', '\x02', '\x05', '\x04', '\x07', '\x06', '\t', '\x08', '\x0b', '\n', '\r', '\x0c', '\x0f', '\x0e', '\x11', '\x10', '\x13', '\x12', '\x15', '\x14', '\x17', '\x16', '\x19', '\x18', '\x1b', '\x1a', '\x1d', '\x1c', '\x9f', '\x1e', '!', ' ', '#', '"', '%', '$', "'", '&', ')', '(', '+', '*', '-', ',', '/', '.', '1', '0', '3', '2', '5', '\x1f', '7', '6', '9', ';', '<', '?', '>', '\xc1', 'a', 'C', 'E', 'D', '\xa0', 'I', 'H', 'K', 'J', 'L', 'Y', 'X', '\x8e', '\x87', '4', 'A', 'd', 'F', '\x95', '\x7f']
                
                payload=""
                target= os.path.join(outputFolder,self.trueFilename)+" "+str(Variation.ChunkIndex)+" "+str(Variation.Index)+" "+str(ijk)+".wav"
                targetFolder=os.path.dirname(target)
                if not os.path.isdir(targetFolder): os.makedirs(targetFolder)

                f2=StringIO()
                
                write_header(f2, samplingRate, numChannels)
                
                while 1:
                    v1,length=unpack(">HH",f.read(4))

                    if length<=4: break
                    length-=8
                    
                    v2,v3=unpack(">HH",f.read(4))

                    if length%76!=0:
                        raise Exception("Strange length: "+str(length))
                        
                    atoms=length/76

                    blocks=[None]*numChannels
                    
                    for atom in xrange(0,atoms,numChannels):
                        for i in xrange(numChannels):
                            blocks[i]=decode(f.read(76))
                            
                        atomData=""
                        for i in xrange(128):
                            for j in xrange(numChannels):
                                atomData+=pack("f",blocks[j][i])
               
                        f2.write(atomData)

                finalPos=f2.tell()
                f2.seek(4)    
                f2.write(pack("I",finalPos-8))
                f2.seek(76)
                f2.write(pack("I",finalPos-76))

                f3=open2(target,"wb")
                f3.write(f2.getvalue())
                
                f2.close()

        for key in ChunkHandles:
            ChunkHandles[key].close()



def write_header(fp, samplingRate, nchannels):
    out = pack('4sl4s','RIFF',0,'WAVE')
    floatSize = 4
    fmt_size = 40
    tag = 0xFFFE
    etag = 3

    out += pack('4slHHllHH', 'fmt ', fmt_size, tag, nchannels, samplingRate,
                nchannels * samplingRate * floatSize, nchannels * floatSize, floatSize * 8)

    out += pack('HHlH14s', 22, floatSize * 8, (1 << nchannels) - 1, etag,
                '\x00\x00\x00\x00\x10\x00\x80\x00\x00\xaa\x008\x9b\x71')

    out += pack("<4sll","fact",4,floatSize)
    out += pack("4sl","data",0)

    fp.seek(0)
    fp.write(out)

    
xaslib = cdll.LoadLibrary("xas")
def decode(inputChars):
    charType = c_char * 76
    charBuf = charType(*inputChars)
    charbufpointer = pointer(charBuf)
    
    floatType = c_float * 128
    floatBuf = floatType()
    floatbufpointer = pointer(floatBuf)
    
    xaslib.decode(charbufpointer, floatbufpointer)
    
    return floatBuf


channelDict={"\x00":1,"\x04":2,"\x0c":4,"\x14":6,"\x1c":8,"\x05":2,"\x01":1}


if outputFolder[-1] not in ("/","\\"): outputFolder+="/"
if ebxFolder[-1] not in ("/","\\"): ebxFolder+="/"
if chunkFolder[-1] not in ("/","\\"): chunkFolder+="/"
if chunkFolder2[-1] not in ("/","\\"): chunkFolder2+="/"


#for ealayer. By default Python opens a new window for a split second and puts focus on it. This info makes no window show up at all.
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
startupinfo.wShowWindow = subprocess.SW_HIDE


try:
    subprocess.Popen([ealayer3Path],startupinfo=startupinfo)
    ealayerSupport=True
    print "EALayer3 tool detected."
except WindowsError:
    ealayerSupport=False
    print "EALayer3 tool not detected."

    
decodeAudio()

User avatar
Vosvoy
veteran
Posts: 127
Joined: Fri Feb 18, 2011 4:58 pm
Has thanked: 15 times
Been thanked: 15 times

Re: Frostbite 2 sound extraction research

Post by Vosvoy » Thu May 09, 2013 9:01 pm

That's god damn good.

Still have issues with Xbox360 version by the way?
Vosvoy

User avatar
durandal217
veteran
Posts: 93
Joined: Tue Jul 17, 2012 3:52 am
Has thanked: 16 times
Been thanked: 11 times

Re: Frostbite 2 sound extraction research

Post by durandal217 » Fri May 10, 2013 12:03 am

Vosvoy wrote:That's god damn good.

Still have issues with Xbox360 version by the way?
I decided to skip the 360 version, nothing was matching up. I just asked a friend if I could copy his files from the pc version and use that, he was cool with it, if not perplexed as to why I wanted just the files for a game we didn't even play anymore.

But he was cool with it, and I got what I needed.

OrangeC
double-veteran
double-veteran
Posts: 864
Joined: Sun Apr 20, 2008 2:58 am
Has thanked: 5 times
Been thanked: 34 times

Re: Frostbite 2 sound extraction research

Post by OrangeC » Fri May 10, 2013 12:31 am

Update

Testing now Army of two devils cartel the PS3 version as it uses EAlayer3. Works for the most part but has problems handling multichannel files.

EDIT: Okay seems i was using wrong arguments. The proper argument from ealayer3 to use is -S all" but i have no way of putting that properly into the python script in this line.

Code: Select all

[ealayer3Path,currentChunkName,"-i",str(Segment.SamplesOffset),"-o",target], stderr=subprocess.PIPE,startupinfo=startupinfo)
any help?

User avatar
Vosvoy
veteran
Posts: 127
Joined: Fri Feb 18, 2011 4:58 pm
Has thanked: 15 times
Been thanked: 15 times

Re: Frostbite 2 sound extraction research

Post by Vosvoy » Fri May 10, 2013 2:57 am

All right. I believe that this time is the right one for console users. As OrangeC said: Frankelstner updated his script that can handle console FB2 games ( Army of Two: Devil's Cartel at least ) and seems to be functional. Here is the last script:

Code: Select all

#needs python 2.7
import string
import sys
from binascii import hexlify
from struct import unpack,pack
import os
from cStringIO import StringIO
import cProfile
import cPickle
import subprocess



ebxFolder=   r"D:\MOHWDUMP\bundles\ebx\mohw\audio" #audio is found in several places
chunkFolder= r"D:\MOHWDUMP\chunks" #grab audio from here
chunkFolder2=r"D:\MOHWDUMP\bundles\chunks" #if the chunk is not found in the first folder, use this one
outputFolder=r"D:\MOHWSOUNDS"
ealayer3Path=r"D:\ealayer3-0.7.0-win32\ealayer3.exe" #https://bitbucket.org/Zenchreal/ealayer3/downloads

ChunkIdObfuscation=False #set to True for some console games when the script does not find any chunks at all
obfuscationPermutation=[3,2,1,0,5,4,7,6,8,9,10,11,12,13,14,15]

#Run through the sound ebx files, find fields with chunk Guids and fieldName = ChunkId.
#The script will overwrite existing files.

#The filename of the ebx file importing an audio chunk becomes the name of the wav file.
#As for the indices, there are three of them in the following order.
#1: Variation.ChunkIndex: Some ebx files import several completely independent audio chunk files. This index differentiates between them. 
#2: Variation.Index: An ebx may use the same audio chunk for several sound variations, this index keeps them apart.
#3: Segment.Index: A variation may contain several segments, so there is another index.


##############################################################
##############################################################
def unpackBE(typ,data): return unpack(">"+typ,data)

def makeLongDirs(path):
    #create folders if necessary and return the file handle
    #first of all, create one folder level manully because makedirs might fail
    path=os.path.normpath(path)
    pathParts=path.split("\\")
    manualPart="\\".join(pathParts[:2])
    if not os.path.isdir(manualPart): os.makedirs(manualPart)
    
    #now handle the rest, including extra long path names
    folderPath=lp(os.path.dirname(path))
    if not os.path.isdir(folderPath): os.makedirs(folderPath)
    

def open2(path,mode="rb"):
    if mode=="wb": makeLongDirs(path)
    return open(lp(path),mode)

def lp(path): #long pathnames
    if len(path)<=247 or path=="" or path[:4]=='\\\\?\\': return path
    return unicode('\\\\?\\' + os.path.normpath(path))

def decodeAudio():
    for dir0, dirs, ff in os.walk(ebxFolder):
        for fname in ff:
            f=open2(dir0+"\\"+fname)
            magic=f.read(4)
            if magic=="\xCE\xD1\xB2\x0F":
                dbx=Dbx(f,unpack)
            elif magic=="\x0F\xB2\xD1\xCE":
                dbx=Dbx(f,unpackBE)
            else:
                f.close()
                continue
            dbx.decode()
            
try:
    from ctypes import *
    floatlib = cdll.LoadLibrary("floattostring")
    def formatfloat(num):
        bufType = c_char * 100
        buf = bufType()
        bufpointer = pointer(buf)
        floatlib.convertNum(c_double(num), bufpointer, 100)
        rawstring=(buf.raw)[:buf.raw.find("\x00")]
        if rawstring[:2]=="-.": return "-0."+rawstring[2:]
        elif rawstring[0]==".": return "0."+rawstring[1:]
        elif "e" not in rawstring and "." not in rawstring: return rawstring+".0"
        return rawstring
except:
    def formatfloat(num):
        return str(num)
def hasher(keyword): #32bit FNV-1 hash with FNV_offset_basis = 5381 and FNV_prime = 33
    hash = 5381
    for byte in keyword:
        hash = (hash*33) ^ ord(byte)
    return hash & 0xffffffff # use & because Python promotes the num instead of intended overflow
class Header:
    def __init__(self,varList): ##all 4byte unsigned integers
        self.absStringOffset     = varList[0]  ## absolute offset for string section start
        self.lenStringToEOF      = varList[1]  ## length from string section start to EOF
        self.numGUID             = varList[2]  ## number of external GUIDs
        self.null                = varList[3]  ## 00000000
        self.numInstanceRepeater = varList[4]
        self.numComplex          = varList[5]  ## number of complex entries
        self.numField            = varList[6]  ## number of field entries
        self.lenName             = varList[7]  ## length of name section including padding
        self.lenString           = varList[8]  ## length of string section including padding
        self.numArrayRepeater    = varList[9]
        self.lenPayload          = varList[10] ## length of normal payload section; the start of the array payload section is absStringOffset+lenString+lenPayload
class FieldDescriptor:
    def __init__(self,varList,keywordDict):
        self.name            = keywordDict[varList[0]]
        self.type            = varList[1]
        self.ref             = varList[2] #the field may contain another complex
        self.offset          = varList[3] #offset in payload section; relative to the complex containing it
        self.secondaryOffset = varList[4]
class ComplexDescriptor:
    def __init__(self,varList,keywordDict):
        self.name            = keywordDict[varList[0]]
        self.fieldStartIndex = varList[1] #the index of the first field belonging to the complex
        self.numField        = varList[2] #the total number of fields belonging to the complex
        self.alignment       = varList[3]
        self.type            = varList[4]
        self.size            = varList[5] #total length of the complex in the payload section
        self.secondarySize   = varList[6] #seems deprecated
class InstanceRepeater:
    def __init__(self,varList):
        self.null            = varList[0] #called "internalCount", seems to be always null
        self.repetitions     = varList[1] #number of instance repetitions
        self.complexIndex    = varList[2] #index of complex used as the instance
class arrayRepeater:
    def __init__(self,varList):
        self.offset          = varList[0] #offset in array payload section
        self.repetitions     = varList[1] #number of array repetitions
        self.complexIndex    = varList[2] #not necessary for extraction
class Complex:
    def __init__(self,desc,dbxhandle):
        self.desc=desc
        self.dbx=dbxhandle #lazy
    def get(self,name):
        pathElems=name.split("/")
        curPos=self
        if pathElems[-1].find("::")!=-1: #grab a complex
            for elem in pathElems:
                try:
                    curPos=curPos.go1(elem)
                except Exception,e:
                    raise Exception("Could not find complex with name: "+str(e)+"\nFull path: "+name+"\nFilename: "+self.dbx.trueFilename)
            return curPos
        #grab a field instead
        for elem in pathElems[:-1]:
            try:
                curPos=curPos.go1(elem)
            except Exception,e:
                raise Exception("Could not find complex with name: "+str(e)+"\nFull path: "+name+"\nFilename: "+self.dbx.trueFilename)
        for field in curPos.fields:
            if field.desc.name==pathElems[-1]:
                return field
            
        raise Exception("Could not find field with name: "+name+"\nFilename: "+self.dbx.trueFilename)

    def go1(self,name): #go once
        for field in self.fields:
            if field.desc.type in (0x0029, 0xd029,0x0000,0x0041):
                if field.desc.name+"::"+field.value.desc.name == name:
                    return field.value
        raise Exception(name)


class Field:
    def __init__(self,desc,dbx):
        self.desc=desc
        self.dbx=dbx
    def link(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)
        
        if self.value>>31:
            extguid=self.dbx.externalGUIDs[self.value&0x7fffffff]
            
            for existingDbx in dbxArray:
                if existingDbx.fileGUID==extguid[0]:
                    for guid, instance in existingDbx.instances:
                        if guid==extguid[1]:
                            return instance
                    

            f=valid(inputFolder+guidTable[extguid[0]]+".ebx")
##            print guidTable[extguid[0]]
            dbx=Dbx(f)
            dbxArray.append(dbx)
            for guid, instance in dbx.instances:
                if guid==extguid[1]:
                    return instance
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
        elif self.value!=0:
            for guid, instance in self.dbx.instances:
                if guid==self.dbx.internalGUIDs[self.value-1]:
                    return instance
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)

        raise Exception("Invalid link, could not find target.")

    def getlinkguid(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)

        if self.value>>31:
            return "".join(self.dbx.externalGUIDs[self.value&0x7fffffff])
        elif self.value!=0:
            return self.dbx.fileGUID+self.dbx.internalGUIDs[self.value-1]
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
    def getlinkname(self):
        if self.desc.type!=0x0035: raise Exception("Invalid link, wrong field type\nField name: "+self.desc.name+"\nField type: "+hex(self.desc.type)+"\nFile name: "+self.dbx.trueFilename)

        if self.value>>31:
            return guidTable[self.dbx.externalGUIDs[self.value&0x7fffffff][0]]+"/"+self.dbx.externalGUIDs[self.value&0x7fffffff][1]
        elif self.value!=0:
            return self.dbx.trueFilename+"/"+self.dbx.internalGUIDs[self.value-1]
        else:
            raise nullguid("Nullguid link.\nFilename: "+self.dbx.trueFilename)
    

         
def valid(fname):
    f=open2(fname,"rb")
    if f.read(4) not in ("\xCE\xD1\xB2\x0F","\x0F\xB2\xD1\xCE"):
        f.close()
        raise Exception("nope")
    return f

class nullguid(Exception):
    def __init__(self, value):
        self.value = value
    def __str__(self):
        return repr(self.value)


numDict={0x0035:("I",4),0xc10d:("I",4),0xc14d:("d",8),0xc0ad:("?",1),0xc0fd:("i",4),0xc0bd:("B",1),0xc0ed:("h",2), 0xc0dd:("H",2), 0xc13d:("f",4)}


class Stub:
    pass



class Dbx:
    def __init__(self, f,unpacker):
        self.unpack=unpacker
        self.trueFilename=""
        self.header=Header(self.unpack("11I",f.read(44)))
        self.arraySectionstart=self.header.absStringOffset+self.header.lenString+self.header.lenPayload
        self.fileGUID, self.primaryInstanceGUID = f.read(16), f.read(16)    
        self.externalGUIDs=[(f.read(16),f.read(16)) for i in xrange(self.header.numGUID)]
        self.keywords=str.split(f.read(self.header.lenName),"\x00")
        self.keywordDict=dict((hasher(keyword),keyword) for keyword in self.keywords)
        self.fieldDescriptors=[FieldDescriptor(self.unpack("IHHII",f.read(16)), self.keywordDict) for i in xrange(self.header.numField)]
        self.complexDescriptors=[ComplexDescriptor(self.unpack("IIBBHHH",f.read(16)), self.keywordDict) for i in xrange(self.header.numComplex)]
        self.instanceRepeaters=[InstanceRepeater(self.unpack("3I",f.read(12))) for i in xrange(self.header.numInstanceRepeater)] 
        while f.tell()%16!=0: f.seek(1,1) #padding
        self.arrayRepeaters=[arrayRepeater(self.unpack("3I",f.read(12))) for i in xrange(self.header.numArrayRepeater)]

        #payload
        f.seek(self.header.absStringOffset+self.header.lenString)
        self.internalGUIDs=[]
        self.instances=[] # (guid, complex)
        for instanceRepeater in self.instanceRepeaters:
            for repetition in xrange(instanceRepeater.repetitions):
                instanceGUID=f.read(16)
                self.internalGUIDs.append(instanceGUID)
                if instanceGUID==self.primaryInstanceGUID:
                    self.isPrimaryInstance=True
                else:
                    self.isPrimaryInstance=False
                inst=self.readComplex(instanceRepeater.complexIndex,f)
                inst.guid=instanceGUID
                
                if self.isPrimaryInstance: self.prim=inst
                self.instances.append((instanceGUID,inst))
    
        f.close()
        

    def readComplex(self, complexIndex,f):
        complexDesc=self.complexDescriptors[complexIndex]
        cmplx=Complex(complexDesc,self)
        
        startPos=f.tell()                 
        cmplx.fields=[]
        for fieldIndex in xrange(complexDesc.fieldStartIndex,complexDesc.fieldStartIndex+complexDesc.numField):
            f.seek(startPos+self.fieldDescriptors[fieldIndex].offset)
            cmplx.fields.append(self.readField(fieldIndex,f))
        
        f.seek(startPos+complexDesc.size)
        return cmplx


    def readField(self,fieldIndex,f):
        fieldDesc = self.fieldDescriptors[fieldIndex]
        field=Field(fieldDesc,self)
        
        if fieldDesc.type in (0x0029, 0xd029,0x0000):
            field.value=self.readComplex(fieldDesc.ref,f)
        elif fieldDesc.type==0x0041:
            arrayRepeater=self.arrayRepeaters[self.unpack("I",f.read(4))[0]]
            arrayComplexDesc=self.complexDescriptors[fieldDesc.ref]

##            if arrayRepeater.repetitions==0: field.value = "*nullArray*"
            f.seek(self.arraySectionstart+arrayRepeater.offset)
            arrayComplex=Complex(arrayComplexDesc,self)
            arrayComplex.fields=[self.readField(arrayComplexDesc.fieldStartIndex,f) for repetition in xrange(arrayRepeater.repetitions)]
            field.value=arrayComplex
            
        elif fieldDesc.type in (0x407d, 0x409d):
            startPos=f.tell()
            f.seek(self.header.absStringOffset+self.unpack("I",f.read(4))[0])
            string=""
            while 1:
                a=f.read(1)
                if a=="\x00": break
                else: string+=a
            f.seek(startPos+4)
            
            if len(string)==0: field.value="*nullString*" #actually the string is ""
            else: field.value=string
            
            if self.isPrimaryInstance and self.trueFilename=="" and fieldDesc.name=="Name": self.trueFilename=string
            
                   
        elif fieldDesc.type==0x0089: #incomplete implementation, only gives back the selected string
            compareValue=self.unpack("I",f.read(4))[0] 
            enumComplex=self.complexDescriptors[fieldDesc.ref]

            if enumComplex.numField==0:
                field.value="*nullEnum*"
            for fieldIndex in xrange(enumComplex.fieldStartIndex,enumComplex.fieldStartIndex+enumComplex.numField):
                if self.fieldDescriptors[fieldIndex].offset==compareValue:
                    field.value=self.fieldDescriptors[fieldIndex].name
                    break
        elif fieldDesc.type==0xc15d:
            field.value=f.read(16)
        elif fieldDesc.type == 0xc13d: ################################
            field.value=formatfloat(self.unpack("f",f.read(4))[0])
        else:
            (typ,length)=numDict[fieldDesc.type]
            num=self.unpack(typ,f.read(length))[0]
            field.value=num
        
        return field
        

    def dump(self,outputFolder):
        dirName=os.path.dirname(outputFolder+self.trueFilename)
        if not os.path.isdir(dirName): os.makedirs(dirName)
        f2=open2(outputFolder+self.trueFilename+EXTENSION,"wb")
        print self.trueFilename
        
        for (guid,instance) in self.instances:
            if guid==self.primaryInstanceGUID: f2.write(instance.desc.name+" "+hexlify(guid)+ " #primary instance\r\n")
            else: f2.write(instance.desc.name+" "+hexlify(guid)+ "\r\n")
            self.recurse(instance.fields,f2,0)
        f2.close()

    def recurse(self, fields,f2, lvl): #over fields
        lvl+=1
        for field in fields:
            if field.desc.type in (0xc14d, 0xc0fd, 0xc10d, 0xc0ed, 0xc0dd, 0xc0bd, 0xc0ad, 0x407d, 0x409d, 0x0089):
                f2.write(lvl*SEP+field.desc.name+" "+str(field.value)+"\r\n")
            elif field.desc.type == 0xc13d:
                f2.write(lvl*SEP+field.desc.name+" "+formatfloat(field.value)+"\r\n")
            elif field.desc.type == 0xc15d:
                f2.write(lvl*SEP+field.desc.name+" "+hexlify(field.value)+"\r\n")
            elif field.desc.type == 0x0035:
                towrite=""
                if field.value>>31:
                    extguid=self.externalGUIDs[field.value&0x7fffffff]
                    try: towrite=guidTable[extguid[0]]+"/"+hexlify(extguid[1])
                    except: towrite=hexlify(extguid[0])+"/"+hexlify(extguid[1])
                elif field.value==0: towrite="*nullGuid*"
                else: towrite=hexlify(self.internalGUIDs[field.value-1])
                f2.write(lvl*SEP+field.desc.name+" "+towrite+"\r\n") 
            elif field.desc.type==0x0041 and len(field.value.fields)==0:
                f2.write(lvl*SEP+field.desc.name+" "+"*nullArray*"+"\r\n")
            else:
                f2.write(lvl*SEP+field.desc.name+"::"+field.value.desc.name+"\r\n")
                self.recurse(field.value.fields,f2,lvl)


    def decode(self):
        if not self.prim.desc.name=="SoundWaveAsset": return

        histogram=dict() #count the number of times each chunk is used by a variation to obtain the right index

        Chunks=[]
        for i in self.prim.get("$::SoundDataAsset/Chunks::array").fields:
            chnk=Stub()
            Chunks.append(chnk)
            chnk.ChunkId=i.value.get("ChunkId").value
            chnk.ChunkSize=i.value.get("ChunkSize").value

            
        variations=[i.link() for i in self.prim.get("Variations::array").fields]

        Variations=[]
        
        for var in variations:
            Variation=Stub()
            Variations.append(Variation)
            Variation.ChunkIndex=var.get("ChunkIndex").value
            Variation.SeekTablesSize=var.get("SeekTablesSize").value
            Variation.FirstLoopSegmentIndex=var.get("FirstLoopSegmentIndex").value
            Variation.LastLoopSegmentIndex=var.get("LastLoopSegmentIndex").value


            Variation.Segments=[]
            segs=var.get("Segments::array").fields
            for seg in segs:
                Segment=Stub()
                Variation.Segments.append(Segment)
                Segment.SamplesOffset = seg.value.get("SamplesOffset").value
                Segment.SeekTableOffset = seg.value.get("SeekTableOffset").value
                Segment.SegmentLength = seg.value.get("SegmentLength").value

            Variation.ChunkId=hexlify(Chunks[Variation.ChunkIndex].ChunkId)
            Variation.ChunkSize=Chunks[Variation.ChunkIndex].ChunkSize
        

            #find the appropriate index
            if Variation.ChunkIndex in histogram: #has been used previously already
                Variation.Index=histogram[Variation.ChunkIndex]
                histogram[Variation.ChunkIndex]+=1
            else:
                Variation.Index=0
                histogram[Variation.ChunkIndex]=1

                
        #everything is laid out neatly now
        #ChunkId, ChunkSize, Index, ChunkIndex, SeekTablesSize, FirstLoopSegmentIndex, LastLoopSegmentIndex
        #Segments with SamplesOffset, SeekTableOffset, SegmentLength

        ChunkHandles=dict() #for each ebx, keep track of all file handles

        for Variation in Variations:
            try:
                f=ChunkHandles[Variation.ChunkId]
            except:
                try:
                    f=open2(chunkFolder+Variation.ChunkId+".chunk")
                    currentChunkName=chunkFolder+Variation.ChunkId+".chunk"
                except IOError:
                    try:
                        f=open2(chunkFolder2+Variation.ChunkId+".chunk")
                        currentChunkName=chunkFolder2+Variation.ChunkId+".chunk"
                    except:
                        print "Chunk does not exist: "+Variation.ChunkId+" "+self.trueFilename
                        return
                ChunkHandles[Variation.ChunkId]=f


            for ijk in xrange(len(Variation.Segments)):
                Segment=Variation.Segments[ijk]
                f.seek(Segment.SamplesOffset)
                magic=f.read(4)
                
                if magic!="\x48\x00\x00\x0c":
                    raise Exception("Wrong XAS magic.")

                audioType=f.read(1) #0x14 is DICE XAS, 0x16 is ealayer3, 0x12 is unknown
                if audioType=="\x16":
                    if not ealayerSupport:
                        print "EALayer3 segment not converted as the tool is missing: "+Variation.ChunkId+" "+self.trueFilename
                        continue
                    target=os.path.join(outputFolder,self.trueFilename)+" "+str(Variation.ChunkIndex)+" "+str(Variation.Index)+" "+str(ijk)+".wav"
                    makeLongDirs(target) #prepare the folder structure

                    process = subprocess.Popen([ealayer3Path,currentChunkName,"-i",str(Segment.SamplesOffset),"-o",target,"-s","-w"], stderr=subprocess.PIPE,startupinfo=startupinfo)
                    
                    process.communicate() #this should set the returncode
                    if process.returncode:
                        print process.stderr.readlines()

                    continue
                elif audioType!="\x14":
                    print "Unknown audio segment (type "+hexlify(audioType)+"): "+Variation.ChunkId+" "+self.trueFilename
                    continue

                

                channelRaw=f.read(1)
                if channelRaw not in channelDict:
                    print self.trueFilename
                
                    
                numChannels=channelDict[channelRaw]
                samplingRate=unpack(">H",f.read(2))[0] #[48000, 11025, 16000, 44100, 24000]
                f.read(4) #the first byte is always 0x40, the second takes one of 82 values, third and fourth can each take any value (i.e. 00 to ff)
                #82 values: ['\x01', '\x00', '\x03', '\x02', '\x05', '\x04', '\x07', '\x06', '\t', '\x08', '\x0b', '\n', '\r', '\x0c', '\x0f', '\x0e', '\x11', '\x10', '\x13', '\x12', '\x15', '\x14', '\x17', '\x16', '\x19', '\x18', '\x1b', '\x1a', '\x1d', '\x1c', '\x9f', '\x1e', '!', ' ', '#', '"', '%', '$', "'", '&', ')', '(', '+', '*', '-', ',', '/', '.', '1', '0', '3', '2', '5', '\x1f', '7', '6', '9', ';', '<', '?', '>', '\xc1', 'a', 'C', 'E', 'D', '\xa0', 'I', 'H', 'K', 'J', 'L', 'Y', 'X', '\x8e', '\x87', '4', 'A', 'd', 'F', '\x95', '\x7f']
                
                payload=""
                target= os.path.join(outputFolder,self.trueFilename)+" "+str(Variation.ChunkIndex)+" "+str(Variation.Index)+" "+str(ijk)+".wav"
                targetFolder=os.path.dirname(target)
                if not os.path.isdir(targetFolder): os.makedirs(targetFolder)

                f2=StringIO()
                
                write_header(f2, samplingRate, numChannels)
                
                while 1:
                    v1,length=unpack(">HH",f.read(4))

                    if length<=4: break
                    length-=8
                    
                    v2,v3=unpack(">HH",f.read(4))

                    if length%76!=0:
                        raise Exception("Strange length: "+str(length))
                        
                    atoms=length/76

                    blocks=[None]*numChannels
                    
                    for atom in xrange(0,atoms,numChannels):
                        for i in xrange(numChannels):
                            blocks[i]=decode(f.read(76))
                            
                        atomData=""
                        for i in xrange(128):
                            for j in xrange(numChannels):
                                atomData+=pack("f",blocks[j][i])
               
                        f2.write(atomData)

                finalPos=f2.tell()
                f2.seek(4)    
                f2.write(pack("I",finalPos-8))
                f2.seek(76)
                f2.write(pack("I",finalPos-76))

                f3=open2(target,"wb")
                f3.write(f2.getvalue())
                
                f2.close()

        for key in ChunkHandles:
            ChunkHandles[key].close()



def write_header(fp, samplingRate, nchannels):
    out = pack('4sl4s','RIFF',0,'WAVE')
    floatSize = 4
    fmt_size = 40
    tag = 0xFFFE
    etag = 3

    out += pack('4slHHllHH', 'fmt ', fmt_size, tag, nchannels, samplingRate,
                nchannels * samplingRate * floatSize, nchannels * floatSize, floatSize * 8)

    out += pack('HHlH14s', 22, floatSize * 8, (1 << nchannels) - 1, etag,
                '\x00\x00\x00\x00\x10\x00\x80\x00\x00\xaa\x008\x9b\x71')

    out += pack("<4sll","fact",4,floatSize)
    out += pack("4sl","data",0)

    fp.seek(0)
    fp.write(out)

    
xaslib = cdll.LoadLibrary("xas")
def decode(inputChars):
    charType = c_char * 76
    charBuf = charType(*inputChars)
    charbufpointer = pointer(charBuf)
    
    floatType = c_float * 128
    floatBuf = floatType()
    floatbufpointer = pointer(floatBuf)
    
    xaslib.decode(charbufpointer, floatbufpointer)
    
    return floatBuf


channelDict={"\x00":1,"\x04":2,"\x0c":4,"\x14":6,"\x1c":8,"\x05":2,"\x01":1}


if outputFolder[-1] not in ("/","\\"): outputFolder+="/"
if ebxFolder[-1] not in ("/","\\"): ebxFolder+="/"
if chunkFolder[-1] not in ("/","\\"): chunkFolder+="/"
if chunkFolder2[-1] not in ("/","\\"): chunkFolder2+="/"


#for ealayer. By default Python opens a new window for a split second and puts focus on it. This info makes no window show up at all.
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
startupinfo.wShowWindow = subprocess.SW_HIDE


try:
    subprocess.Popen([ealayer3Path,"-s","-w"],startupinfo=startupinfo)
    ealayerSupport=True
    print "EALayer3 tool detected."
except WindowsError:
    ealayerSupport=False
    print "EALayer3 tool not detected."

    
decodeAudio()
For PS3 users:
From Frankelstner:

ChunkId FECAAFC2309C055D7E7D61C4C669F9C0 in EBX

Chunk ID In the filechunk name: c2afcafe9c305d057e7d61c4c669f9c0

lol

It's simple as pie.

FE CA AF C2 30 9C 05 5D 7E7D61C4C669F9C0
C2 CA AF FE 30 9C 05 5D 7E7D61C4C669F9C0 #0 with 3
C2 AF CA FE 30 9C 05 5D 7E7D61C4C669F9C0 #1 with 2
C2 AF CA FE 9C 30 05 5D 7E7D61C4C669F9C0 #4 with 5
C2 AF CA FE 9C 30 5D 05 7E7D61C4C669F9C0 #7 with 6

Ergo: 3 2 1 0 5 4 7 6 8 9 10 11 12 13 14 15
So, look at line 20 in the script and change False to True to allow the obfuscation permutation.

This one:

Code: Select all

ChunkIdObfuscation=False #set to True for some console games when the script does not find any chunks at all
obfuscationPermutation=[3,2,1,0,5,4,7,6,8,9,10,11,12,13,14,15]
As OrangeC mentionned, it works for PS3 version as they are EALayer3 encoded.
Last edited by Vosvoy on Fri May 10, 2013 10:59 pm, edited 3 times in total.
Vosvoy

Post Reply