Page 1 of 3
publicly importing DoA3 to Noesis(voyearism, gang bang)
Posted: Sat Jul 13, 2013 10:59 am
by b0ny
hi all.
I'm a lazy person, and it's only getting worse. I want to make a DoA3 importer(python), to learn how to make a
noesis plugin, and how to do both "the right way"(if there is any), but my laziness is killing me.
So I decided to do this on public, so there would be a chance to be publicly embarrassed and this won't let me relax too much. Also i hope you guys can help me if I'm stuck.
My plan is to:
- Reverse the doa3 "xpr" and "cat" formats (and maybe "mot")
- Import as much information as possible to
noesis
- Write a nice descriptive code, with functions and whatever the real programmers use to make programs.
- Use advices of the
xentax community when stuck.
As you see this is a very clever plan, with many idealistic goals. I'll post each time I'll have a progress, thus I'll double/triple post, so in the first place
i ask for permission to create this thread.
My second question is pure technical -
does Noesis support morphing animation?
I hope i can have your assistance when I'll have a question.
Re: publicly importing DoA3 to Noesis(voyearism, gang bang)
Posted: Sat Jul 13, 2013 11:19 am
by chrrox
Re: publicly importing DoA3 to Noesis(voyearism, gang bang)
Posted: Sat Jul 13, 2013 12:57 pm
by b0ny
Thanks. Good news for me
______________________
First of all i downloaded latest version of Noesis(and "PyScripter" for editing python scripts.)
Copypasted the template for Noesis python plugin from
forum.xentax.com/viewtopic.php?t=7760(thank you Chrrox one more time). Found that
bs.read("I") will read a tuple and
bs.read("I")[0] will read an int, and also
bs.readUInt() will read an int.
I have a request for SeƱor Casaroja. To add to reading functions like bs.readUInt() the functionality of bs.seek()
when importing you have to read a lot from file. this will make things more convenient. so this code
Code: Select all
bs.seek(0x28)
nameForOffset = bs.readUInt()
bs.seek(nameForOffset)
bs.readFloat()
could be written like this
and the function without arguments(bs.readUInt()) will behave like it does now
Re: publicly importing DoA3 to Noesis(voyearism, gang bang)
Posted: Sat Jul 13, 2013 2:17 pm
by MrAdults
That wouldn't make much sense. If anything, you'd want that argument to be size in the native type (so, count) instead of bytes, that's the point of using typed reads. You're also trying to read a float from a list of 0x28 ints, you wouldn't just seek to 0x28 and then read a single int, as that would also be unintuitive default behavior for the argument. If you want behavior like that, I'd suggest making your own passthrough class around NoeBitStream and using that instead. That way you can do whatever kind of madness with your arguments that you want. Or you can use an entirely different bit/byte reader class, it's up to you. You're given the raw bytearray representing the data, so you can do whatever you want.
I'd also suggest not getting caught up on semantics as you dive into things, it will only hold you back and give you an excuse to delay tackling the hard stuff.

Re: publicly importing DoA3 to Noesis(voyearism, gang bang)
Posted: Sat Jul 13, 2013 3:16 pm
by b0ny
MrAdults wrote:That wouldn't make much sense. If anything, you'd want that argument to be size in the native type (so, count) instead of bytes, that's the point of using typed reads. You're also trying to read a float from a list of 0x28 ints, you wouldn't just seek to 0x28 and then read a single int, as that would also be unintuitive default behavior for the argument. If you want behavior like that, I'd suggest making your own passthrough class around NoeBitStream and using that instead. That way you can do whatever kind of madness with your arguments that you want. Or you can use an entirely different bit/byte reader class, it's up to you. You're given the raw bytearray representing the data, so you can do whatever you want.
I'd also suggest not getting caught up on semantics as you dive into things, it will only hold you back and give you an excuse to delay tackling the hard stuff.

My logic tells me that every
seek is always followed by a
read, and making them one, is what my logic desires. I thought that could help beginners that only start to make python scripts (thats imho)
I'm stuck - where can i find information on "rapi" methods?
I need to deswizzle xbox1 images,
and i need to find the Noesis plugin folder to use the doa3 skeletons ripped from the executable,
what dds formats are supported by Noesis
Re: publicly importing DoA3 to Noesis(voyearism, gang bang)
Posted: Sat Jul 13, 2013 3:24 pm
by MrAdults
I know why you want it, it just doesn't make sense as a default argument for those functions. You can take any of the approaches I mentioned instead.
Xbox is just morton order tiling. chrrox has some scripts that demonstrate how to untile using noesis.morton2D, like his Naughty Dog one.
All the info that exists for the rapi and noesis modules is in __NPReadMe.txt and scattered throughout these forums.
Lots of DDS pixel formats are supported, but I'm guessing you meant DXT. DXT1-5, BC4, BC5, and some custom packing methods are supported. DXT1, 3, and 5 are natively supported as texture formats, others need to be decoded (before passing back as native texture data) using imageDecodeDXT.
Re: publicly importing DoA3 to Noesis(voyearism, gang bang)
Posted: Sat Jul 13, 2013 4:53 pm
by b0ny
I know I'm pretty slow and I need to be faster if I don't want to loose all my precious voyeurs...
At this point this script can import textures:
Code: Select all
#Noesis Python model import+export test module, imports/exports some data from/to a made-up format
from inc_noesis import *
import noesis
import rapi #rapi methods should only be used during handler callbacks
#registerNoesisTypes is called by Noesis to allow the script to register formats.
#Do not implement this function in script files unless you want them to be dedicated format modules!
def registerNoesisTypes():
handle = noesis.register("Dead or Alive 3", ".xpr")
noesis.setHandlerTypeCheck(handle, noepyCheckType)
noesis.setHandlerLoadModel(handle, noepyLoadModel) #see also noepyLoadModelRPG
noesis.logPopup()
#print("The log can be useful for catching debug prints from preview loads.\nBut don't leave it on when you release your script, or it will probably annoy people.")
return 1
loadDoA3Only = False #doa3 only or also doa2u and doax
#check if it's this type based on the data
def noepyCheckType(data):
bs = NoeBitStream(data)
if bs.readUInt() != 0x30525058: #'XPR0' magic
print("no xpr0 magic(30525058)")
return 0
bs.seek(0x14) #Seek to MDL
if bs.readUInt() != 0x4C444D: #'MDL' magic
print("no mdl magic(4c444d)")
return 0
if loadDoA3Only:
bs.seek(0x28)
OBJoffset = bs.readUInt()
bs.seek(OBJoffset + 8)
if bs.readUInt() != 0:#OBJoffset + 8 = 0 for doa3 and 4 for doax doa2u
print("no obj+8", OBJoffset + 8)
return 0
return 1
#load the model
def noepyLoadModel(data, mdlList):
ctx = rapi.rpgCreateContext()
bs = NoeBitStream(data)
print("hi there")
#importing code goes here
bs.seek(0) #start from the file beginning
xprMagic = bs.readUInt()
xprSize = bs.readUInt()
dataBlockOffset = bs.readUInt()
#parse xbox1 xpr chunks
vertexBuffersOffsets = []
imagesInfos = []
while True:
chunkBase = bs.tell()
chunkID = bs.readUInt()
#80000000-MDL; 40001-texture; 800001-vbuf; 830001-???
chunkSize = {0x800001:0xC, 0x40001:0x14, 0x830001:0xC}.get(chunkID, -1)
if chunkID == 0x80000000:
chunkSize = bs.readUInt() + 8
elif chunkID == 0x800001:
vertexBuffersOffsets.append(dataBlockOffset + bs.readUInt())
elif chunkID == 0x40001:
textureOffset = bs.readUInt()
unknownAlways0 = bs.readUInt()
flags = bs.readUInt()
imagesInfos.append((dataBlockOffset + textureOffset, flags))#absolute offset and flags
elif chunkID == 0xffffffff:
break
elif chunkID != 0x80000000:
print("not a doa3 chunk %x"%chunkID)
break
bs.seek(chunkBase + chunkSize)#seek to next chunk
#load images
texList = []
texNames = []
cleanName = rapi.getInputName()
cleanName = cleanName[cleanName.rfind("\\")+1: -4]#get the file name without extension
for textureIndex, (textureOffset, flags) in enumerate(imagesInfos):
doafmt = (flags & 0xFF00) >> 8
Width = 1 << ((flags & 0xF00000) >> 20)
Height = 1 << ((flags & 0xF000000) >> 24)
bpp = {6:4, 0x28:2, 0x19:1, 0xC:0.5, 0xF:1}[doafmt] #6-ARGB32 28-GB16 19-A8 C-DXT1 F-DXT5
noMipSize = int(Width * Height * bpp)
bs.seek(textureOffset)
textureData = bs.readBytes(noMipSize)
textureName = cleanName + "_%02d"%textureIndex
if doafmt == 0xC:
textureFormat = noesis.NOESISTEX_DXT1
elif doafmt == 0xF:
textureFormat = noesis.NOESISTEX_DXT5
elif doafmt in (6, 0x28, 0x19):
argbfmt = {6:"b8g8r8a8", 0x28:"b8g8", 0x19:"r8"}[doafmt]#why argb is reversed, big endian?
untwid = bytearray()
for x in range(Width):
for y in range(Height):
idx = noesis.morton2D(x, y)
untwid += textureData[idx*bpp:idx*bpp+bpp]
textureData = rapi.imageDecodeRaw(untwid, Width, Height, argbfmt)
textureFormat = noesis.NOESISTEX_RGBA32
else:
print("Unsuported texture format - %x"%doafmt)
texList.append(NoeTexture(textureName, Width, Height, textureData, textureFormat))
texNames.append(textureName)
#find doa version (3/2U/X)
doaVesion = 0
bs.seek(0x28)
OBJoffset = bs.readUInt()
bs.seek(OBJoffset + 8)
if bs.readUInt() != 0:#[OBJoffset + 8] = 0 for doa3 and 4 for doax doa2u
doaVersion = 3 #doa3
else:
doaVersion = 2 #doa2U (or doaX)
if doaVersion != 3:#stop parsing if not doa3
mdl = NoeModel([])
mdl.setModelMaterials(NoeModelMaterials(texList, []))
mdlList.append(mdl)
rapi.rpgClearBufferBinds()
return 1
#test load the vertex buffers
pass
#end
rapi.rpgClearBufferBinds()
return 1
Re: publicly importing DoA3 to Noesis(voyearism, gang bang)
Posted: Sat Jul 13, 2013 9:45 pm
by shakotay2
MrAdults wrote:I'd also suggest not getting caught up on semantics as you dive into things, it will only hold you back and give you an excuse to delay tackling the hard stuff.

My today's deja-vu! 
(Last monday morning: my teamleader speaking to me.)
b0ny wrote:hi all.
I'm a lazy person, and it's only getting worse.
The teamleader is watching you, dude!

Re: publicly importing DoA3 to Noesis(voyearism, gang bang)
Posted: Sun Jul 14, 2013 5:44 am
by b0ny
shakotay2 wrote:MrAdults wrote:I'd also suggest not getting caught up on semantics as you dive into things, it will only hold you back and give you an excuse to delay tackling the hard stuff.

My today's deja-vu! 
(Last monday morning: my teamleader speaking to me.)
b0ny wrote:hi all.
I'm a lazy person, and it's only getting worse.
The teamleader is watching you, dude!

I'm not sure if someone can do something with this situation
_______________
I'm confused with importing normals to Noesis:
where do i put the normals when using this function?
Code: Select all
mesh = NoeMesh(faces, vertices, meshName, meshName)
Code: Select all
rapi.rpgBindPositionBufferOfs(VertBuff, noesis.RPGEODATA_FLOAT, 0x20, 0)
rapi.rpgBindNormalBufferOfs(VertBuff, noesis.RPGEODATA_FLOAT, 0x20, 12)
rapi.rpgBindUV1BufferOfs(VertBuff, noesis.RPGEODATA_FLOAT, 0x20, 24)
rapi.rpgCommitTriangles(facesdata, noesis.RPGEODATA_USHORT, int(len(facesdata)/2), noesis.RPGEO_TRIANGLE, 1)
rpgCommitTriangles wants binary buffer with triangles, while i have a tristrip buffer, i converted manually this tristrip buffer by operating with pairs of bytes to emulate shorts length. this doesn't look right too me with my list of tristrips.
(i just realized the concept of bindings - the vertex data layers can be in different buffers)
the doa3 xpr format looks like this:
Code: Select all
mdl:
obj1:
vertex buffer 1 (pos,normal,uv):
number of verteices
vertex buffer offset
number of indices
tristrip buffer offset
vertex buffer 2 (?)
vertex buffer 3 (pos,color,uv)
vertex buffer 4 (?)
material 1:
material color
texture index
used vertex buffer
range of used tristrip indices(startindex, number of indices)
material 2
material 3
obj2
obj3
my idea is to iterate through the materials that use the same vertex buffer and create a mesh, but how do i assign images then. if only one material per mesh can be used then it will be a pain to create a new list of vertices used by a material, to create an object with a single material
[edit]
ARRRRGH! this gang bang is ruined!

Re: publicly importing DoA3 to Noesis(voyearism, gang bang)
Posted: Mon Jul 15, 2013 12:03 am
by MrAdults
You can set normals on a NoeMesh after creating it, but you shouldn't be making NoeMeshes yourself. Stick to the rpg interface.
For triangle strips, simply use RPGEO_TRIANGLE_STRIP instead of RPGEO_TRIANGLE.
The rpg interface will allow you to specify materials on a per-triangle basis. Just use rpgSetMaterial before the commit.
Re: publicly importing DoA3 to Noesis(voyearism, gang bang)
Posted: Mon Jul 15, 2013 6:30 am
by b0ny
MrAdults wrote:The rpg interface will allow you to specify materials on a per-triangle basis. Just use rpgSetMaterial before the commit.
Isn't this this just a convention - I still have to load the data per material(one material - one commit)?
Is there any way to load the materials post factum(after committing), I mean not [one mesh/commit one material] but [one mesh/commit many materials]. Though maybe inside the Noesis, when the data is loaded using the rpg interface, it uses multiple materials per mesh, this rpg interface puts the entire model in one mesh, which is bad for exporting, coz you don't have per mesh names in this case. Is this possible somehow to "make commits to different meshes"?
What am I doing wrong?
Re: publicly importing DoA3 to Noesis(voyearism, gang bang)
Posted: Tue Jul 16, 2013 3:23 am
by MrAdults
You're misunderstanding something. You can specify per triangle materials with the RPG interface. Simply commit different sections of your index buffer with different rpgSetMaterial calls.
You refer to "multiple materials per mesh", which makes no sense. Either you're confusing "material" and "texture", or you want to do blended multi-pass rendering, which is what NoeMaterial's setNextPass is for.
Re: publicly importing DoA3 to Noesis(voyearism, gang bang)
Posted: Tue Jul 16, 2013 8:46 am
by b0ny
MrAdults wrote:You're misunderstanding something. You can specify per triangle materials with the RPG interface. Simply commit different sections of your index buffer with different rpgSetMaterial calls.
You refer to "multiple materials per mesh", which makes no sense. Either you're confusing "material" and "texture", or you want to do blended multi-pass rendering, which is what NoeMaterial's setNextPass is for.
I see... But inside noesis each of this commits is saved to a separate mesh(if a material is set for it) for which I have to specify a unique name. It's like the mesh is some property of the maerial and not viceversa.
DoA3 model is splited in a lot of meshes(bodyparts like: hand,wrist,forearm,elbow,arm,...), and has tons of materials per mesh(one material per texture), if exporting souch a model from noesis and importing somewhere else you'll get tons of objects, it will get a lot of time to understand where each of these pieces goes.
The only program I made importing scripts for is Blender, and it allows me to load the mesh and then iterate through triangles and set a certain material for it. Maybe i should be more flexible and nor complain about one program using it's own way(Blender has it's own flaws i can complain about all day long

)
Anyway Noesis is a great program. But I'm not good enough for it, I need some time to get better, find a more decent job and get more responsible. Then I'll have a happy long life with you - Noesis.
Sorry guys, this show is over seems like I'm not the guy

Re: publicly importing DoA3 to Noesis(voyearism, gang bang)
Posted: Tue Jul 16, 2013 4:43 pm
by MrAdults
That's incorrect. Commits are pooled in a context buffer. You can do as many commits for a single material as you want.
Re: publicly importing DoA3 to Noesis(voyearism, gang bang)
Posted: Wed Jul 17, 2013 4:52 pm
by b0ny
MrAdults wrote:That's incorrect. Commits are pooled in a context buffer. You can do as many commits for a single material as you want.
ok.
MrAdults, can you please help me with something. i now you know a lot about vectors and things, and i lack qualification to fix this.
the story is - the doa3 tristrip is split in "subsets", some of those subsets have a counterclockwise winding and it's impossible to now for sure if it's cw or ccw.(
the model looks like this)
so i thought that i could generate a normal from the cw winding and compare it with normals of vertices for the first face, to find out if i need to reverse the winding to point in the "right" direction.
but when calculating the angle between the generated normal and vertice's normals i get a complex number, and python gives me an error because of this. how do i fix the angle function to avoid these complex numbers.
this code bellow will repeat the error i get. please help me fix this. thanks
Code: Select all
import math
#-----------------
#Calculate the cross product of two vectors
def cross(a, b):
c = [a[1]*b[2] - a[2]*b[1],
a[2]*b[0] - a[0]*b[2],
a[0]*b[1] - a[1]*b[0]]
return c
def sub(u, v):
return [ u[i]-v[i] for i in range(len(u)) ]
def normalize(v):
magnitude = math.sqrt(sum(v[i]*v[i] for i in range(len(v))))
return [ v[i]/magnitude for i in range(len(v)) ]
#Calculates the angle between two vectors
def angle(a,b):
dotproduct = sum([a[i]*b[i] for i in range(len(a))])
veclengtha = sum([a[i] for i in range(len(a))])**.5 #Calculates the size of a vector
veclengthb = sum([b[i] for i in range(len(b))])**.5 #Calculates the size of a vector
print(dotproduct/(veclengtha*veclengthb),dotproduct,veclengtha,veclengthb)
return math.acos(min(max(dotproduct/(veclengtha*veclengthb),1),-1))# (dotproduct/(veclengtha*veclengthb) is a complex number and i don't now what to do
def getwinding(v):
print(v)
#generate a normal from triangle winding
windnorm = normalize(cross(sub(v[2][0],v[0][0]), sub(v[1][0],v[0][0])))
#compare generated normal to each of three vertice's normals(if angle is higher than 90 for more then one vertices then CCW)
if sum([(1 if (angle(windnorm,v[i][1]) * 57.2957795) < 90 else -1) for i in range(3)]) > 0: return True
return False
def main():
face = [((-0.05612413212656975, 0.09935212135314941, -0.03539071977138519), #position1
(-0.8244166374206543, -0.3322310745716095, -0.45821359753608704)), #normal1
((-0.06070474162697792, 0.12217497825622559, -0.038759298622608185),#position2
(-0.8547890186309814, -0.10589562356472015, -0.5080569386482239)), #normal2
((-0.07193990796804428, 0.12713682651519775, -0.005527373403310776),#position3
(-0.9910546541213989, -0.08559581637382507, -0.10239160805940628))] #normal3
winding = getwinding(face)
if __name__ == '__main__':
main()