[Help] Weights issue in Noesis script

[srbvn]

Hi everyone,
After months giving up, I decide to fight again with the boneWeight and animation in Noesis.
Mesh and bone are loaded correctly (or as I think so) but bone weight isn't.
There are 3 issues:

• I do not know how to use functions for bone weight in Noesis in my case.
• Why is there a 4x4 Mat for parent bone under each 4x4 Mat for each bone, or maybe I missunderstand about its role. See below for reference.
• How to test for the weight and bone when I have no animation file, is it anyway to create a simple animation for testing purpose?

Hope someone could help or give hints. Many thanks.
Here is the file structure:

...
edit 2. the ksword3d models look static - very simple, vertices 3 floats, normals 3 floats, UV 2 floats + 1 long, triangle strips?

I add structure of bone as below:

Code: Select all

bone offset 4 bytes - byte 188th from the beginning
After jumping to bone offset
number of bones 4 bytes
char bone-name 30 bytes
char parent-bone-name 30 bytes
int number of child bones 4 bytes
char child-bone-name[] 30 bytes
float transform-matrix[4][4] 64 bytes //need to be inversed
float parent bone transform-matrix[4][4] 64 bytes //need to be inversed
int number of vertex connected to this bone 4 bytes 
int vertex id[] 4 bytes 
float weight[] 4 bytes // weighting information corresponding to vertex id

Here is the code:
https://pastebin.com/vTbzXe8A
And sample file:
https://drive.google.com/file/d/14G-9xR ... nnPoB/view

[jayn23]

I took a very quick look at your script and your weight list and vertex number should be equal, but vert number is 569 while weight list length is 1.
its probably some small mistake in the code, ill try and take a better look later.

Another problem i think you have is you are phrasing mesh then skeleton but weight are in the skeleton section, so you basically read the weights once the mesh is already defied, i dont think it will work like that.

try using NoeMesh() and NoeModel() - that way you can insert all the data at the end

edit:
just from a few prints, one mistake is what you are appending to bidx which should hold bone ID and you are appending vert ID.

[srbvn]

I took a very quick look at your script and your weight list and vertex number should be equal, but vert number is 569 while weight list length is 1.
its probably some small mistake in the code, ill try and take a better look later.

Another problem i think you have is you are phrasing mesh then skeleton but weight are in the skeleton section, so you basically read the weights once the mesh is already defied, i dont think it will work like that.

try using NoeMesh() and NoeModel() - that way you can insert all the data at the end

Hi jayn23,
You are right, I think I should try to re-write the script with NoeMesh() and NoeModel() - not sure how to write, but will update here soon .

edit:
just from a few prints, one mistake is what you are appending to bidx which should hold bone ID and you are appending vert ID.

Currently, I am trying to reshape the weighting data to create Buffers for rpgBindBoneIndexBuffer and rpgBindBoneWeightBuffer., since the file structure looks like figures no.1 while the functions require the figure no. 2. Honestly, I am in trouble with this step since I do not know much about Python. Do you have any hints for this, or we should have another method that get rid of rpgBindBoneIndexBuffer and rpgBindBoneWeightBuffer. Thanks.


Figure no. 1


Figure no. 2

[jayn23]

Currently, I am trying to reshape the weighting data to create Buffers for rpgBindBoneIndexBuffer and rpgBindBoneWeightBuffer., since the file structure looks like figures no.1 while the functions require the figure no. 2. Honestly, I am in trouble with this step since I do not know much about Python. Do you have any hints for this, or we should have another method that get rid of rpgBindBoneIndexBuffer and rpgBindBoneWeightBuffer. Thanks.

here is a working code with weights applied correctly - tested it in 3ds max, and i am using NoeMesh() and NoeModel() so you can learn how to use them

Code: Select all

from os.path import *
from math import *
from inc_noesis import *

class FileData:

    def __init__(self):
        self.BoneID = []
        self.Weights = []
        
def registerNoesisTypes():
    handle = noesis.register("unk game", ".ksw")    
    noesis.setHandlerTypeCheck(handle, noepyCheckType)
    noesis.setHandlerLoadModel(handle, noepyLoadModel)
    #opens debug consle
    #noesis.logPopup()
    return 1


def noepyCheckType(data):
    '''Verify that the format is supported by this plugin. Default yes'''
    #I preform my check while reading mesh data
    return 1


def noepyLoadModel(data, mdlList): 
     meshName = "test"
     Positions = []
     Normals = []
     UV = []
     PolygonIndex = []
     meshes = []
     VertData = []
     
     bs = NoeBitStream(data, NOE_LITTLEENDIAN)
     bs.seek(140,0)
     numMesh = bs.readInt()
     print("numMesh",numMesh)
     numVerts = bs.readInt()
     print("numVerts",numVerts)
     numFace = bs.readInt()
     print("numFace",numFace)
     unk1 = bs.readInt()
     vertOff = bs.readInt()
     normOff = bs.readInt()
     unk2 = bs.readInt()
     UVOff = bs.readInt()
     unk3 = bs.readInt()
     unk4 = bs.readInt()
     faceoff = bs.readInt()
     unk5 = bs.readInt()
     boneOff = bs.readInt()
     print("boneOff",boneOff)
     
     bs.seek(vertOff,0)
     for i in range(numVerts):
        vx = bs.readFloat()
        vy = bs.readFloat()
        vz = bs.readFloat()
        VertData.append(FileData())
        Positions.append(NoeVec3([vx,vy,vz]))
            
     bs.seek(normOff,0)
     for i in range(numVerts):
        nx = bs.readFloat()
        ny = bs.readFloat()
        nz = bs.readFloat()
        #print(str(nx ) + str(ny ) + str(nz))
        Normals.append(NoeVec3([nx,ny,nz]))
 
     bs.seek(UVOff,0)
     for i in range(numVerts):     
        tu = bs.readFloat()
        tv = bs.readFloat()
        UV.append(NoeVec3([tu,tv,0.0])) 
            
     bs.seek(faceoff,0)
     for j in range(numFace*3):
        PolygonIndex.append(bs.readInt())
            
     boneList = []
     
     bs.seek(boneOff,0)
     boneCount = bs.readUInt()
      
    # read bones loop
     for i in range(boneCount):
        # read bone name
        boneName = bs.readBytes(30).decode('utf-8', 'ignore').replace('\x00','').replace('\xCD','')
        # read parent name of the current bone
        parentBoneName = bs.readBytes(30).decode('utf-8', 'ignore').replace('\x00','').replace('\xCD','')
        # read the number of child bone
        numChildren = bs.readUInt()
        # read child bone name
        for j in range(numChildren):
            childBoneName = bs.readBytes(30).decode('utf-8', 'ignore').replace('\x00','').replace('\xCD','')
        # read 4x4 bone matrix
        boneMat = NoeMat44.fromBytes( bs.readBytes(0x40), NOE_LITTLEENDIAN )
        # convert and inverse to 4x3 matrix
        boneMat43 = boneMat.toMat43().inverse()
        # read 4x4 parent bone matrix
        parentMat = NoeMat44.fromBytes( bs.readBytes(0x40), NOE_LITTLEENDIAN )
        # convert and inverse to 4x3 matrix
        parentMat43 = parentMat.toMat43().inverse()
        # read the number of vertices deformed by this bone
        numVertices = bs.readUInt()
        if numVertices > 0:
            weightList = []
            VertexList = []
            weights = []
            # read vertex IDs
            for j in range(numVertices):
                vertexID = bs.readUInt()
                VertData[vertexID].BoneID.append(i)
                VertexList.append(vertexID)
            # read weights correspondent to vertex IDs
            for j in range(numVertices):
                weight = bs.readFloat()
                VertData[VertexList[j]].Weights.append(weight)
                weights.append(weight)
 
        # set bones for Noesis to display      
        boneList.append(NoeBone(i, boneName, boneMat43, parentBoneName, parentIndex = -1)) 
       
     for k in range(numVerts):
        weightList.append(NoeVertWeight(VertData[k].BoneID, VertData[k].Weights))
        
     a = len(weightList)
     print("len weight =", a)  
     mesh = NoeMesh(PolygonIndex, Positions, meshName)
     mesh.setWeights(weightList)
     mesh.setNormals(Normals)
     mesh.setUVs(UV)
     meshes.append(mesh)
     mdl = NoeModel(meshes)   
     mdl.setBones(boneList) 
     mdlList.append(mdl)
     
     return 1
            
        

[srbvn]

Yeah, it works correctly, thanks so much jayn23.
I think it's time for researching animation. File structure is below, I think my code is ok except matrices part or may be the parameters for NoeAnim. I also get a warning every time load file:

WARNING: Number of frame matrices should be at least numBones*numFrames

Attached is example of animation files which should work for the previous model.
Hope you could take a look on this. Thanks again.
https://pastebin.com/4ga09TG8

Code: Select all

MINA 4 bytes string // signature
unk1 4 bytes // 00 00 00 00
numAnim 4 bytes //Uint
frameRate 4 bytes //Uint
animName 30 bytes //string
numBones 4 bytes //Uint
numFrames 4 bytes //Uint
unk2 4 bytes // float 33 or 60
boneNames [] 30 *  numBones bytes // string
boneMatrices [] 64 * numBones * numFrames bytes // 4x4 matrices, may be not need to invert
4 bytes // end of file

[jayn23]

Hi,

I believe i found the problem:
in the code below you are building a new bonelist that is not equal to your skeleton bonelist - i think this is incorrect, the animation looks at the entire skeleton, in this case it has 62 bones while animation and your new list "animBones"is using only 51 (the one that i looked at), this leads WARNING: Number of frame matrices should be at least numBones*numFrames

what you need is use the skeleton bonelist and for animFrameMats is to do is per frame and bone name apply the new matrices to bones that are used, for the others that are static insert the matrix from you skeleton bone list (per frame)

i hope this was clear enough if not ill try to explain better.

Code: Select all

for i in range(numBones):
        animBoneName = aniF.readBytes(30).decode('ascii', 'ignore').replace('\x00','').replace('\xCD','')
        for j in range(len(boneList)):
            if animBoneName == boneList[j].name:
                animBones.append(boneList[j])

[srbvn]

what you need is use the skeleton bonelist and for animFrameMats is to do is per frame and bone name apply the new matrices to bones that are used, for the others that are static insert the matrix from you skeleton bone list (per frame)

Hello,
I could understand your words but I am in trouble with the algorithm. In my mind, the code should be something like this:

Code: Select all

bones = list of NoeBone of the model # or you call skeleton bonelist
animBones = list of bone names in the .MIN file
animMat =  list of matrices in the .MIN file
trueMat = [NoeMat43()] * numFrames * len(bones
for i in range(numFrames):
    for j in range(len(bones)):
        if bones[j].name is in [all elements of animBones[].name]:
             trueMat = bones[j].getMatrix().__mul__(current matrix of current animBones at current frame)
        else:
             trueMat = bones[j].getMatrix()
anim = NoeAnim(animName, bones , numFrames, trueMat , frameRate)

Is my thought correct? Hope you could help me to transfer to python script if the algorithm is ok. I have been trying for 2 hours but it seems not work as I expected .

updated:
Here is my current code, but it still not work, with an error "IndexError: list index out of range" at line "if boneList[j].name in [animBones[k].name for k in range(numBones)]:":
A full version is at https://pastebin.com/cvSzKLRP

Code: Select all

     # load animation -------------------------------------------------------------------
     anims = rapi.loadPairedFile("JX2 ani", ".min")
     aniF = NoeBitStream(anims)
     signature = aniF.readBytes(8)
     numAnims = aniF.readUInt()
     print('numAnims: ', numAnims)
     frameRate = aniF.readUInt()
     print('frameRate: ', frameRate)
     #animName may not be read correctly since it is Chinese string when using ascii or utf-8, 
     animName = aniF.readBytes(30).decode('ascii', 'ignore').replace('\x00','').replace('\xCD','')
     print('animName: ', animName)
     numBones = aniF.readUInt()
     print('numBones: ', numBones)
     numFrames = aniF.readUInt()
     print('numFrames: ', numFrames)
     unk6 = aniF.readBytes(4)
     animBones = []
     idBones = [-1] * len(boneList)
     for i in range(numBones):
        animBoneName = aniF.readBytes(30).decode('ascii', 'ignore').replace('\x00','').replace('\xCD','')
        for j in range(len(boneList)):
            if animBoneName == boneList[j].name:
                animBones.append(boneList[j])
                idBones[j] = i
     animFrameMats = []
     numMats = numBones * numFrames
     print('numMats: ', numMats)
     for i in range(numMats):
        frameMat44 = NoeMat44.fromBytes(aniF.readBytes(64))
        frameMat43 = frameMat44.toMat43()
        animFrameMats.append(frameMat43)
     trueFrameMats = [NoeMat43()] * numFrames * len(boneList)
     # for i in range(numFrames):
        # for j in range(len(boneList)):
            # trueFrameMats.append(boneList[j].getMatrix())
     for i in range(numFrames):
        for j in range(len(boneList)):
            curId = (i + 1) * (j + 1) - 1
            if boneList[j].name in [animBones[k].name for k in range(numBones)]:
                if idBones[j] != -1:
                    trueFrameMats[curId] = boneList[j].getMatrix().__mul__(animFrameMats[(i + 1) * (idBones[j] + 1) - 1])
                else:
                    print('Error: idBones[', j, '] is = -1')
            else:
                trueFrameMats[curId] = boneList[j].getMatrix()
     #print(len(trueFrameMats))
     anim = NoeAnim(animName, boneList, numFrames, animFrameMats, frameRate)
     #print(anim.__repr__())
     animList = []
     animList.append(anim)

[srbvn]

Hi,
Here is my current script, although it still not work correctly, I think there are some errors in the appending framematrix but have no clue how to fix them. Keep trying now .
Updated part is from line 129.
https://pastebin.com/cP9G8LBC

Also, some animations may have more bones than in the model's skeleton, or vise versa. It may not affect the result, as if a bone in MIN file do not exist in skeleton, we just ignore it. If a bone in the skeleton does not exist in the MIN file, we use its original matrix for each frame. It's my thought, but the script somehow still not work

Hope jayn23 or someone could take a glance.
Many thanks.

[jayn23]

Sorry haven't been near my computer for the past day,

ill try and write the code see if it works, i did something slimier in my total war warhammer script.

Also, some animations may have more bones than in the model's skeleton, or vise versa. It may not affect the result, as if a bone in MIN file do not exist in skeleton, we just ignore it. If a bone in the skeleton does not exist in the MIN file, we use its original matrix for each frame. It's my thought, but the script somehow still not work

this sounds weird, i have never seen an animation file that has more bones than the skeleton

[srbvn]

this sounds weird, i have never seen an animation file that has more bones than the skeleton

Yes, and I think it's because there are 4 class of characters in the game that models are extracted, and some animations are only used for a specific class (which has more bones than others). I think we just ignore the animBone that not in the skeleton and see if the script work. Thanks in advance and nice weekend.

[jayn23]

So the animation i am getting is horrible so its incorrect.

edit:
didn't think about it until now, but animation matrix in noesis is assumed to be in local space while the matrix in the bonelist and possibly the animation as well are in world space. so they need to be converted before we apply them, that is probably at least 1 part of the problem.

ill post my code, but to be clear the animation is a mess
(no conversion world to local done)

Code: Select all

# load animation -------------------------------------------------------------------
     anims = rapi.loadPairedFile("JX2 ani", ".min")
     aniF = NoeBitStream(anims)
     signature = aniF.readBytes(8)
     numAnims = aniF.readUInt()
     print('numAnims: ', numAnims)
     frameRate = aniF.readUInt()
     print('frameRate: ', frameRate)
     #animName may not be read correctly since it is Chinese string when using ascii or utf-8,
     animName = aniF.readBytes(30).decode('ascii', 'ignore').replace('\x00','').replace('\xCD','')
     print('animName: ', animName)
     numBones = aniF.readUInt()
     print('numBones: ', numBones)
     numFrames = aniF.readUInt()
     print('numFrames: ', numFrames)
     unk6 = aniF.readBytes(4)
     
     animBones = []
     animFrameMats = []
     
     MatIndex = []  
     MatIndex2 = []
     
     for i in range(numBones):    
        animBoneName = aniF.readBytes(30).decode('ascii', 'ignore').replace('\x00','').replace('\xCD','')
        #print(animBoneName)
        for j in range(len(boneList)):
            if animBoneName == boneList[j].name:
                MatIndex.append(j)
                MatIndex2.append(i)
                #print(boneList[j].name)
                #print("boneList[j].index",boneList[j].index)
                break

     for i in range(numFrames):
     
        TempMat = []
        
        for j in range(len(boneList)):
            TempMat.append(boneList[j].getMatrix()) 
            
        for k in range(len(MatIndex)):
            frameMat44 = NoeMat44.fromBytes(aniF.readBytes(64))
            frameMat43 = frameMat44.toMat43() 
            frameMat43 = frameMat43.inverse()
            if k in MatIndex2:
                print(k)
                TempMat[MatIndex[k]] = frameMat43
            
        animFrameMats += TempMat

     anim = NoeAnim(animName, boneList, numFrames, animFrameMats, frameRate)
     #print(anim.__repr__())
     animList = []
     animList.append(anim)

[srbvn]

didn't think about it until now, but animation matrix in noesis is assumed to be in local space while the matrix in the bonelist and possibly the animation as well are in world space. so they need to be converted before we apply them, that is probably at least 1 part of the problem.

In the model file (.ksw), there is another matrix below every bone matrix, do you think it will be useful in our case? We are currently not using it but somehow the skeleton still looks ok

Code: Select all

        # read 4x4 bone matrix
        boneMat = NoeMat44.fromBytes( bs.readBytes(0x40), NOE_LITTLEENDIAN )
        # convert and inverse to 4x3 matrix
        boneMat43 = boneMat.toMat43().inverse()
        # read 4x4 parent bone matrix
        parentMat = NoeMat44.fromBytes( bs.readBytes(0x40), NOE_LITTLEENDIAN )
        # convert and inverse to 4x3 matrix
        parentMat43 = parentMat.toMat43().inverse()

I am not clear about converting from world space to local space. Do you mean we need to convert both matrices in boneList and animation, or only the animation?
If trying to reconstruct matrices of boneList: boneMat43 = boneMat43.__mul__(parentMat43.inverse()) or boneMat43 = boneMat43.__mul__(parentMat43), the skeleton is so weird as images below.


Hope to hear interesting hints from you.

[jayn23]

I am not clear about converting from world space to local space. Do you mean we need to convert both matrices in boneList and animation, or only the animation?
If trying to reconstruct matrices of boneList: boneMat43 = boneMat43.__mul__(parentMat43.inverse()) or boneMat43 = boneMat43.__mul__(parentMat43), the skeleton is so weird as images below.

you need to convert only the animation matrix to local space, the bonelist is expected to be in world space,
here is a link to a thraed where they convert from world to local space, MrAdults is the creator of noesis so he knows what he is talking about
viewtopic.php?f=16&t=15065

there is another matrix below every bone matrix, do you think it will be useful in our case? We are currently not using it but somehow the skeleton still looks ok

i have seen some games that include both world and local space matrix, you say both matrix work without doing any conversion on either of them? hmmm

[srbvn]

you need to convert only the animation matrix to local space, the bonelist is expected to be in world space,
here is a link to a thraed where they convert from world to local space, MrAdults is the creator of noesis so he knows what he is talking about
viewtopic.php?f=16&t=15065

Thanks, I will spend time for this thread.

i have seen some games that include both world and local space matrix, you say both matrix work without doing any conversion on either of them? hmmm

No, as you see in the script, I mean we append only the boneMat43 to the boneList while there is nothing to do with parentMat43. The latter is named "parentMat43" just because I guess so (and not sure ). I wonder how could we check if boneMat43 is the worldspace matrix and parentMat43 is the localspace matrix (and not the matrix of parent bone like I guess)? Anyway, I could not thank enough for your help, and I will update here soon about my work.

Code: Select all

	# read 4x4 bone matrix
        boneMat = NoeMat44.fromBytes( bs.readBytes(0x40), NOE_LITTLEENDIAN )
        # convert and inverse to 4x3 matrix
        boneMat43 = boneMat.toMat43().inverse()
        # read 4x4 parent bone matrix
        parentMat = NoeMat44.fromBytes( bs.readBytes(0x40), NOE_LITTLEENDIAN )
        # convert and inverse to 4x3 matrix
        parentMat43 = parentMat.toMat43().inverse()

[srbvn]

you need to convert only the animation matrix to local space, the bonelist is expected to be in world space,
here is a link to a thraed where they convert from world to local space, MrAdults is the creator of noesis so he knows what he is talking about
viewtopic.php?f=16&t=15065

I try to mimic the script in the thread provided. Unfortunately, the result is still a mess . Could you help to take a glance if I need to modify something? Here is the full version: https://pastebin.com/eWxmzYZj

Code: Select all

     for i in range(numFrames):
        TempMat = []
            
        for j in range(len(boneList)):
            TempMat.append(boneList[j].getMatrix()) 
            
        for k in range(len(MatIndex)):
            frameMat44 = NoeMat44.fromBytes(aniF.readBytes(64))
            frameMat43 = frameMat44.toMat43() 
            frameMat43 = frameMat43.inverse()
            if k in MatIndex2:
                # print(k)
                TempMat[MatIndex[k]] =  TempMat[MatIndex[k]] * frameMat43
                # check if current bone have parent
                if boneList[MatIndex[k]].parentName != 'Scene Root':
                    # get index of the parentBone in boneList 
                    x = [boneList[y].name for y in range(boneCount)].index(boneList[MatIndex[k]].parentName)
                    TempMat[MatIndex[k]] = TempMat[MatIndex[k]] * TempMat[x].inverse()
            
        animFrameMats += TempMat

     anim = NoeAnim(animName, boneList, numFrames, animFrameMats, frameRate)
     animList = []
     animList.append(anim)