noesis Gamebryo nif importer

Post questions about game models here, or help out others!
finale00
M-M-M-Monster veteran
M-M-M-Monster veteran
Posts: 2382
Joined: Sat Apr 09, 2011 1:22 am
Has thanked: 170 times
Been thanked: 300 times

noesis Gamebryo nif importer

Post by finale00 » Fri Jan 13, 2012 7:05 am

nif files always seemed amusing cause it's just a bunch of chunks ordered in some tree structure with nodes all over the place.

Documentation is available on niftools, so why not.

The file structure for later formats is convenient because you can quickly just skip everything that you don't need (which is usually the case cause we probably don't care about half the things in there), and if you suddenly needed something you can just throw in a parser function for it.

Earlier formats looks like it'll have to be parsed each struct at a time cause the node sizes in the header were introduced in 20.2.0.7...

Most of the games I have that use the nif format is archived away ATM so the only samples I have to work with are from West & East fantasy which is supported by nifskope (eg: can compare with what it loads)

Currently parses the geometry only. No textures (cause I have to do some image parsing...and I'm not really building a node tree so I haven't thought about how I the material will be assigned.

http://xtsukihime.webs.com/Noesis%20Plu ... ryo_nif.py

It seems like it would be easier to just write separate plugins for each version, rather than a single plugin for all versions, cause that would require some actual planning on how to write the code without making too much of a mess.

For anyone that's having a hard time following the docs, you basically need to parse each chunk one at a time.

First you start with the header: http://niftools.sourceforge.net/doc/nif/Header.html
If you're working with anything after 20.2.0.7, it is eas(ier) because you can skip stuff by seeking offsets (ie: minimal coding)

Otherwise, you're going to end up going back to the index, searching for the object you want, and then following it. You can load it in nifskope and follow it in block view to make it easier.

Darkfox
VVIP member
VVIP member
Posts: 688
Joined: Fri Jul 04, 2003 6:11 pm
Has thanked: 33 times
Been thanked: 16 times

Re: noesis Gamebryo nif importer

Post by Darkfox » Fri Jan 13, 2012 10:40 pm

Is a good cause, as the conversion process at current for nif models is a bit awkward to say the least. My primary interest is getting SMT Imagine demon models.

finale00
M-M-M-Monster veteran
M-M-M-Monster veteran
Posts: 2382
Joined: Sat Apr 09, 2011 1:22 am
Has thanked: 170 times
Been thanked: 300 times

Re: noesis Gamebryo nif importer

Post by finale00 » Fri Jan 13, 2012 10:54 pm

What kind of importers do they have right now anyways?
It looks like they've stopped supporting their 3dmax and blender plugins, and nifskope only exports geometry.

User avatar
howfie
double-veteran
double-veteran
Posts: 930
Joined: Fri Jul 08, 2011 12:06 pm
Location: Torrance, CA
Has thanked: 10 times
Been thanked: 270 times

Re: noesis Gamebryo nif importer

Post by howfie » Fri Jan 13, 2012 11:09 pm

Isn't nifskope that one project that's says it's 600,000 lines of code and worth several millions of dollars lol?

finale00
M-M-M-Monster veteran
M-M-M-Monster veteran
Posts: 2382
Joined: Sat Apr 09, 2011 1:22 am
Has thanked: 170 times
Been thanked: 300 times

Re: noesis Gamebryo nif importer

Post by finale00 » Sat Jan 14, 2012 1:44 am

lol ya they have a gif on their page. 4 devs, 322k+ lines, $4.6 million
It's definitely one of the more interesting game modding tools, kind of like the one that works with the unreal engine.

It looks like there's no documentation for some of the structs that appeared in games like RO2, catherine, or microvolts. Referred to the microvolts blender script for it, but the vertices are still a little odd.

The more I write, the more the file gets bloated.

Anyways I've done some minor edits to add the NiDataStream### objects, but am not getting the vertices right (particularly skinned models like monsters).

But some of the stuff comes out. This is one of the tutorial maps for RO2. Looks like static meshes are fine.

Image

pixellegolas
ultra-veteran
ultra-veteran
Posts: 423
Joined: Mon Aug 11, 2008 11:30 pm
Has thanked: 27 times
Been thanked: 15 times

Re: noesis Gamebryo nif importer

Post by pixellegolas » Sat Jan 14, 2012 2:25 pm

nice going, I am eagerly hoping that you will crack this. Alot of the new nif games are really nice looking but seems nif-forum only look at skyrim now :)

finale00
M-M-M-Monster veteran
M-M-M-Monster veteran
Posts: 2382
Joined: Sat Apr 09, 2011 1:22 am
Has thanked: 170 times
Been thanked: 300 times

Re: noesis Gamebryo nif importer

Post by finale00 » Sat Jan 21, 2012 8:26 pm

There isn't much to crack.

Most of the newer formats don't really add anything new (lots of NiDataBlock018, 118, 124, ...) but other than that, the docs on niftools should be sufficient.

I'm not putting too much time towards this atm cause of issues like papers and projects and stuff, but the script so far is somewhat flexible (just write a function for a chunk and add the case under parse_nodes(). It might get done eventually.

finale00
M-M-M-Monster veteran
M-M-M-Monster veteran
Posts: 2382
Joined: Sat Apr 09, 2011 1:22 am
Has thanked: 170 times
Been thanked: 300 times

Re: noesis Gamebryo nif importer

Post by finale00 » Wed Feb 08, 2012 5:53 am

Need a way to structure the files.

Basically, we know that gamebryo has a whole slew of formats from their first engine to what...30.whatever now?
Each version basically introduced new stuff, removed old stuff, or changed existing stuff.

The docs on niftools explains what was available in what version, since what version, and until what version.

Without using anything fancy like their XML file, how should I be organizing the plugins?
I don't want to cram everything into one big nif plugin, as trying to look for anything is pretty hard. Debugging would also be hard.

I want to use inheritance properly.
I can write separate plugins for each format, but since they share a lot of stuff, it'd probably easier to just push everything up to a parent class and only write whatever I need for the new format.

Put the stuff in a package? Just have a bunch of plugins lying around?
Looking for ideas.

finale00
M-M-M-Monster veteran
M-M-M-Monster veteran
Posts: 2382
Joined: Sat Apr 09, 2011 1:22 am
Has thanked: 170 times
Been thanked: 300 times

Re: noesis Gamebryo nif importer

Post by finale00 » Thu Feb 23, 2012 3:28 am

I've been looking at 20.6.0.0 using the catherine blender import for reference.

The vert data is stored in NiDataStream018, but it is rather weird.

It indicates the size of the vertex by "building" a bunch of pieces of together.

So for example, you might read in an integer count of 6, followed by 6 integers that provides the size of the vertex data but it's not obvious *what* it might be.

If you read "197687" that means there's another 12 bytes in your vertex struct.
But that could be vertex coords...or vertex normals. Or something else.

And it doesn't always appear in the same order either.

Sometimes it'll piece it together like

Code: Select all

4 bytes
4 bytes
12 bytes
12 bytes
4 bytes
8 bytes
Other times it'll piece it together like

Code: Select all

12 bytes
12 bytes
4 bytes
8 bytes
It's not like every vertex buffer has the same data in a single model either. You could have one mesh with 48 byte vertices and another mesh with 32 byte vertices.

Maybe there's another struct that determines the order that they appear in, but I don't know which one it might be cause there are so many chunks.

When I write out the chunks, it might narrow down where the data is stored though.

pixellegolas
ultra-veteran
ultra-veteran
Posts: 423
Joined: Mon Aug 11, 2008 11:30 pm
Has thanked: 27 times
Been thanked: 15 times

Re: noesis Gamebryo nif importer

Post by pixellegolas » Thu Feb 23, 2012 8:08 pm

Yeah I guess there is a reason why nifskope has taken some time to make the update :)

Ninja
veteran
Posts: 84
Joined: Sat Feb 26, 2011 3:44 am
Has thanked: 1 time
Been thanked: 20 times

Re: noesis Gamebryo nif importer

Post by Ninja » Fri Mar 02, 2012 7:00 am

Not sure if this is any help.
In 30.0.0.2 the vertex data is stored in NiDataStream 1 18 , along with the normals, UVs and such.
The NiDataStream 0 18 contains the triangle data.
All three of the NiDataStreams i've found, 0 18, 1 18 and 3 3, all follow the same basic format.

Code: Select all

C8160000 - 5832   <----- data length
00000000
01000000
00000000
E6010000 - 486     <----- number of elements in data
01000000
3704     - 1079   <---- ????
0300                 <---- ????
+ 5832 bytes data
+ terminator byte (usually 01)


The references to what each node is, is referenced in the NiMesh.

finale00
M-M-M-Monster veteran
M-M-M-Monster veteran
Posts: 2382
Joined: Sat Apr 09, 2011 1:22 am
Has thanked: 170 times
Been thanked: 300 times

Re: noesis Gamebryo nif importer

Post by finale00 » Fri Mar 02, 2012 7:47 am

Oh, nice find.
NiMesh 495541
NiTexturingProperty 495774
NiSourceTexture 495814
NiDataStream018 495850
NiDataStream118 497763
NiDataStream33 508464
NiSkinningMeshModifier 508509
NiMesh 509159
NiTexturingProperty 509414
NiSourceTexture 509463
NiSourceTexture 509499
NiMaterialProperty 509535
NiDataStream018 509603
NiDataStream118 522720
NiDataStream33 619117
NiSkinningMeshModifier 619442
Here's some sample output containing chunk names and offsets that I used.

Cutting out some of those niMesh chunks, I quickly noticed that near the end of the chunk, it gives you a count followed by some more odd integers.

The count is the same as the number of vertex data, so I can probably figure out what these odd integers mean and then just use a dictionary or something when I'm binding my buffers to bind the correct data...

This can't be a coincidence. Or maybe it is.
On the left is portion from NiMesh. The right is the portion from NiDataStream118

Image

I should be able to quickly narrow it down by finding the ones that have like 12 bytes per vertex.

Looking at some of the values even more, it looks like there is some sort of chunk numbering going on (the first integer of the chunk. Only applies to Nodes and Meshes it seems)

You can see some others integers increasing in count by 1 after every few bytes. Top left example shows 0x0268, 0x0269, 0x026A. I don't know what they actually mean though.

Maybe they keep track of the number of groups of data as well...
Last edited by finale00 on Fri Mar 02, 2012 9:31 am, edited 1 time in total.

finale00
M-M-M-Monster veteran
M-M-M-Monster veteran
Posts: 2382
Joined: Sat Apr 09, 2011 1:22 am
Has thanked: 170 times
Been thanked: 300 times

Re: noesis Gamebryo nif importer

Post by finale00 » Fri Mar 02, 2012 8:56 am

NiMesh is annoying.
ya...

Here's how I parsed it

Code: Select all

def parse_NiMesh_part(self, count):
		
		for i in range(count):
				chunkNum = self.inFile.readUInt()
				self.inFile.readByte()
				count2 = self.inFile.readUInt()
				self.inFile.read("%dH" %(count2 - 1))
				count3 = self.inFile.readUInt()
				for j in range(count3):
						self.inFile.read('2L')

def parse_NiMesh(self):
		
		nodeNum = self.inFile.readUInt()
		self.inFile.read("2L")
		self.inFile.readUInt()
		self.inFile.readUShort()
		self.inFile.read('12f')
		count = self.inFile.readUInt()
		self.inFile.read('%dL' %count)
		self.inFile.read('5l') #-1 0 -1 0 ?
		self.inFile.read('4f')
		count2 = self.inFile.readUInt()
		self.parse_NiMesh_part(count2)
And that's just getting the general structure.

In the helper function, the very last two lines is what is interesting.
One of the iterations will run into the case where the number of parts is equal to the number of vertex parts. Unfortunately there is no real way to determine when that would arise, or whether it actually has anything to do with it.

Ninja
veteran
Posts: 84
Joined: Sat Feb 26, 2011 3:44 am
Has thanked: 1 time
Been thanked: 20 times

Re: noesis Gamebryo nif importer

Post by Ninja » Fri Mar 02, 2012 7:30 pm

Yep, it's going to be annoying having that many formats, wouldn't have thought they would rewrite the structures too much, not unless they rewrite the graphics engine.
Anyway, here's a quick look at a NiMesh from 30.0.0.2
(from criticals red blood samples - chevalier_arm_00_m.nif)
(125 nodes, 120 names)

Code: Select all

61000000 - name reference 'Arm'
07000000 
65000000 66000000 67000000 68000000 69000000 6A000000 6B000000 

FFFFFFFF 
1600
00000000 00000000 00000000 0000803F
00000000 00000000 00000000 0000803F
00000000 00000000 00000000 0000803F
0000803F

03000000 
72000000 6D000000 6C000000
FFFFFFFF

01000000
6E000000

FFFFFFFF
00000000
00000000
00010000
75E98640 
E6C4AF3E 
C97E1541 
4E107E3E

09000000
 node id                                      name id
73000000 00 01000000 01000000    6F000000 00000000 - INDEX     -triangles
74000000 00 01000000 01000000    70000000 00000000 - TEXCOORD
75000000 00 01000000 01000000    71000000 00000000 - POSITION_BP    - x,y,z
76000000 00 01000000 01000000    72000000 00000000 - NORMAL_BP
77000000 00 01000000 01000000    73000000 00000000 - BLENDINDICES
78000000 00 01000000 01000000    74000000 00000000 
79000000 00 01000000 01000000    75000000 00000000
7A000000 00 01000000 01000000    76000000 00000000 
7B000000 00 01000000 01000000    77000000 00000000 

01000000 
7C000000            
The data and structure is unimportant at this point, it's the references that seem to matter more.
Think it's going to be important to list both the node list and name list complete with their index value to sort it out.
You can clearly see, the Nimesh is like a build list of parts, it even references the textures.
Not seen it reference the bones yet, think these are part of the NiNode system which starts with the scene root.
So NiMesh must be part of a NiNode ?? and the Nodes are part of a scene tree??
Got the vertices(POSITION_BP) into noesis for models with 1 object, models with more than one object just load the 1st object, which is why i'm looking at NiMesh for a solution.

Any chance of uploading or pointing to the file your working on for comparison?

finale00
M-M-M-Monster veteran
M-M-M-Monster veteran
Posts: 2382
Joined: Sat Apr 09, 2011 1:22 am
Has thanked: 170 times
Been thanked: 300 times

Re: noesis Gamebryo nif importer

Post by finale00 » Fri Mar 02, 2012 7:37 pm

I realize that while there are many formats, I haven't set up the file to be as flexible as it should for the future.
So now it just hardcodes a bunch of versions that I looked at. Maybe in the future I will just create one large file and have other versions inherit from it and override some methods if necessary.

http://db.tt/XkAI9sIm

It uses the Sanae3D package cause I do things like outputting values into a text file and plotting points.
While I can just copy all of the methods over, I just do it for convenience and quick testing.

But hmm node references...

Maybe 30.x is also different from 20.6.x

Post Reply