Page 1 of 3
noesis Gamebryo nif importer
Posted: Fri Jan 13, 2012 7:05 am
by finale00
nif files always seemed amusing cause it's just a bunch of chunks ordered in some tree structure with nodes all over the place.
Documentation is available on niftools, so why not.
The file structure for later formats is convenient because you can quickly just skip everything that you don't need (which is usually the case cause we probably don't care about half the things in there), and if you suddenly needed something you can just throw in a parser function for it.
Earlier formats looks like it'll have to be parsed each struct at a time cause the node sizes in the header were introduced in 20.2.0.7...
Most of the games I have that use the nif format is archived away ATM so the only samples I have to work with are from
West & East fantasy which is supported by nifskope (eg: can compare with what it loads)
Currently parses the geometry only. No textures (cause I have to do some image parsing...and I'm not really building a node tree so I haven't thought about how I the material will be assigned.
http://xtsukihime.webs.com/Noesis%20Plu ... ryo_nif.py
It seems like it would be easier to just write separate plugins for each version, rather than a single plugin for all versions, cause that would require some actual planning on how to write the code without making too much of a mess.
For anyone that's having a hard time following the docs, you basically need to parse each chunk one at a time.
First you start with the header:
http://niftools.sourceforge.net/doc/nif/Header.html
If you're working with anything after 20.2.0.7, it is eas(ier) because you can skip stuff by seeking offsets (ie: minimal coding)
Otherwise, you're going to end up going back to the
index, searching for the object you want, and then following it. You can load it in nifskope and follow it in block view to make it easier.
Re: noesis Gamebryo nif importer
Posted: Fri Jan 13, 2012 10:40 pm
by Darkfox
Is a good cause, as the conversion process at current for nif models is a bit awkward to say the least. My primary interest is getting SMT Imagine demon models.
Re: noesis Gamebryo nif importer
Posted: Fri Jan 13, 2012 10:54 pm
by finale00
What kind of importers do they have right now anyways?
It looks like they've stopped supporting their 3dmax and blender plugins, and nifskope only exports geometry.
Re: noesis Gamebryo nif importer
Posted: Fri Jan 13, 2012 11:09 pm
by howfie
Isn't nifskope that one project that's says it's 600,000 lines of code and worth several millions of dollars lol?
Re: noesis Gamebryo nif importer
Posted: Sat Jan 14, 2012 1:44 am
by finale00
lol ya they have a gif on their page. 4 devs, 322k+ lines, $4.6 million
It's definitely one of the more interesting game modding tools, kind of like the one that works with the unreal engine.
It looks like there's no documentation for some of the structs that appeared in games like RO2, catherine, or microvolts. Referred to the microvolts blender script for it, but the vertices are still a little odd.
The more I write, the more the file gets bloated.
Anyways I've done some minor edits to add the NiDataStream### objects, but am not getting the vertices right (particularly skinned models like monsters).
But some of the stuff comes out. This is one of the tutorial maps for RO2. Looks like static meshes are fine.
Re: noesis Gamebryo nif importer
Posted: Sat Jan 14, 2012 2:25 pm
by pixellegolas
nice going, I am eagerly hoping that you will crack this. Alot of the new nif games are really nice looking but seems nif-forum only look at skyrim now
Re: noesis Gamebryo nif importer
Posted: Sat Jan 21, 2012 8:26 pm
by finale00
There isn't much to crack.
Most of the newer formats don't really add anything new (lots of NiDataBlock018, 118, 124, ...) but other than that, the docs on niftools should be sufficient.
I'm not putting too much time towards this atm cause of issues like papers and projects and stuff, but the script so far is somewhat flexible (just write a function for a chunk and add the case under parse_nodes(). It might get done eventually.
Re: noesis Gamebryo nif importer
Posted: Wed Feb 08, 2012 5:53 am
by finale00
Need a way to structure the files.
Basically, we know that gamebryo has a whole slew of formats from their first engine to what...30.whatever now?
Each version basically introduced new stuff, removed old stuff, or changed existing stuff.
The docs on niftools explains what was available in what version, since what version, and until what version.
Without using anything fancy like their XML file, how should I be organizing the plugins?
I don't want to cram everything into one big nif plugin, as trying to look for anything is pretty hard. Debugging would also be hard.
I want to use inheritance properly.
I can write separate plugins for each format, but since they share a lot of stuff, it'd probably easier to just push everything up to a parent class and only write whatever I need for the new format.
Put the stuff in a package? Just have a bunch of plugins lying around?
Looking for ideas.
Re: noesis Gamebryo nif importer
Posted: Thu Feb 23, 2012 3:28 am
by finale00
I've been looking at 20.6.0.0 using the catherine blender import for reference.
The vert data is stored in NiDataStream018, but it is rather weird.
It indicates the size of the vertex by "building" a bunch of pieces of together.
So for example, you might read in an integer count of 6, followed by 6 integers that provides the size of the vertex data but it's not obvious *what* it might be.
If you read "197687" that means there's another 12 bytes in your vertex struct.
But that could be vertex coords...or vertex normals. Or something else.
And it doesn't always appear in the same order either.
Sometimes it'll piece it together like
Code: Select all
4 bytes
4 bytes
12 bytes
12 bytes
4 bytes
8 bytes
Other times it'll piece it together like
It's not like every vertex buffer has the same data in a single model either. You could have one mesh with 48 byte vertices and another mesh with 32 byte vertices.
Maybe there's another struct that determines the order that they appear in, but I don't know which one it might be cause there are so many chunks.
When I write out the chunks, it might narrow down where the data is stored though.
Re: noesis Gamebryo nif importer
Posted: Thu Feb 23, 2012 8:08 pm
by pixellegolas
Yeah I guess there is a reason why nifskope has taken some time to make the update
Re: noesis Gamebryo nif importer
Posted: Fri Mar 02, 2012 7:00 am
by Ninja
Not sure if this is any help.
In 30.0.0.2 the vertex data is stored in NiDataStream 1 18 , along with the normals, UVs and such.
The NiDataStream 0 18 contains the triangle data.
All three of the NiDataStreams i've found, 0 18, 1 18 and 3 3, all follow the same basic format.
Code: Select all
C8160000 - 5832 <----- data length
00000000
01000000
00000000
E6010000 - 486 <----- number of elements in data
01000000
3704 - 1079 <---- ????
0300 <---- ????
+ 5832 bytes data
+ terminator byte (usually 01)
The references to what each node is, is referenced in the NiMesh.
Re: noesis Gamebryo nif importer
Posted: Fri Mar 02, 2012 7:47 am
by finale00
Oh, nice find.
NiMesh 495541
NiTexturingProperty 495774
NiSourceTexture 495814
NiDataStream018 495850
NiDataStream118 497763
NiDataStream33 508464
NiSkinningMeshModifier 508509
NiMesh 509159
NiTexturingProperty 509414
NiSourceTexture 509463
NiSourceTexture 509499
NiMaterialProperty 509535
NiDataStream018 509603
NiDataStream118 522720
NiDataStream33 619117
NiSkinningMeshModifier 619442
Here's some sample output containing chunk names and offsets that I used.
Cutting out some of those niMesh chunks, I quickly noticed that near the end of the chunk, it gives you a count followed by some more odd integers.
The count is the same as the number of vertex data, so I can probably figure out what these odd integers mean and then just use a dictionary or something when I'm binding my buffers to bind the correct data...
This can't be a coincidence. Or maybe it is.
On the left is portion from NiMesh. The right is the portion from NiDataStream118
I should be able to quickly narrow it down by finding the ones that have like 12 bytes per vertex.
Looking at some of the values even more, it looks like there is some sort of chunk numbering going on (the first integer of the chunk. Only applies to Nodes and Meshes it seems)
You can see some others integers increasing in count by 1 after every few bytes. Top left example shows 0x0268, 0x0269, 0x026A. I don't know what they actually mean though.
Maybe they keep track of the number of groups of data as well...
Re: noesis Gamebryo nif importer
Posted: Fri Mar 02, 2012 8:56 am
by finale00
NiMesh is annoying.
ya...
Here's how I parsed it
Code: Select all
def parse_NiMesh_part(self, count):
for i in range(count):
chunkNum = self.inFile.readUInt()
self.inFile.readByte()
count2 = self.inFile.readUInt()
self.inFile.read("%dH" %(count2 - 1))
count3 = self.inFile.readUInt()
for j in range(count3):
self.inFile.read('2L')
def parse_NiMesh(self):
nodeNum = self.inFile.readUInt()
self.inFile.read("2L")
self.inFile.readUInt()
self.inFile.readUShort()
self.inFile.read('12f')
count = self.inFile.readUInt()
self.inFile.read('%dL' %count)
self.inFile.read('5l') #-1 0 -1 0 ?
self.inFile.read('4f')
count2 = self.inFile.readUInt()
self.parse_NiMesh_part(count2)
And that's just getting the general structure.
In the helper function, the very last two lines is what is interesting.
One of the iterations will run into the case where the number of parts is equal to the number of vertex parts. Unfortunately there is no real way to determine when that would arise, or whether it actually has anything to do with it.
Re: noesis Gamebryo nif importer
Posted: Fri Mar 02, 2012 7:30 pm
by Ninja
Yep, it's going to be annoying having that many formats, wouldn't have thought they would rewrite the structures too much, not unless they rewrite the graphics engine.
Anyway, here's a quick look at a NiMesh from 30.0.0.2
(from criticals red blood samples - chevalier_arm_00_m.nif)
(125 nodes, 120 names)
Code: Select all
61000000 - name reference 'Arm'
07000000
65000000 66000000 67000000 68000000 69000000 6A000000 6B000000
FFFFFFFF
1600
00000000 00000000 00000000 0000803F
00000000 00000000 00000000 0000803F
00000000 00000000 00000000 0000803F
0000803F
03000000
72000000 6D000000 6C000000
FFFFFFFF
01000000
6E000000
FFFFFFFF
00000000
00000000
00010000
75E98640
E6C4AF3E
C97E1541
4E107E3E
09000000
node id name id
73000000 00 01000000 01000000 6F000000 00000000 - INDEX -triangles
74000000 00 01000000 01000000 70000000 00000000 - TEXCOORD
75000000 00 01000000 01000000 71000000 00000000 - POSITION_BP - x,y,z
76000000 00 01000000 01000000 72000000 00000000 - NORMAL_BP
77000000 00 01000000 01000000 73000000 00000000 - BLENDINDICES
78000000 00 01000000 01000000 74000000 00000000
79000000 00 01000000 01000000 75000000 00000000
7A000000 00 01000000 01000000 76000000 00000000
7B000000 00 01000000 01000000 77000000 00000000
01000000
7C000000
The data and structure is unimportant at this point, it's the references that seem to matter more.
Think it's going to be important to list both the node list and name list complete with their index value to sort it out.
You can clearly see, the Nimesh is like a build list of parts, it even references the textures.
Not seen it reference the bones yet, think these are part of the NiNode system which starts with the scene root.
So NiMesh must be part of a NiNode ?? and the Nodes are part of a scene tree??
Got the vertices(POSITION_BP) into noesis for models with 1 object, models with more than one object just load the 1st object, which is why i'm looking at NiMesh for a solution.
Any chance of uploading or pointing to the file your working on for comparison?
Re: noesis Gamebryo nif importer
Posted: Fri Mar 02, 2012 7:37 pm
by finale00
I realize that while there are many formats, I haven't set up the file to be as flexible as it should for the future.
So now it just hardcodes a bunch of versions that I looked at. Maybe in the future I will just create one large file and have other versions inherit from it and override some methods if necessary.
http://db.tt/XkAI9sIm
It uses the Sanae3D package cause I do things like outputting values into a text file and plotting points.
While I can just copy all of the methods over, I just do it for convenience and quick testing.
But hmm node references...
Maybe 30.x is also different from 20.6.x