Final Fantasy XI .DAT (PC)

[paulgswanson]

Thanks for the link. there was quite a bit of helpful info in there, Google translate makes my head hurt though. I'm assuming those scripts only worked on character models with skeletons? The translation chrome gave me was....difficult.

[TehDave]

Thanks for the link. there was quite a bit of helpful info in there, Google translate makes my head hurt though. I'm assuming those scripts only worked on character models with skeletons? The translation chrome gave me was....difficult.

Lucky you, I can't even get the page to load.

Any chance you have the scripts and would be willing to upload them?

[Humm08]

Was the source for Altana View ever posted anywhere?

[alloy]

Back like in 2007 ffxitool had its source code up somewhere on the japanese net.
Maybe it still is if you know japanese google-fu

http://cosmosa.jp/ffxitool/

Just a bunch on tools that can play music load models, levels and all kinds of data for ffxi.

The guy who wrote Altana had access to the ffxitool source. His prog is just a lot more polished.

But this is were it all ffxi data ripping started.

Also the ffxi tool page i posted up there has some links to what seems to be info on data structure and whatnot.

And you can find a link to dl the actual programs on the page i linked up there

http://cosmosa.jp/ffxitool/kaimemo.html
http://cosmosa.jp/ffxitool/generator.html
http://cosmosa.jp/ffxitool/note.html

Hope anyone here can make any sense or if its even useful to you guys.

And also theres a page dedicated to what i guess is his tool version for ff14

http://cosmosa.jp/ffxivtool/

I have no idea what it does because i don't have that game and i literally just found out about it lol.

[MrAdults]

I ended up re-reversing this damn thing. It'll be in the Noesis release I'm planning to put up in a few minutes.




Animation and maps and everything are supported. Still haven't totally figured out how blending is specified for certain map objects, so some stuff may or may not be blended when it shouldn't or should be. But I did figure out terrain blending, so that's nice. I came across a code snippet for MZB/MMB "decryption" after extensive Japanese Googling, which I'm pretty sure was written by the author of ffxitools, but am not sure. But, thanks to that guy for being able to load the encrypted maps, I guess.

Character sets are also supported, via a new format called "ff11datset". An example sample_hume01.ff11datset will be included in the Noesis scenes directory. It works like this:

Code: Select all

NOESIS_FF11_DAT_SET
;^ must be the first line of the file

;search for dats using a path retrieved from a registry key
setPathKey	"HKEY_LOCAL_MACHINE" "SOFTWARE\PlayOnlineUS\InstallFolder" "0001"

;search for dats on a path relative to this file 
;setPathRel	"./"

;search for dats on an absolute path
;setPathAbs	"c:/whatever/ff11/"

dat			"__skeleton"	"ROM/27/82.dat"
dat			"__animation"	"ROM/27/82.dat"
dat			"face"			"ROM/27/87.dat"
dat			"head"			"ROM/27/103.dat"
dat			"body"			"ROM/28/7.dat"
dat			"hands"			"ROM/28/52.dat"
dat			"waist"			"ROM/28/84.dat"
dat			"legs"			"ROM/28/116.dat"
dat			"weapon"		"ROM/29/20.dat"

I'll probably put together a little Python script at some point that just tosses up a GUI to let you scroll through body parts and just generates+loads these files as you scroll through.

[MrAdults]

Here's the source:

http://www.richwhitehouse.com/index.php ... t=Snippets

Try not to pretend you wrote it. Unbelievable how many people I came across showing viewers they "wrote" and refusing to send them to people, when clearly all they did was find an old source distribution of ffxitools. Probably some interesting behavioral statistics to be collected there.

[falconcool]

Here is my maxscript to load hi-poly character model in charcter creation.
It still doen't support skeleton /animation.
I'm not good at matrix calculation.I've discussed it with fatduck in his dead forum before,still not finished yet.
MrAdult,hope you can add them into noesis too.Tyvm for doing this project.

Code: Select all

--if (heapSize < 200000) then
--heapSize = 2000000 -- allow ~ 40 MB instead of just 7.5 MB. Prevents"Runtime Error: Out of scripter memory"
fname = getOpenFileName \ 
caption:"FFXI Model File" \
types:" FFXI Character Model File(*.dat)|*.dat|All files (*.*)|*.*|" \

clearlistener()
f = fopen fname "rb"


------------------------------------------Function Define Start
-------------------------------------------------------------------------------------
fn ReadFixedString bstream fixedLen =
(
   local str = ""
   for i = 1 to fixedLen do
   (
      str += bit.intAsChar (ReadByte bstream #unsigned)
   )
   str
)

-------------------------------------------------------------------------------------
------------------------------------------Function Define End
global Meshpointer_array = #()
global Bonepointer_array = #()
global Skinpointer_array  = #()
f_size = getFileSize fname
print "File size"
print f_size
f_lines = (f_size / 16)  as integer
fseek f 0 #seek_set
for i = 1 to (f_lines-1) do (
	pointer_pos = (ftell f) as integer
	magic_string1 = readlong f #unsigned
	fseek f 6 #seek_cur
	magic_num = readshort f #unsigned
	fseek f 4 #seek_cur
	if  magic_string1 == 1162629459 do( ------check if "SQLE"
		
		if magic_num == 10 do( -----check if mesh
			append Meshpointer_array [pointer_pos,0]
		)	----if magic_num = 10
		
		if magic_num == 11 do( --------check if skeleton
			append Bonepointer_array [pointer_pos,0]
		)----	if magic_num == 11
		
		if magic_num == 21 do( ------check if skin
			append Skinpointer_array [pointer_pos,0]
		) -----	if magic_num == 21				
	)----if  magic_string1 == 1162629459
)----or i = 1 to f_lines 
fseek f 0 #seek_set


for i = 1 to (Meshpointer_array.count) do (
  global Vert_array = #()
  global Normal_array = #()
  global UV_array = #()
  global Face_array=#()  
  global facechunk_array =#()
    
  mesh_start = Meshpointer_array [i][1] as integer
	
  fseek f mesh_start #seek_set------------------offset to chunk starting position

  fseek f 96 #seek_cur----------------------------where start to read mesh

  vtcount = readlong f #unsigned
  fseek f 12 #seek_cur
  print "vertice count check"
  print vtcount

    for j=1 to vtcount do( ----read vert
        vx = readfloat f
        vy = readfloat f
        vz = readfloat f
       append Vert_array [vx,vy,vz]	   
	)	----for j=1 to vtcount
	
   fseek f 8 #seek_cur	

	
	 for j=1 to vtcount do( ----read normal
        nx = readfloat f
        ny = readfloat f
        nz = readfloat f
	   append Normal_array[nx,ny,nz]
	 )--for j=1 to vtcount
	 
  fseek f 8 #seek_cur	 
    for j = 1 to vtcount do ( ----readUV
	     Uvx= ( ( readfloat f ) * -1 ) + 1
	     Uvy= readfloat f
	     --Uvy -= 1
	     append UV_array[Uvx,Uvy,0]
	)----j = 1 to vtcount

   code_count = readlong f #unsigned--------------------------------------face index start
   section_count = readlong f #unsigned
	
    for j = 1 to section_count do(
	    fid_pos = ftell f
	    fid_count = readshort f #signed
			
	     if fid_count < 0 do  (   ---------------------------- tri-strip
	      fid_num = fid_count * -1
	      fseek f 2 #seek_cur----skip 3 bytes of 0xff
		  backface=1 
		  global fid_array =#()
			 
			 for k = 1 to fid_num do (
				 fid = ( readlong f #unsigned )+ 1
				 append fid_array[fid,0]				 
			 )----for k = 1 to fid_num
			 
			 for k = 1 to (fid_num - 2) do(
				 f1 = fid_array[k][1]
				 f2 = fid_array[k+1][1]
				 f3 = fid_array[k+2][1]
				 backface=backface*-1
				   				   
                   if f1!=f2 and f1!=f3 and f2!=f3 then(
					   
                        if (backface==1) then (   
                           append Face_array [f1,f2,f3]
                        )    
    
                        if (backface==-1) then (
                           append Face_array [f1,f3,f2]
                        )						
                    )	----if f1!=f2 and f1!=f3 and f2!=f3
			 )----	 for k = 1 to (fid_num - 2)
	     )----if fid_count < 0 
		
	if fid_count > 0 do ( ---------------------------------------------trilist array
	  fseek f fid_pos #seek_set
	  print "Trilist array found ! position check!"
	  print fid_pos
		
	  fid_count = readshort f #signed
	    if fid_count < 0 do(
			
	      fid_num = fid_count * -1	
	    )----if fid_count < 0
	  
       fseek f 2 #seek_cur----skip 2 bytes of 0xff	
	  fid_num = fid_count
		print fid_num
	   for k = 1 to (fid_num / 3) do (
		   f1 = (readlong f #unsigned ) + 1
		   f2 = (readlong f #unsigned )+ 1
		   f3 = (readlong f #unsigned )+ 1
		   append Face_array [f1,f2,f3]
		   		   
	   )---- for k = 1 to (fid_count / 3)
	   
	   ) ----if fid_count>0 
    )----	 for j = 1 to section_count

  -------------------------------------------------------------------------------------build mesh
  -----------------------------------------------------------------------------------
  msh = mesh vertices:Vert_array faces:Face_array
  msh.numTVerts = UV_array.count
  buildTVFaces msh
    for j = 1 to UV_array.count     do setTVert  msh j UV_array[j]
    for j = 1 to Face_array.count   do setTVFace msh j Face_array[j]
    for j = 1 to Normal_array.count do setNormal msh j Normal_array[j]
  --msh.name = chunk_name
  
)----for i = 1 to (Meshpointer_array.count)




gc()
fflush f
fclose f

And here is the data list you can find those hi-poly models/animations.

[MrAdults]

Huh? Noesis already supports all of the character parts, and piecing them together as mentioned above. And animations. And levels.

[falconcool]

Huh? Noesis already supports all of the character parts, and piecing them together as mentioned above. And animations. And levels.

The character creation has high poly models different with normal in game models.Check those files in the data list i added please:)

[MrAdults]

Seems your list is wrong for the US/NA release (or maybe just wrong), then. For example 63/104 isn't mesh data. You should refer to them by file ID's from the tables instead of the .dat names to be consistent.

[falconcool]

Seems your list is wrong for the US/NA release (or maybe just wrong), then. For example 63/104 isn't mesh data. You should refer to them by file ID's from the tables instead of the .dat names to be consistent.

ya,,some of the data in the list is wrong,,but their number is close
63/104 is texture
try 63/103 with my script

[MrAdults]

I figured that other animation format out, but it's pretty awful and worse than the in-game one. Not sure why it exists.

I might add support to Noesis at some point, but I don't see much point, cause it's self-evident and has nothing in common with the main .dats. Maybe I'll just do it in an external script.

[falconcool]

I figured that other animation format out, but it's pretty awful and worse than the in-game one. Not sure why it exists.

I might add support to Noesis at some point, but I don't see much point, cause it's self-evident and has nothing in common with the main .dats. Maybe I'll just do it in an external script.

I'm guessing that's was done by 2 different teams.1st team design the high poly models,2nd team use those models to design in game low-poly count models with optimized animation format.Because this game was designed to be working on very old ps2 with very limited ram size 10 years ago.
I viewed some maps from rom1 folder and found some blending issue you mentioned.Some trees leaves still lacking of correct alpha channel.And lacking of water materials.
Here is the actual in game video of those map area.Hope it will help your coding,Regards:)
https://www.youtube.com/watch?v=kzrmwcSizIY

[MrAdults]

I meant, the format itself is worse. It's bigger and offers worse compression quality for pbchannel or limited functionality for framechannel. Considering the other format had to have existed prior to ship, there's no reason for this format to exist in tandem, even if the content pipeline was entirely different.

[TehDave]

Oh shit, did it finally happen? Can I finally get the models out with weights and animations?

Always wanted to do stuff with FFXI's models but my lack of rigging expertise limited me.