XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Mon Jul 16, 2018 3:24 pm

All times are UTC + 1 hour


Forum rules


Please click here to view the forum rules



Post new topic Reply to topic  [ 7 posts ] 
Author Message
 Post subject: Phantasy Star Online V2/BB - Map Extraction
PostPosted: Thu Dec 17, 2015 10:34 am 
Offline
mega-veteran
mega-veteran
User avatar

Joined: Fri Dec 03, 2010 1:20 pm
Posts: 207
Location: France
Has thanked: 130 times
Have thanks: 31 times














You can make the ads go away by registering

Image

Hello all !

Does anyone can help us to extract map content from both PSOV2 (PC) and PSOBB ?


Image

_________________
My Mediafire Folder : http://www.mediafire.com/?sbd2c9r8ok3u3


Last edited by zaykho on Tue Mar 22, 2016 8:03 am, edited 3 times in total.

Top
 Profile  
 
 Post subject: Re: Phantasy Star Online V2/BB - Map Extraction
PostPosted: Thu Dec 17, 2015 2:17 pm 
Offline
ultra-n00b

Joined: Wed Aug 13, 2014 1:10 pm
Posts: 5
Has thanked: 5 times
Have thanks: 6 times
Hey guys Kion here. I've been working on extracting Pso maps specifically and trying to make all of my documentation and source along the way available as well. For the most part I can parse the binary files in Pso and extract them to json format. Such as this json representation of the psychowand from psobb: link. I don't have a lot of experience with 3d, so right now the main thing I'm having trouble with is applying UV, 3d matrix math, vertex color and things like that. If anyone has any experience with the .nj filetype or similar Ninja0Lbrary models, then the stage files from PSO is just a file containing a list of inner-nj files.

nj
Right now i'm working on nj files using the Katana SDK documentation and Kryslin's ExMldNet source to read nj files. Parsing the information isn't too hard, it's how to apply the rotation and translation matrixes from parent to child and applying the uv is what i'm having trouble with now. I'm hoping that a decent understanding of nj will help me approach the other file formats.

n.rel
In Pso Version 2 to for the PC (and PSO for the dreamcast as well, i think) use a format called n.rel for stage model files. The pointer to the header of the file is at EOF - 16. The header is four dwords long:
Code:
stage = {
    num_sections : dword,
    iff_str : dword,
    section_addr : dword,
    texture_addr : dword
  };

First it has the number of sections. Next it has some kind of IFF string which seems to be " HD" in most cases. The section address is the header to the start of the section list and the texture address a pointer to the list of texture names for the model. The list of texture names follows the exact same format at the NJTL structure.

As for the section list, each entry seems to have the structure of:
Code:
[{
   id : dword,
   pos : {
     dx : single,
     dy : single,
     dz : single
   },
   rot : {
     x : int,
     y : int,
     z : int,
   },
   radius : single(),
   //pointers
   model_addr_a : dword,
   unknown : dword,
   model_addr_b : dword,
   //number of
   num_model_a : dword,
   num_unknown : dword,
   num_model_b : dword,
   end : dword
}
...]


This is not an exact science, but the best guess i have based on the information. The first number seems to be a sequential number for the section number. Followed by the translation for each section. The rotation only seems to be across the y-axis for any given section. Following that there are two groups of model pointers, a and b. I'll get to those in a second. As for the unknown entry i'm not entirely sure since it seems to point to a list of 0x1000 values. As for the end value, i'm not entirely sure if it's a

Model list A and model list B are both a list of pointers to .nj models inside the n.rel file. The first entry is a dword to the start of the nj model. Following that, the model A list struct seems to have a length of 0x2c bytes which is most often filled with zero's. The model B list has a struct length of 0x38 bytes following the first struct. I'm not sure what the difference is, but my best guess is that the model A list is static content while the model B list might have extra pointers to animation key frames.

Basically each section seems to be a scene inside the map. Such as a shop, a room, or a segment of a stage which is first rendered at the origin. It is then rotated around the y-axis in the left or right direction and then translated to where it needs to be relative to the rest of the stage. I wrote a small script to export each section of an n.rel map to it's own model which can be viewed in noesis, but I'd like to be able to export the whole stage file directly if possible.

xj
The xj file format seems to be a derivative of the nj file format or at least similar. Rather than use ninja-chunks it uses polygons which seem to be a pointer to a list of shorts describing the indices. I've done what I can to document the xj file format here: link. For the most part I can read and render the models, but I'm still missing out on some information which I'm hoping to fill in with a better understanding on nj.

n.rel (again)
PSOBB uses n.rel models, but with internal xj models as opposed to inner-nj models. Some of the struct sizes are different. I ported Schthack's model viewer to nodejs, then I went back to try and parse xj models, then I went back to maps again. The result of what I was able to analyze is in this folder: here. For the most part I'm able to parse and render vertices, still missing some texture information and y-translation, but everything else seems to more or less work okay.


Top
 Profile  
 
 Post subject: Re: Phantasy Star Online V2/BB - Map Extraction
PostPosted: Sat Dec 26, 2015 3:53 pm 
Offline
M-M-M-Monster veteran
M-M-M-Monster veteran
User avatar

Joined: Fri Apr 20, 2012 9:24 am
Posts: 2307
Location: Nexus, searching for Jim Kirk
Has thanked: 523 times
Have thanks: 1197 times
PsowKion wrote:
The result of what I was able to analyze is in this folder: here.
Looks great in Noesis with textures! (Though the meshes are not that detailed.)

Finally I managed to get that nasty beast compiled using my old Delphi2005 PE:

Is there any newer source version of the mapviewer?


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.


_________________
"you can't always get things served on a silver tray"
extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-jm9.zip

"We are Microsoft. You will be assimilated. Resistance is Futile."


Top
 Profile  
 
 Post subject: Re: Phantasy Star Online V2/BB - Map Extraction
PostPosted: Sat Dec 26, 2015 7:58 pm 
Offline
mega-veteran
mega-veteran
User avatar

Joined: Fri Dec 03, 2010 1:20 pm
Posts: 207
Location: France
Has thanked: 130 times
Have thanks: 31 times
shakotay2 wrote:
PsowKion wrote:
The result of what I was able to analyze is in this folder: here.
Looks great in Noesis with textures! (Though the meshes are not that detailed.)

Finally I managed to get that nasty beast compiled using my old Delphi2005 PE:


Nice !!


shakotay2 wrote:
Is there any newer source version of the mapviewer?


I don't think, but I will let PsowKion answer this, he know better than me about this subject.

_________________
My Mediafire Folder : http://www.mediafire.com/?sbd2c9r8ok3u3


Top
 Profile  
 
 Post subject: Re: Phantasy Star Online V2/BB - Map Extraction
PostPosted: Sun Dec 27, 2015 9:37 am 
Offline
ultra-n00b

Joined: Wed Aug 13, 2014 1:10 pm
Posts: 5
Has thanked: 5 times
Have thanks: 6 times
As far as I can tell Schthack's map viewer source is incomplete. For the most part it's able to display maps and get the textures, but it's missing some details like rotation. If you look at the dates of the .pas files in the archive it looks like the map viewer was made in 2008. I think he then went back to the drawing board and fixed his mistakes with the model viewer which he released in 2009, which seems to display everything, but he never released the source code for that.

If you want a slimmed down port, I was able to port Schthack's map viewer to Nodejs and modified it to export .obj files. The source is over on github. After that I wasn't sure how to follow up, so I decided to scale down and focus on .xj model types, which the source for is in the same repository.

The problem that I faced for a while was parsing the information wasn't too hard, but I don't have a lot of experience with 3d, so exporting the models turned out harder than expected. I turned to Kryslin's ExMLDNet and started working on .nj models to see if there were any hits that could help me fill in the holes I had for PSOBB maps and .xj models. The good news is while still pretty primitive, I was able to parse NJ files and export the model structure correctly. The source for that is over in his repo.

Taking that a step further, I tried applying that information to PSO version 2 stages, and I finally got somewhere. My most recent attempt, which can be found on github here. Is finally able to export stage files from the game pretty accurately, admittedly while lacking textures at the moment. There's still more information I need to test, document, and fill in, I've just been catching up on some much needed sleep for the last couple of days.

Preview:
Image


Top
 Profile  
 
 Post subject: Re: Phantasy Star Online V2/BB - Map Extraction
PostPosted: Sun Dec 27, 2015 11:02 am 
Offline
M-M-M-Monster veteran
M-M-M-Monster veteran
User avatar

Joined: Fri Apr 20, 2012 9:24 am
Posts: 2307
Location: Nexus, searching for Jim Kirk
Has thanked: 523 times
Have thanks: 1197 times
Keep up your nice work! :)
I myself don't know about .xj and .nj model formats. Is this really needed?
Why not use some more popular format? SMD, FBX, DAE whatever.
PsowKion wrote:
I turned to Kryslin's ExMLDNet and started working on .nj models to see if there were any hits that could help me fill in the holes I had for PSOBB maps and .xj models.
The holes which are fixed by Noesis using face culling?

ExMLDNet uses Visual Basic? Really? Well, ok, just another transformation hurdle to transform it to javascript. :D

Anyway, one of my next aims is to get familiar with Node.js.
This one seems to be a good start:
http://www.tutorialspoint.com/nodejs/
which was my lucky first random hit on inet search.

(Know a better one, maybe?)

_________________
"you can't always get things served on a silver tray"
extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-jm9.zip

"We are Microsoft. You will be assimilated. Resistance is Futile."


Top
 Profile  
 
 Post subject: Re: Phantasy Star Online V2/BB - Map Extraction
PostPosted: Sun Dec 27, 2015 1:00 pm 
Offline
ultra-n00b

Joined: Wed Aug 13, 2014 1:10 pm
Posts: 5
Has thanked: 5 times
Have thanks: 6 times
shakotay2 wrote:
I myself don't know about .xj and .nj model formats. Is this really needed? Why not use some more popular format? SMD, FBX, DAE whatever.


For testing I've been exporting to .obj and would like to try writing a noesis plugin down the line once I get a better handle on the file formats and 3d in general. As for .nj, it's a "Ninja-Chunk Model" file format that Sega developed and used for Phantasy Star Online in the Dreamcast, and for a lot of other games. And .xj is a variation of that format which Sega used for Phantasy Star Online Blue Burst on the PC several years later. The stage files use these model formats internally to build scenes which make up parts of the stage.

Quote:
ExMLDNet uses Visual Basic? Really? Well, ok, just another transformation hurdle to transform it to javascript

I skipped over most of the details for textures, but I was able to port the important parts for how to parse the triangle strip, and the matrix functions needed to render the model correctly. Here's the code I ported for how to parse .nj models: njcm.js, and here's the code I ported for the matrix functions: MatrixUtil.js.

Quote:
The holes which are fixed by Noesis using face culling?

In this case I meant it figuratively. I can pretty much parse out all of the vertexes, normals, and triangle strips from the file format. Often they are intended to be double sided and fixed with culling. The "holes" that I meant were lapses in understanding like flags, or vertex alpha and other parts of the file which I'm able to read, don't completely understand what they do from the file definition. Mainly small things for flags on when to flip the UV or things like that.

Quote:
Anyway, one of my next aims is to get familiar with Node.js.

To be honest I just starting using Nodejs on a whim one day for this kind of thing and it ended out working pretty well. If you're looking at Nodejs tutorials, I think a lot of them are going to be server oriented. Here's a quick video I made for the how/why I use Nodejs: Youtube Link. Personally I'm just using it for file I/O stuff in this case, so there's nothing too special about it beyond personal preference.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: coderxxx, Resiliaxia and 9 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group