XeNTaX Forum Index
Forum MultiEx Commander Tools Tools Home
It is currently Mon Nov 19, 2018 2:51 pm

All times are UTC + 1 hour


Forum rules


Please click here to view the forum rules



Post new topic Reply to topic  [ 44 posts ]  Go to page Previous  1, 2, 3
Author Message
 Post subject: Re: JX3 HD Remake 剑网3 (.dat files)
PostPosted: Thu Oct 04, 2018 12:11 pm 
Offline
ultra-veteran
ultra-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 370
Has thanked: 16 times
Have thanks: 209 times
shakotay2 wrote:
because the BBox calculated from the unsigned vertex data in hex2obj doesn't match with the real values so far. Need to look what's going wrong there:
# BBox 0.000000 255.468750, 0.000000 255.468750, 0.000000 255.468750

Why does it need to match with the calculated bounding box? I think they're using unsigned shorts to map with the floats within the BBox so that they won't lose too much precision. So we just need to divide the bounding box space into 65535 segments on each axis to get their corresponding scale units.

_________________
"No investigation means no right to speak."
- An Imitable Workflow for Reverse Engineering a Game Model
- Reverse Model Wireframe


Say it with action: click the Image when you get helped.


Top
 Profile  
 
 Post subject: Re: JX3 HD Remake 剑网3 (.dat files)
PostPosted: Thu Oct 04, 2018 1:00 pm 
Offline
M-M-M-Monster veteran
M-M-M-Monster veteran
User avatar

Joined: Fri Apr 20, 2012 9:24 am
Posts: 2495
Location: Nexus, searching for Jim Kirk
Has thanked: 599 times
Have thanks: 1278 times
yeah, my bad, I thought the BBox auto calculation went wrong, but seems it doesn't :D

65535? depends on your divider when creating the mesh - I just need to divide 5376.0376 by 255.469 for the x-scaling for example.

well, seems to fit:


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.


_________________
Bigchillghost, Reverse Engineering a Game Model: http://forum.xentax.com/viewtopic.php?f=29&t=17889
extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-jm9.zip


Top
 Profile  
 
 Post subject: Re: JX3 HD Remake 剑网3 (.dat files)
PostPosted: Fri Oct 05, 2018 1:26 am 
Offline
beginner
User avatar

Joined: Sat Mar 25, 2017 1:40 am
Posts: 23
Has thanked: 6 times
Have thanks: 1 time
shakotay2 wrote:
Is possible, but generally that splitting results in random submeshes (SMs). It's just kind a luck that the static models seem to have roundabout 500 faces clusters here, imho.
So for characters it's more sensefull to insert those lines manually, as there's aren't so many SMs, head, upper body, lower body for example.

I see, thank you...
I'm just wondering, is it possible for you to add some code to the Make_obj-JX3.exe tool to make it so the output .obj file had the same name as the .mesh file?
And if it's not too much trouble then maybe add batch feature to the tool too? Cuz this game had too many files and going through each .mesh at a time is really tedious and tiring to find the player models. Sorry for nagging so much... Just ignored my babbling if it's too much trouble =)

EDITED:
I tested a few of the meshes using Make_obj-JX3.exe tool and these 3 samples are the one that cannot be imported into 3D Software after forming the obj file.
https://drive.google.com/open?id=1FoZn5 ... t1Qogf4E5y


Top
 Profile  
 
 Post subject: Re: JX3 HD Remake 剑网3 (.dat files)
PostPosted: Fri Oct 05, 2018 11:06 am 
Offline
M-M-M-Monster veteran
M-M-M-Monster veteran
User avatar

Joined: Fri Apr 20, 2012 9:24 am
Posts: 2495
Location: Nexus, searching for Jim Kirk
Has thanked: 599 times
Have thanks: 1278 times
petventh18 wrote:
I'm just wondering, is it possible for you to add some code to the Make_obj-JX3.exe tool to make it so the output .obj file had the same name as the .mesh file?
And if it's not too much trouble then maybe add batch feature to the tool too? Cuz this game had too many files
yeah, I was thinking about this since a long time. The batch feature is simple for console apps, but a GUI app needs an auto exit feature like SendMessage (hwnd, WM_CLOSE, 0L, 0L) ; which I used; but not sure about the side effects. :D

But didn't you plan to create a Noesis script?

Quote:
EDITED:
I tested a few of the meshes using Make_obj-JX3.exe tool and these 3 samples are the one that cannot be imported into 3D Software after forming the obj file.
yeah, I see. Would require to use the addresses from the "magic table" at 0x90 where I only take the counts from so far. (Auto calculation of addresses doesn't seem to work for all files.)

_________________
Bigchillghost, Reverse Engineering a Game Model: http://forum.xentax.com/viewtopic.php?f=29&t=17889
extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-jm9.zip


Top
 Profile  
 
 Post subject: Re: JX3 HD Remake 剑网3 (.dat files)
PostPosted: Fri Oct 05, 2018 8:04 pm 
Offline
beginner
User avatar

Joined: Sat Mar 25, 2017 1:40 am
Posts: 23
Has thanked: 6 times
Have thanks: 1 time
shakotay2 wrote:
But didn't you plan to create a Noesis script?

I've only started learning C# language a year ago (so still super noob when come to program languages)... And when I tried python script along with some example plugin from other games and some tutorial on Youtube... it's basically a nightmare for noobie such as myself since every script seem to different from one another when it come to section like
Code:
def noepyLoadModel(data, mdlList):
or
Quote:
class mesh(object):
or
Code:
class mesh(SanaeObject):
... There's so many variation that I can't do it with my current skill... I guess I'll have to start learning python first before attempting again... :oops: :oops: :oops:

Quote:
yeah, I see. Would require to use the addresses from the "magic table" at 0x90 where I only take the counts from so far. (Auto calculation of addresses doesn't seem to work for all files.)

Since can't change the offset from the tool, I guess will have to manually using Hex2Obj... :(


Top
 Profile  
 
 Post subject: Re: JX3 HD Remake 剑网3 (.dat files)
PostPosted: Fri Oct 05, 2018 10:25 pm 
Offline
M-M-M-Monster veteran
M-M-M-Monster veteran
User avatar

Joined: Fri Apr 20, 2012 9:24 am
Posts: 2495
Location: Nexus, searching for Jim Kirk
Has thanked: 599 times
Have thanks: 1278 times
petventh18 wrote:
it's basically a nightmare for noobie such as myself since every script seem to different from one another when it come to section like
I have a deja vue :D
Yea, I know that feeling; here's some simple script for beginners with some Noesis/python knowledge:
viewtopic.php?f=16&t=9534&p=141865&hilit=snowman#p141865
(ah, see, there's some math with the face indices buffer, feel free to ask, why, that's a matter of the format, not of python)
(Noesis is a powerful tool, there's absolutely no doubt, but the first scripting steps, are not so nice. :cry: )

Quote:
Since can't change the offset from the tool, I guess will have to manually using Hex2Obj... :(
That's a good start, imho.
But for hundreds of *.mesh files you might prefer this one (cmd line tool, GUI pops up and exits):

tested with 3 files only
  • you need the dlls from the previous zip
  • you need to adjust the "path to the exe" in the .cmd file.
  • blanks in *.mesh pathname not supported!
  • start the cmd file in the folder where your *.mesh files reside

btw: you might drag 'n drop a single *.mesh onto the exe, too


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.


_________________
Bigchillghost, Reverse Engineering a Game Model: http://forum.xentax.com/viewtopic.php?f=29&t=17889
extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-jm9.zip


Top
 Profile  
 
 Post subject: Re: JX3 HD Remake 剑网3 (.dat files)
PostPosted: Fri Oct 05, 2018 11:48 pm 
Offline
beginner
User avatar

Joined: Sat Mar 25, 2017 1:40 am
Posts: 23
Has thanked: 6 times
Have thanks: 1 time
shakotay2 wrote:
...here's some simple script for beginners with some Noesis/python knowledge:
viewtopic.php?f=16&t=9534&p=141865&hilit=snowman#p141865
(ah, see, there's some math with the face indices buffer, feel free to ask, why, that's a matter of the format, not of python)....
But for hundreds of *.mesh files you might prefer this one (cmd line tool, GUI pops up and exits):


Thanks a lot for the info and for the tool update... it'll save me lots of time to sort through those models...

You mentioned that the UVs is 2 bytes off... so restart the tool and type in the new value on the editbox... then load the same mesh again, correct? Because it's hard to find the correct texture for the correct model since they're name differently... would be so nice if the name of texture match well like how Umodel arrange them... or even like how Ninja Ripper did it...
I wonder how the developer even keep track on all their models/texture with all these kind of names... or is it that the extract tool automatically rename them?


Top
 Profile  
 
 Post subject: Re: JX3 HD Remake 剑网3 (.dat files)
PostPosted: Sat Oct 06, 2018 1:52 am 
Offline
ultra-veteran
ultra-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 370
Has thanked: 16 times
Have thanks: 209 times
petventh18 wrote:
I wonder how the developer even keep track on all their models/texture with all these kind of names... or is it that the extract tool automatically rename them?

There do seem to be some text recording the asset names but not every package outputs them, and some of these info is in json format which makes it so annoying/difficult to read. Yep you're correct the tool names them automatically using the orders of their chunks.

_________________
"No investigation means no right to speak."
- An Imitable Workflow for Reverse Engineering a Game Model
- Reverse Model Wireframe


Say it with action: click the Image when you get helped.


Top
 Profile  
 
 Post subject: Re: JX3 HD Remake 剑网3 (.dat files)
PostPosted: Sat Oct 06, 2018 3:59 am 
Offline
beginner
User avatar

Joined: Sat Mar 25, 2017 1:40 am
Posts: 23
Has thanked: 6 times
Have thanks: 1 time
Bigchillghost wrote:
There do seem to be some text recording the asset names but not every package outputs them, and some of these info is in json format which makes it so annoying/difficult to read. Yep you're correct the tool names them automatically using the orders of their chunks.

Ah I c... thanks for clearing that up... I opened the json file on Notepad++ and only some .srt are recorded in it...
ex: "srtFile": "data\\source\\maps_source\\树入库临时保存\\s_雪竹019c4_st.srt"
But the output file will be something like 0000XXXX.SRT

And the .SRT files itself does have some texture name recorded like
(SP_wanhuashupi001H.tga SP_wanhuashupi001H_Nor.tga SP_wanhuashupi001 S_wanhuazitengluo002.tga S_wanhuazitengluo002_Nor.tga Atlas S_wanhuazitengluo002_Billboards.tga S_wanhuazitengluo002_Billboards_Nor.tga...)
So based on that we can see some diffuse and normal texture that are corresponding to the wanhuashupi001.mesh...

But because everything was renamed, it will take a lot of guessing game to match the correct texture into the model (assuming that the UV is correct after turning the mesh to obj)...

If it's not too much trouble then may I ask if it's possible to modify the extraction tool to not rename the file based on the chunk but leave it as is? (Doesn't matter if it's Chinese characters)... Some folder had over 6000 files, so trying to guess their texture based on the current name is quite literary a nightmare.
That or maybe had a .log file that keep track of the original name to the current exacted name?
And again, if it's too much of a hassle then forget about it... I'm already very thankful that you created the tool as it is. :D


Top
 Profile  
 
 Post subject: Re: JX3 HD Remake 剑网3 (.dat files)
PostPosted: Sat Oct 06, 2018 4:51 am 
Offline
ultra-veteran
ultra-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 370
Has thanked: 16 times
Have thanks: 209 times
petventh18 wrote:
assuming that the UV is correct after turning the mesh to obj

Well, about that, have you checked if they're mapping?
petventh18 wrote:
If it's not too much trouble then may I ask if it's possible to modify the extraction tool to not rename the file based on the chunk but leave it as is?

Maybe. That would require more research on the organization of the chunks, which can take some time. I'll see what I can do with it. But don't expect too much yet.

Edit: I can say that it's impossible to read filenames now because:
- The assets are linked by different objects in the json files, usually more than one times for each asset, and you can't locate the asset names with the MD5 hashes coz they don't even esixt in the json;
- There do seem to be a filelist in a certain json file, and there're some texture names in a binary chunk(using the ext .def in the following version), but their amounts and extensions don't match with the actual outputs, nor their sums.
petventh18 wrote:
Doesn't matter if it's Chinese characters...

I was wondering if the names will output correctly if they contain any Chinese. I hope I don't have to deal with the character encoding stuffs.
petventh18 wrote:
Some folder had over 6000 files, so trying to guess their texture based on the current name is quite literary a nightmare.

Ah, I can imagine that. :D
petventh18 wrote:
That or maybe had a .log file that keep track of the original name to the current exacted name?

Sounds good, but not necessary, nor possible. Because the tool didn't read any filenames at all. It just decompresses each chunk and assigns the index of the loop to the string buffer as the output filename (in hex). The simplest solution is to save every chunk with a decimal index name so that you can sort them with your Windows explorer. It still needs your guessing work though. But it's an alternative. Better than nothing.

Edit: Now you can have it:

Changes:
- Changed naming patterns from hex to decimal index;
- No text/json files will be merged;
- Improve detection of text/json data (first 4 bytes only).
- The files with the signature "\x08\x00\x00\x00" will be separated from the TBD files, using the extension ".8";
- The binary chunk(s) containing texture names will use the extension ".def".


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.


_________________
"No investigation means no right to speak."
- An Imitable Workflow for Reverse Engineering a Game Model
- Reverse Model Wireframe


Say it with action: click the Image when you get helped.


Last edited by Bigchillghost on Sat Oct 06, 2018 8:51 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: JX3 HD Remake 剑网3 (.dat files)
PostPosted: Sat Oct 06, 2018 8:05 am 
Offline
M-M-M-Monster veteran
M-M-M-Monster veteran
User avatar

Joined: Fri Apr 20, 2012 9:24 am
Posts: 2495
Location: Nexus, searching for Jim Kirk
Has thanked: 599 times
Have thanks: 1278 times
petventh18 wrote:
shakotay2 wrote:
...here's some simple script for beginners with some Noesis/python knowledge:
viewtopic.php?f=16&t=9534&p=141865&hilit=snowman#p141865
(ah, see, there's some math with the face indices buffer, feel free to ask, why, that's a matter of the format, not of python)....
But for hundreds of *.mesh files you might prefer this one (cmd line tool, GUI pops up and exits):


Thanks a lot for the info and for the tool update... it'll save me lots of time to sort through those models...
I wasn't sure if it would work for 100s of models... :D

Quote:
You mentioned that the UVs is 2 bytes off... so restart the tool and type in the new value on the editbox... then load the same mesh again, correct?
Forget about that; that's for the old version with auto calculation of addresses; sometimes it gave incorrect values (2 bytes off for example) for the uvs' start.

The cmdline version gets all values from the magic table at 0x90 (counts and addresses) now. But it "relies" on a 12 bytes offset in the uvs' block (3 floats to be skipped before the uv data).

btw: this format is a very good start to be used with a Noesis script (you don't need the face indices buffer math in my previous linked sample).
The codelines for getting the face indices should look like this (untested):
faceBuff = bs.readBytes(faceCount*3*4)
rapi.rpgCommitTriangles(faceBuff, noesis.RPGEODATA_UINT, faceCount*3, noesis.RPGEO_TRIANGLE, 1)
UINT because of DWord face indices
and you don't need this weird
b = struct.pack('H' * len(idxBuf), *idxBuf) for this format

here's a working Noesis script, tested with 00000161.mesh:
Code:
from inc_noesis import *
import noesis
import rapi

def registerNoesisTypes():
    handle = noesis.register("JX3", ".mesh")
    noesis.setHandlerTypeCheck(handle, noepyCheckType)
    noesis.setHandlerLoadModel(handle, noepyLoadModel)
    return 1

def noepyCheckType(data):
    bs = NoeBitStream(data)
    bs.seek(0x54, NOESEEK_ABS)
    idstring = bs.readUInt()
    if idstring != 0x4D455348:
        print("not a JX3 file!")
        return 0
    return 1

def noepyLoadModel(data, mdlList):   
    ctx = rapi.rpgCreateContext()
    bs = NoeBitStream(data)
    #rapi.rpgClearBufferBinds()
    bs.seek(0x90, NOESEEK_ABS)
    vertsCount = bs.readUInt()
    faceCount = bs.readUInt()
    bs.seek(0xA8, NOESEEK_ABS)
    uvAddr = bs.readUInt()
    uvAddr += 12
    bs.seek(0xB4, NOESEEK_ABS)
    FIaddr = bs.readUInt()

    bs.seek(0x12C, NOESEEK_ABS)
    VertBuf = bs.readBytes(vertsCount * 6)
    bs.seek(uvAddr, NOESEEK_ABS)
    uv1Buf = bs.readBytes(vertsCount * 6)
    bs.seek(FIaddr, NOESEEK_ABS)
    idxBuf = bs.readBytes(faceCount*3*4)
    rapi.rpgBindPositionBufferOfs(VertBuf, noesis.RPGEODATA_USHORT, 6, 0)
    rapi.rpgBindUV1Buffer(uv1Buf, noesis.RPGEODATA_USHORT, 6)
    rapi.rpgCommitTriangles(idxBuf, noesis.RPGEODATA_UINT, faceCount*3, noesis.RPGEO_TRIANGLE, 1)

    mdl = rapi.rpgConstructModel()
    mdlList.append(mdl)

    return 1
(Maybe I should have used some NOESEEK_REL constants but I'm too lazy to calculate the required offsets. :D )

Also I didn't care for the normals buffer, as always.
addr can be found in the magic table; refer to the struct that Bigchillghost posted.

_________________
Bigchillghost, Reverse Engineering a Game Model: http://forum.xentax.com/viewtopic.php?f=29&t=17889
extracting simple models: http://forum.xentax.com/viewtopic.php?f=29&t=10894
Make_H2O-ForzaHor3-jm9.zip


Top
 Profile  
 
 Post subject: Re: JX3 HD Remake 剑网3 (.dat files)
PostPosted: Mon Oct 08, 2018 1:15 pm 
Offline
ultra-veteran
ultra-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 370
Has thanked: 16 times
Have thanks: 209 times
Bigchillghost wrote:
Now the problem is to get correct normal vectors:
Image

I can't believe that they actually packed the normals in this way. You need to perform the following operations to get the correct normals:
Code:
for (int i = 0; i < Vcount; i++)
{
   dblNM[i].x = charNM[i].x / 127.0;
   dblNM[i].y = charNM[i].y / 127.0;
   dblNM[i].z = charNM[i].z / 127.0;
   if (dblNM[i].x < 0.0)
      dblNM[i].x = -dblNM[i].x - 1.0;
   if (dblNM[i].y < 0.0)
      dblNM[i].y = -dblNM[i].y - 1.0;
   if (dblNM[i].z < 0.0)
      dblNM[i].z = -dblNM[i].z - 1.0;
   dblNM[i].x = -dblNM[i].x;
   dblNM[i].y = -dblNM[i].y;
   dblNM[i].z = -dblNM[i].z;
}

Test result:

Is that a common case?


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.


_________________
"No investigation means no right to speak."
- An Imitable Workflow for Reverse Engineering a Game Model
- Reverse Model Wireframe


Say it with action: click the Image when you get helped.


Top
 Profile  
 
 Post subject: Re: JX3 HD Remake 剑网3 (.dat files)
PostPosted: Tue Oct 09, 2018 2:41 am 
Offline
beginner
User avatar

Joined: Sat Mar 25, 2017 1:40 am
Posts: 23
Has thanked: 6 times
Have thanks: 1 time
@shakotay2
Thanks for the script... It make it a lot easier for noobie me :) Although I couldn't get the CMD version of the tool to work at all. I place the prev folder with some .mesh inside of it. But it just won't do anything when click or drag a mesh file to it. I made sure to rename the path inside the .cmd file as well...

But it's very hard matching the texture and model so I decided to buy an Xoyo account and tried my luck with Ninja Ripper... good thing it worked...
Image
Image

So if anyone don't wanna sort through the mesh manually then try Ninja Ripper... the down side is that you can only rip the model you see and you'll have fking nightmare trying to register for a game account without a chinese phone number. And the face and hair position don't match properly so you have to manually trying to put it together.


Top
 Profile  
 
 Post subject: Re: JX3 HD Remake 剑网3 (.dat files)
PostPosted: Sat Nov 03, 2018 4:13 pm 
Offline
ultra-veteran
ultra-veteran
User avatar

Joined: Tue Jul 05, 2016 9:37 am
Posts: 370
Has thanked: 16 times
Have thanks: 209 times
Probably this is pointless but here's the truth: since I've updated my binary FBX builder to support face materials and other features,
this format had been the perfect sample for testing purpose. Wouldn't it be a waste if it'd never got released? :D

For example here's one of the material group of the entire mesh from 000005E0.mesh (7 in total):
Image

Note that the UVs endoced as WORD are a bit off the correct position which I have no clue of why.
I made a little correction for this model only so it's not guaranteed to work for all UVs.
For float UVs you don't have worry about it though.

Just run the tool without prams to see the usage.


Attachments:


You do not have the required permissions to view the files attached to this post. Register to gain access.


_________________
"No investigation means no right to speak."
- An Imitable Workflow for Reverse Engineering a Game Model
- Reverse Model Wireframe


Say it with action: click the Image when you get helped.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 44 posts ]  Go to page Previous  1, 2, 3

All times are UTC + 1 hour


Who is online

Users browsing this forum: Darko, Maximmum and 7 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group