Part way to finding a vertex header, need some pointers

[St3ve]

Oh, I see. My original script just read in the terrain vertices until it reached what might be the vertex color header.

[St3ve]

I took a look at the vertex clouds again, I'm starting to wonder if the reason I can't see the mechs is because the their vertex data is "off" and forming a different shape- if the header itself is read in as a float wouldn't that cause the resulting vertex coordinates to be one off, thus altering the entire pattern as the script reads in the first coordinate as the second coordinate and so on? I altered the the script to read in everything and I couldn't recognize even the level data.

[shakotay2]

if the header itself is read in as a float wouldn't that cause the resulting vertex coordinates to be one off

would spoil things, yes.
Just read 11 vertices per block after the 4 bytes header and you're fine.

So the vertices are not so bad (displaying 11 verts per block you can even see two steps, not on this pic, though), but autocreated face indices don't work as expected (tri stripped FIs look even worse):

autoFIs.JPG

v -42.000000 4.000000 -30.000000
v -42.000000 8.000000 -30.000000
v -43.000000 4.000000 -29.000000
v -43.000000 8.000000 -29.000000
v -35.000000 4.000000 -29.000000
v -35.000000 8.000000 -29.000000
v -37.000000 4.000000 -29.000000
v -37.000000 8.000000 -29.000000
v -27.000000 4.000000 -29.000000
v -27.000000 8.000000 -29.000000
v -29.000000 4.000000 -29.000000
#-----------------------
v -13.000000 4.000000 -29.000000
v -13.000000 8.000000 -29.000000
v -14.000000 4.000000 -30.000000
v -14.000000 8.000000 -30.000000
v -5.000000 4.000000 -29.000000
v -5.000000 8.000000 -29.000000
v -6.000000 4.000000 -30.000000
v -6.000000 8.000000 -30.000000
v 3.000000 4.000000 -29.000000
v 3.000000 8.000000 -29.000000
v 2.000000 4.000000 -30.000000

[St3ve]

Assuming my script is throwing off the mech vertices, I suppose working backwards with files or savestates is impossible and I would need to go through them by hand. I just wonder how I'll know when or if I find the mech header, the formatting must be less obvious than the terrain header or he would have found it by now.

[shakotay2]

I'm not familiar with the PS2 savestates, what they contain exactly.
I'm surprised that they contain textures and part of a level (arena) because for PC savegames the senseful philosophy
is to save only changes of states (this is what PS2 savestates claims to be but seems it isn't?).

So I'd expect positions of mechs and characters, contents of chests, whatsoever will change in a game.

Anyways, you spoke of "parts of the mechs", where did you get them from, how do they look like?
Where do you know from that the mechs are stored in savestates?

What is the structure of PS2 archives, what do they contain?
I don't know how much you know about this (me I don't know anything, I don't have a console).

Did this unknown guy you spoke of ever extract a mech from a savestate?

btw: forget about the 11 vertices, I used the 3rd byte of the search pattern (terrain header) as a count now as suggested by you
Also for values greater than 10000.0 I set them to 1000.0

here's another point cloud, not a mech:

eeMemory-bin.jpg

(forget about the startaddress, I used a cut part from eeMemory.bin)

Then a promissing block with assumed face indices followed but sadly it was a TIM2 data block (sky_b003).

[St3ve]

To my understanding, a PS2 savestate is simply a RAM dump, hence the ability to extract game assets from it. I tried switching around parts of the mechs in-game in hopes of finding different XYZ values when comparing two savestates, this did not work as I did not factor in changes to mech and inventory stats and thus thousands of differences were detected. PS2 archives differ per game but all the PS2 Front Mission entries use an unencrypted DVDIMAGE.DAT archive with a DVDIMAGE.POS file list. Micheal (the PS23Dformat guy) made it so far is unpacking the archive and finding the terrain header but hasn't found the mech header. There has to be some trace of them in the code but I just don't know how to pick up the trail.

[St3ve]

I've been thinking, if using a script to pull out vertices isn't working, why not pull out headers? It's probably too resource intensive but If I could put each byte into an array then loop through it while comparing entries through re.match I can see how many times said byte appears in the code. From there I can look at the less frequent bytes for variants of a one that likely involve vertices like the terrain header.

[St3ve]

I had this idea the other day after thinking about doing another file comparison- I'm going to write a python script that reads from either two drastically different or very similar savestates and writes the differences/shared values to a third file. Theoretically, either approach would narrow things down to just the model data and stats, at least making it easier to navigate the file. Also, I would like to thank shakotay2. I would still being staring at vertex clouds without your help.

[St3ve]

Made the file comparison script using

Code: Select all

a = open(r'C:\Users\Greg\Documents\Front Mission savestates\FM4 hanger no weapons.bin', 'rb')
b = open(r'C:\Users\Greg\Documents\Front Mission savestates\FM4 hanger one weapon.bin', 'rb')
c = open(r'C:\Users\Greg\Documents\Front Mission savestates\differences.bin', 'wb')

data1 = a.read(4)
data2 = b.read(4)

while data1 != b'' or data2 != b'':

    if data1 != data2:

        c.write(data2)

    data1 = a.read(4)
    data2 = b.read(4)

and managed to create a 40 Kb file that hopefully contains a weapon, the major difference between the two savestates. I've tried getting textures out of it with TextER but it says there's nothing there which is pretty concerning. I've uploaded the file here since the forum doesn't take .bins.

[St3ve]

I managed to take a look at the differences file and found about three groups of floats capped by what might be headers.


Wrote some code.

Code: Select all

import struct
import re
import bpy

coordinates = [0.0, 0.0, 0.0]
vertices = []

f = open(r'C:\Users\Greg\Documents\Front Mission savestates\differences.bin', 'rb')
data = f.read(4)

check = 4.2367777022189116e-38
check1 = 4.228643444893199e-38
check2 = 6.462348535570529e-27

while re.match(b'\x00\x80\x05\x6C', data) is None:
    data = f.read(4)

data = f.read(4)

while re.match(b'\x07\x00\x00\x50', data) is None:
    value = struct.unpack('<f', data)

    if value[0] == check or value[0] == check1 or value[0] == check2:
        value = [0.0, 1]
    coordinates[0] = value[0]

    data = f.read(4)
    value = struct.unpack('<f', data)
    if value[0] == check or value[0] == check1 or value[0] == check2:
        value = [0.0, 1]
    coordinates[1] = value[0]

    data = f.read(4)
    value = struct.unpack('<f', data)
    if value[0] == check or value[0] == check1 or value[0] == check2:
        value = [0.0, 1]
    coordinates[2] = value[0]

    vertex = (coordinates[0], coordinates[1], coordinates[2])
    vertices.append(vertex)
    print(vertex)
    data = f.read(4)

mymesh = bpy.data.meshes.new("vertices")
myobject = bpy.data.objects.new("vertices", mymesh)
bpy.context.scene.objects.link(myobject)
mymesh.from_pydata(vertices,[],[])

And this is the result of two of the three groups.


I have no idea what else I can do at this point. I don't understand how the terrain is so simple but the the rest of the meshes are a mystery and I'm starting to wonder if python or blender is altering the floats somehow. If anyone has ANY ideas I would love to hear them.

[St3ve]

Just tried my file comparison script on a savestate with a mech and one without and didn't get anything that even might have been viable. At this point, I have two theories- mech and weapon data is loaded wholesale into the memory and thus can't be splintered off with the method I'm using or I have managed to isolate the data I'm looking for, it's just not in the form of a float which is odd given terrain data is. Anyone have experience converting bytes?

[shakotay2]

mech and weapon data is loaded wholesale into the memory and thus can't be splintered off with the method I'm using

this could explain it.

btw: can you apply changes to the savestates and load them successfully then?
If so you could overwrite some floats (with 00 00 00 00 or better 00 00 80 3F for example).

It's a tedious task but it should give more insight into the meaning of the changed values.

[St3ve]

It might be possible but at this point I'm more concerned about the floats. Is it possible for different assets to use different float types? I tried a script using doubles and got next to nothing but after months of this it seems that byte long floats akin to the terrain vertices might not be it either.

[shakotay2]

that byte long floats akin to the terrain vertices

never heard of "byte long floats".
From the point clouds I've found so far I wouldn't know why using IEEE754 floats shouldn't do the trick?

(You could try half floats or shorts, though.)

[St3ve]

I keep forgetting to add the 4. I haven't tried 2byte floats since struct.unpack won't accept them. I need to take a step back a learn more about mesh formatting, It could be that the vertices are somehow padded and don't directly follow each unlike the terrain, I just wish I had more examples to work with.