TMX file format help

[Barubary]

Although your version of that region looks a bit neater, it is not entirely correct. If the if-statement would read 'if(!BigEndian)', it would have been. (it seems I made the same mistake in another method though)
The color format of the TMX files is either 'ABGR' in BigEndian format, or 'RGBA' in LittleEndian format (since you read R->G->B->A if you read it per byte).

I've updated the code, but have not yet uploaded a new version of the executable.


EDIT: might as well put the adjusted plugin here as well:

Code: Select all

--Name this Palette_TMX.lua
if readDWORD(0x8) == 811093332 then
   if readWORD(0x16) == 0x14 then
      format = 7; --graphics is 4 bit
      palSize = 16;
      tiled = false;
   else
      format = 7;
      palSize = 256;
      tiled = true;
   end
   
   paletteOffset = 0x40;   
   order = 'BGR';
   bigendian = true;
   alphaAtStart = true;
   ignoreAlpha = false;
   enableAlphaStrech = true;
   alphaStretch = {};
   alphaStretch[0] = 0;
   alphaStretch[1] = 128;
   tilesize = {};
   tilesize[0] = 8;
   tilesize[1] = 2;
   setData2(paletteOffset,palSize*4);
else
   warning = 'File is not TMX: '..readDWORD(0x8)
end

[bbrcher]

Hah, thanks for that. I was wracking my brain wondering what was up cause I thought, no way he made an endian mistake, but I tried every option of palette order and such EXCEPT alpha at the start, because when I looked at the hex it was at the end... oh so silly of me. I eventually just figured that instead of grabbing the bytes in chunks you'd just grab them in order since they were just single bytes, not shorts or words or whatever.

I do like your program a lot and it's code it pretty easy to understand so I've been teaching myself C# through emulating your design and making a RMD file inspector (not viewer yet since adding a 3D context seems a little more involved at the moment.).

RMD files from the Persona games have TMX file data embedded into them but they are swizzled. I hacked in some ugly unswizzle code into TGGD, here it is if you are interested (this is for PS2, and I believe it is slightly different for other systems):

Code: Select all

       #region Swizzle
                    byte[] d1 = new byte[nNecessBytes];
                    byte[] d2 = new byte[nNecessBytes];
                    for (int i = 0; i < nNecessBytes && !atEnd; i++)
                    {
                        d1[i] = Next(out atEnd);
                        d2[i] = 0;
                    }

                    uint w = Width;
                    uint h = Height;

                    for (uint y = 0; y < h; y++)
                    {
                        for (uint x = 0; x < w; x++)
                        {
                            uint block_location = (uint)((y & (~0xf)) * w + (x & (~0xf)) * 2);
                            uint swap_selector = (uint)((((y + 2) >> 2) & 0x1) * 4);
                            uint posY = (uint)((((y & (~3)) >> 1) + (y & 1)) & 0x7);
                            uint column_location = (uint)(posY * w * 2 + ((x + swap_selector) & 0x7) * 4);

                            uint byte_num = ((y >> 1) & 1) + ((x >> 2) & 2);     // 0,1,2,3

                            d2[y * w + x] = d1[block_location + column_location + byte_num];
                        }
                    }

                    for (int i = 0; i < nNecessBytes; i++)
                    {
                        *(bmptr++) = palette[d2[i]];
                    }
                    #endregion

I put this into the linear 8BPP section because that's what I use--also added a bool for swizzling since you can't swizzle non-powers-of-two. (Source is Sparky's code)

[PersonaRipper]

This topic was -extremely- helpful for getting the images out of Persona PS2 games. I am trying to take this one step further, but have hit a brick wall. I am hoping someone here can figure out what to do!

I have found on a Persona game disc, a series of .BIN files. I opened one in a text editor, and discovered the first bit of text says it contains a .TMX file. Using TiledGGD I could not initially make sense of the file. I then opened a .TMX file, and immediately opened a .BIN file in the same window, and got a character image in a somewhat muddled fashion. I didn't load the palette from the .BIN, for the result examples I am posting.

The file header in the .BIN is not exactly like the .TMX, especially given the .BIN specifies a .TMX file name before the TMX data. I have been looking for a way to either extract the file from the BIN, edit the bin to match the TMX format, or formulate a settings combination that will let TiledGGD use the .BIN directly.

Attached are some example files I have been using.
-The .TMX file, I load before the .BIN, so it carries over the image settings and lets me see what the image roughly is.
- Target1&2 are .BIN files containing .TMX data.
- Result1&2 are the muddled images I get out of TiledGGD from the respective .BINs

Here are 2 reference images for the Target .BINs:
http://megamitensei.wikia.com/wiki/File ... nistP3.jpg
http://megamitensei.wikia.com/wiki/File ... ssions.jpg

I hope someone has some ideas, I'm stumped.

[bbrcher]

I didn't look at the bin files yet, but I did look at the tmx you provided. The first part is the main image of Aigis (512x512) and the seconded small image (512x64) is just her eye's shut for when she blinks.

The format is slightly different than the TMX file in that either your exporter or the file replaces nulls (0x00) with spaces (0x20) and so the GDD won't read it nicely... since it will try to make a huge image. This screws up the actual image as well because the 0x00's used as indexes into the palette are now 0x20 instead of 0x00. I got some nice pictures from it though, only the alpha channel was confused (never got to full transparency).

Also, some of the TMX header got clipped off--there are 8 bytes before TMX that GDD looks for to know it's a proper TMX file. Also, the end of the TMX file you gave might be missing too, I'm not sure.

**after looking at the bins

The bins do hold TMX files, in this format: Name of TMX file, padding, size of TMX file, TMX file, Name of next TMX file, ...

Looks like you copied the text side of the HEX viewer maybe to get your TMX file before, make sure to use the binary side (those spaces are what are hurting you). The TMX file name probably uses the max length char string of 252 so you can use that to find your files better. BINs can hold other file types as well, so maybe just searching for the TMX header will be enough.

[PersonaRipper]

Thank you for the fast reply!

About the TMX file, I messed up and that wasn't the one I meant to post. I was going to give another clean TMX from the game, but uploaded an edited and renamed BIN instead. Thankfully that mistake lead to additional insight!

With your help I made significant progress. I've been able to pull the main character images from the BINs. I've found that the padding between file name and the start of the TMX proper ends at a hex value of 02 (being the first value of the tmx file.) However, when I apply this rule to the 2nd embedded file (The closed eye portion), it won't work.

What I've done so far:
- Used a hex editor to delete all the values prior to the 2nd file's file name.
-- Found 0x02 and removed the padding+filename from before it. Saved as a new .TMX
-- Loaded new tmx image and found a scattered mess.(A small block of black with red and green dots, most of the palette looks like an old tv test pattern instead of the browns and blacks the character palette should have)
----Opened the first image from the BIN, then opened the closed-eye TMX, which lets me see the image, but there is a couple rows of non-sense and the image is more pixelated than it should be. I -think- what is happening is the secondary TMX uses the same palette as the first, or my method is somehow corrupting the palette data.

You'd think a degree in programming would be sufficient, but we never really covered graphics.

[bbrcher]

Are you talking about the same bin files you linked last time?

I'm pretty sure they are separate files because the 1st palette doesn't work well with the 2nd picture.

Target2.bin :

First TMX starts @ 0x100 (0x100)
The 2nd int32 into the TMX is the size of the file (0x40440), so 0x100+that number is the 2nd file in the bin location.
Second TMX starts @ 0x100 + the end of the First (0x40540) = (0x40640)

Something that might be useful would be to make a BIN extractor that would take a bin and pull out all the files in them for you to look at better. Could have an option to put them in folders as well if ya want. There are a lot of BIN files in the archives and you might find something useful.

[PersonaRipper]

Yeah, it is the same files I am talking about.

One of the issues (the muddled mess with totally incorrect palette), was just a late night oversight. Apparently with the code supplied in this thread, GDD only applies the correct TMX settings if the file extension is in all caps. The files I was generating were lowercase, so I was getting the default settings. Simple enough to solve.

I'm not sure about the other issue, because the nearly correct results I got was from a manually hex-edited file. I was pretty careful about integrity. I probably found some new and impressive way to mess up the file though. I'll try again!

Since there are so many .BINs on the disc, I am writing a small C++ program to extract the files automatically.

How could you tell that the 2nd int was the file length? Does that come with knowing graphic file formats, or well educated guessing, etc?

[bbrcher]

I guess it's mostly from experience in looking at files. Most file formats have some sort of data size or file size number included as an integrity check. I use XVI32 and it has a handy data inspector that tells you what the values at the cursor are in different formats--so I just start at the beginning of the file and move back and forth with the arrow keys to see numbers that make sense.

Yeah, I found the case sensitivity of the bindings to be a little annoying as well. You can edit the code to not send in the regex string in as case sensitive or just add a new binding with tmx as well as TMX.

Glad you are making progress, let us know what you get and do with all this stuff

[PersonaRipper]

Yeah, for now I am using the case sensitivity to my advantage, as any time I have a file I want to test, I do that 'Load a .TMX, then load a .tmx to see if I am roughly on the right track' test. For simplicity sake, I'll refer to .TMX as the files that are properly loaded with settings applied, and .tmx as ones that load, but use a previously loaded .TMX to properly display. Make sense?

With your significant help I've gotten quite close to cracking this thing. I now have a small program that can find and extract TMX files from the .BIN with a fair amount of accuracy. Once it is complete, I will post the source file here for others.

TMX observations:
File starts with 0x02000000, then the file size, as you stated, such as 0x40840000, the next 4 byte block spells out TMX0.
After that there are a few bytes I am not sure about. They seem to be a padding of 4 bytes (0x00 each), then two bytes that together equal 1 (possibly the 'Skip Size' as read by GGD?). The following 2 byte group appears to be the width of the image (usually 0x0002 -> 512), followed by another 2 bytes for the height.

Attached is progress I have made with the Target2.BIN of Aigis.
Inside is:
- i_bust_03_b2.TMX - Aigis's body shot as ripped by my application.
- i_me_03_b2.TMX - Aigis's closed eyes frame as ripped by my application.
- manualEdit.tmx - Aigis's closed eyes file, but I used a hex editor and did it myself. This file is byte for byte the same as the other version, but a bit longer at the end, as my application removes the padding.
- aigis_head.png - shows the current progress on the eyes closed image, which is nearly correct.

I found that on the full body TMX, the previously mentioned height and width values are both 512. When I looked at the closed eye TMX, they are 512 and 64. However, it does not appear that GGD is reading it right, or respecting the image size. If the closed-eye TMX is loaded first, it is loaded as a 64x64 image, and the palette looks incomplete.

If the eye TMX is loaded, then the body TMX, then the eye tmx, we get the .png I included, which is a 512x512 image with roughly the correct results. Ideally, there must be a way to get the image loaded at first try in the right 512x64 size, but I think I am done for tonight.

As for why I am working so hard to do this, it is for my users!
I have written a Persona fusion calculator / database / assistant program for Persona 3 and 4 called 'Igor's Index'
The project page is here.
It lets you look at all the vital info on any persona, predicts fusions, and lets you search for persona based on your specific needs. If you are a player, as I think you are given your knowledge of Aigis, you may want to check it out. Totally free of charge, of course.

By popular demand, I (with the aid of this forum topic) managed to include images of each Persona on their profile. An update I am ever so slowly working on, involves the party members, so I wanted to get clean images of each of them, and found they were trapped in these .BIN files. With your additional help, I've now got the means to extract the profile shots of each party member and social link. Frankly, I could stop right there, but now I am personally curious about getting the closed eye frames, or possibly just doing it for completest sake. Curiosity may have killed the cat, but it just sucks away my spare time. When I was younger I wanted to be in game design, so seeing how things are packaged on a professional disc like this is also kind of fun.

[bbrcher]

That's a pretty sweet program you got there

I'm able to get good pictures from all of your TMX's btw, so I'm not sure what's up with your weird sizes. Are your plugin files exactly like the ones posted in this thread? Make sure you use the corrected palette lua file that Barubary gave. The png file looks exactly what I get if I load the 64 pixel tall image in as generic data instead of specific data btw. The garbage on the top is the palette.

As for the structure of the header, I don't remember all of my work on that but here is the gist:

Code: Select all

struct TMXHeader {
	int32		startUnknown;	//0x02000000
	int32		tmxSize; 	 	//size of file including this header
	char[4] 	magicTag; 		// "TMX0"
	int32		unknown;  		//padding 0
	int16		unknown;  		//non zero, may be different version value
	int16		tmxWidth;
	int16		tmxHeight;
	int32		tmxFlags; 		//so far only discerned 2 formats
	int16		tmxEnd;   		//0x00FF 
	char[36]	reserved; 		//nulls
}; //0x40 bytes

Somewhere in there is a versioning value--I saw that it was different for P3, FES and P4, but I don't remember what was different anymore.

[PersonaRipper]

....... Wow.
Lessons:
- Don't work on something like this all night and expect no mistakes.
- Don't edit files when you have no clue what you are doing.

Thanks again bbarcher. The problem this time was NOT with my application. A few days ago I made a new set of .lua files for .tmx format, thinking I could experiment and tweak settings there if need be (without modifying the .TMX binding). Somehow, I brilliantly managed to edit the Graphics_TMX.lua file to
width = readWORD(0x14);, where the value is supposed to be 0x12. D'oh. No wonder it wasn't reading correctly.

Also, I somehow never thought of making a struct to deal with that data. I guess I've been approaching this problem far too linearly. *kneels* I have much to learn.

Everything is now reading correctly! I will provide the source code soon, once I've cleaned up my comments and such.

[PersonaRipper]

Alright, the code seems to be ready!
It is a simple batch application via the command line (or drag and drop?). I just compiled it, put it in the directory in question and used the command:
name_of_executable *.BIN
, and the program did the rest.
For ease of use, I made it convert the file extension to all caps, but that is easy enough to comment out if you want otherwise.

This has me wondering, does TiledGGD have a batch mode? Some command line arguments to make it translate a whole directory of .TMX to .png? That would save me a heck of a lot of time and effort. I already played around with it, and googled to no results.

[bbrcher]

Pretty sure it doesn't, but the source is available, I say flex some more of those programming muscles and add it in. I'm sure the owner (Barubary) would be fine with the addition

As for the extractor, I've got a request/suggestion: Pull the first 252 bytes for the file name and then the 253-256 bytes for the file size and extract that file instead of going into the TMX header. This way you can extract any file from a BIN. There are a lot more files that are stored in BIN's than just TMX's, one of them being a SPR file that you can use to get more image info out of. Also, I'm currently working on a model extractor and there are plenty of those hanging around in BIN's as well. Be pretty cool to add to your program an animating persona eh?

[PersonaRipper]

Good points!

I was so focused in on the problem that it wasn't generic enough for other possible purposes. I've made the suggested change and tested it on a few tmx-bins. Few issues I found is if the filename is not stated, the extraction probably will fail for that BIN, and if the filename includes a directory path, if the directory doesn't exist the files won't be written. Just to be safe, have a 'data' subdirectory, as thats the only one I've seen. I'd rather not use non-standard libraries to fix the directory issue, and it doesn't appear possible in standard C++ from my quick glance on google.

I guess I will take a look at GGD, see what I can do. The problems just keep stacking up! But it is this, or manually saving some 900 images.

As for the models, I'm not that interested in them....yet, at least. I personally prefer sprites over polygons, and the still images seem to have satisfied my users. It is certainly a possibility for the future though. I wish you luck on the modeling though, and would be interested in seeing the finished app.

[bbrcher]

OK, you got me off my lazy butt. I modified your code to just search for the TMX0 string and then pull out the TMX files from any file you give it. As an added bonus/spammer, it'll convert it to a 32b BMP file as well. Too bad windows ignores the alpha channel, I had to get a trialware app to view it to make sure it was coming out correctly. I may change this to output a TGA later, I dunno.

I didn't try this with every file type, but I think it should do the trick. For kicks I dropped the BTL.CVM (~300MB file) from Persona 4 onto it and got 1713 tmx/bmps. I think only a few of them are swizzled so they look like junk.

*only supports 4/8bit TMX files atm, and I know there are 1/2/16/24/32bit versions out there somewhere, just have to find them and add a check*

**uses a locally global counter for unknown## filenaming, so after each drag/drop I'd move those files somewhere else or just drag all the files you want to search at the same time**

***related to that, if there are two files with the same name... they will be overwritten by any later extraction***