TMX Explorer for Persona 3, Persona 4, and related games
Posted: Sun Aug 01, 2010 2:29 am
TMX Explorer 1.0
Update 10/14/2013: Lots of people keep asking me where to get this since my website went down a few years ago. Here you go: http://www.mediafire.com/download/3ibi3 ... plorer.zip
Note for anyone who was familiar with this tool in the original thread "TMX File Format Help," all releases will now be in this post, which will be updated as required.
Hello everyone, I had been posting this tool in another topic where we (and by "we" I mean mostly other people who I learned from!) basically figured out the TMX format. Instead of me making a new post in that mostly dead thread every month or so when I make a new release, I decided to incorporate all of that information over here. With that I proudly release TMX Explorer 1.0
Overall features
-Capable of opening and displaying 4 or 8 bpp TMX files, including stand alone TMX files and ones contained in BIN, CVM, or SPR files.
-Capable of exporting any of the aforementioned images into PNG format(including transparency).
-Code is as optimized as I (an admitted novice) can make it. Even reading CVM files is fast now.
Note: I know the directory navigation is a big sketchy. If you have some files you want to open in say a directory called "Persona 4 Data" then in the program you would navigate to that folder using the "SPACE" and arrow keys until you find the directory. Then when the "<---" cursor is pointing at the "Persona 4 Data" folder, you would push "ENTER." You would NOT push SPACE on the folder you want to open. Also for any that don't know ".." selection moves you to the parent directory of the current directory, I.E. one directory back. I know this is confusing, and I COULD change it, however I have gotten used to using the application in this way and I suspect maybe anyone else who has used it might have as well. Changing it after so many releases doesn't seem to be a great idea. Comments on this and anything else are appreciated!
Written in FreeBASIC 0.20.0
Small changes since 1.0
-Added ISO to allowed file types, so you can read TMX directly from the ISO of a valid game. Be careful with this because if you open some other ISO you'll be waiting a long time with little result. You can break from reads and exit with ESCAPE still.
-Minor code correction with likely no implications.
Changes in 1.0:
-The position of some of the information displayed on screen was changed for better readability.
-It is no longer possible for file names to clash, the file names are now generated with (filename_image position within file) If the file clashes, underscores are added to the file name until it does not.
-I looked at more of the game data files in a hex editor and discovered SPR files also contain TMX graphics. Thus SPR was added to the list of allowed file extensions.
-You can now change drives in the directory explorer by pressing V. The interface is very simple, requiring you to type a new drive in the format C: - If the new typed drive is invalid, the application will silently return to where you previously were. This is because the only way I could have made an interactive list of drives to choose from was to include the Windows headers. Though I have not yet tried it since I don't have a working Linux box, the source should compile on Linux with zlib dependency. This attempt at portability is also why the directory navigation is as it is.
-Quick search is now fully implemented. While this is active for all file types, it has the most effect on CVM files. While files are still scanned "on the fly" for the TMX0 signature and TMX files, once a file has been found and loaded, its position is stored in an array. That way if you navigate away from it and back to it, it does not need to be scanned for again. This has less impact than I originally intended because of the next change listed, but it is still quite useful. This information is only saved PER CONTAINER FILE and PER SESSION. So if you load a different container or close the app, we start from scratch.
-Files are no longer read in 1 byte increments, they are now read in chunks based on the file size. This has the following effects: Reading small files(BIN,TMX,SPR) is slightly slower than it could otherwise be, but still much faster than 0.8 version. Big files(CVM) are many, many times faster than in 0.8
Also the memory usage for the application has changed because of this. What this means is that programs memory usage will very briefly increase from time to time as determined by this formula:
Base memory usage in MB + (0.005 * size of file being read in MB)
Base memory usage is around 6.5MB so for the Persona 4 DATA.CVM(2862.4 MB) the memory will briefly spike to:
6.5 + (14.312)
20.812MB
This is not a lot of memory by today's standards for sure, but it is roughly a triple increase on big files, so I thought I'd mention it.
-I added a catch all solution for cases where the program gets confused and wants to read strange data or the user opens a file of valid extension(such as BIN, a reasonably common extension) that is in fact not the kind of BIN from these games and thus contains no image data. In this case the application will present the error:
"Fatal error. Unable to find TMX signature. Exit in 5 seconds."
and properly terminate itself.
Important note: This error will happen when you reach the end of CVM files since I am still having trouble catching EOF on large files. So if you are viewing a CVM and get this message, you probably hit the end of file. I am working on this.
-I cleaned up the code as much as I could: Removed unused variables, fixed spacing and indenting, improved commenting etc.
Features new to 0.8
Major changes:
You can now directly open the CVM files without having to extract them with IsoBuster first!
Note: At the moment, this can be a bit slow. When you first load a CVM expect to wait 20+ seconds for the first image to appear. This is due to the large size of the CVM file (2 Gigabytes+ for some of them) and the fact it is being scanned per byte for the TMX0 signature. However, once you reach the first image and are using S/W keys to move around, the time in between images should be less than one second. If for some reason (I.E. you opened a 2GB CVM on a slow computer) you find it loading and loading and loading (there is a progress update on screen, so as long as it keeps updating the app is not locked), you can break out of the application with ESCAPE.
File name is no longer read from the container file as that was pretty sketchy anyway. It is now generated with this formula:
Container file name + position within container file + current date
Also, a feature was implemented that will prevent you from overwriting already exported files of the same name. If you push ENTER to save a PNG and get a RED message confirmation, the filenames clashed and you did not save the file. This would only happen if A - You already saved a PNG of that same file or B - If you did something unlikely, like opened the DATA.CVM from Persona 3, exported the 5th file, then opened the DATA.CVM from Persona 4 and tried to export the 5th file. Regardless, I would suggest you move your exported PNG files to somewhere other than the application directory after each use. For instance, if you extract a bunch of stuff from Persona 3, and then are going to extract a bunch of stuff from Persona 4, move the extracted Persona 3 stuff out of the main program directory before you move on to avoid clashes.
A quick searching feature is partially implemented to speed up reading of CVM files. Any TMX you have already navigated to within the CVM has its position stored (per session only!) so it is quicker when you navigate "backwards."
Minor Changes:
Should be faster, as files are no longer read from the start each time you move further in, the position is saved. This was necessary to allow efficient reading of large CVM files.
Changed the file index variable from a regular integer to an unsigned long integer. This was part of the reason I was having trouble reading the CVM files as their size was larger than the limit of the variable.
You can no longer select directories without valid files.
Instead of displaying a partial list of files a directory contains, you now either see the message:
"This directory contains readable files." in GREEN if the directory can be opened
or
"This directory does not contain readable files." in RED if the directory cannot be opened"
Known Bug:
I don't think the progress indicator that says "Reading byte X of Y" is correct, but really it's only there to assure the user the application is doing something and not locked.
Compatability:
Confirmed working with:
Persona 3: FES (confirmed by me)
Persona 3 Portable (confirmed by taleds)
Persona 4 (confirmed by me)
Disclaimer:
You use this at YOUR OWN RISK. The author disclaims any and all liability arising from any and all uses of this application.
License:
http://myriaddreams.com/index.php?optio ... 4&Itemid=9
Credits:
PersonaRipper for his help with some programming problems and providing me the source to his and bbrchers extractors which I converted to FreeBASIC code for large parts of the program
Simon Nash for FreeBASIC PNG Library and Jean-loup Gailly and Mark Adler for zlib library
taleds for the compatibility report for P3P
Request:
So far the TMX explorer has been successfully used with Persona 3, Persona 3:FES, Persona 3 Portable, and Persona 4. If you own another game which uses the TMX format and try this application, please let me know whether it works or not at my e-mail address which is in the about section of the program, or through a PM on this message board.
Screenshot:
Enjoy!
~Blyss
Download
Use the attachment or http://myriaddreams.com/projects/applic ... plorer.zip
Update 10/14/2013: Lots of people keep asking me where to get this since my website went down a few years ago. Here you go: http://www.mediafire.com/download/3ibi3 ... plorer.zip
Note for anyone who was familiar with this tool in the original thread "TMX File Format Help," all releases will now be in this post, which will be updated as required.
Hello everyone, I had been posting this tool in another topic where we (and by "we" I mean mostly other people who I learned from!) basically figured out the TMX format. Instead of me making a new post in that mostly dead thread every month or so when I make a new release, I decided to incorporate all of that information over here. With that I proudly release TMX Explorer 1.0
Overall features
-Capable of opening and displaying 4 or 8 bpp TMX files, including stand alone TMX files and ones contained in BIN, CVM, or SPR files.
-Capable of exporting any of the aforementioned images into PNG format(including transparency).
-Code is as optimized as I (an admitted novice) can make it. Even reading CVM files is fast now.
Note: I know the directory navigation is a big sketchy. If you have some files you want to open in say a directory called "Persona 4 Data" then in the program you would navigate to that folder using the "SPACE" and arrow keys until you find the directory. Then when the "<---" cursor is pointing at the "Persona 4 Data" folder, you would push "ENTER." You would NOT push SPACE on the folder you want to open. Also for any that don't know ".." selection moves you to the parent directory of the current directory, I.E. one directory back. I know this is confusing, and I COULD change it, however I have gotten used to using the application in this way and I suspect maybe anyone else who has used it might have as well. Changing it after so many releases doesn't seem to be a great idea. Comments on this and anything else are appreciated!
Written in FreeBASIC 0.20.0
Small changes since 1.0
-Added ISO to allowed file types, so you can read TMX directly from the ISO of a valid game. Be careful with this because if you open some other ISO you'll be waiting a long time with little result. You can break from reads and exit with ESCAPE still.
-Minor code correction with likely no implications.
Changes in 1.0:
-The position of some of the information displayed on screen was changed for better readability.
-It is no longer possible for file names to clash, the file names are now generated with (filename_image position within file) If the file clashes, underscores are added to the file name until it does not.
-I looked at more of the game data files in a hex editor and discovered SPR files also contain TMX graphics. Thus SPR was added to the list of allowed file extensions.
-You can now change drives in the directory explorer by pressing V. The interface is very simple, requiring you to type a new drive in the format C: - If the new typed drive is invalid, the application will silently return to where you previously were. This is because the only way I could have made an interactive list of drives to choose from was to include the Windows headers. Though I have not yet tried it since I don't have a working Linux box, the source should compile on Linux with zlib dependency. This attempt at portability is also why the directory navigation is as it is.
-Quick search is now fully implemented. While this is active for all file types, it has the most effect on CVM files. While files are still scanned "on the fly" for the TMX0 signature and TMX files, once a file has been found and loaded, its position is stored in an array. That way if you navigate away from it and back to it, it does not need to be scanned for again. This has less impact than I originally intended because of the next change listed, but it is still quite useful. This information is only saved PER CONTAINER FILE and PER SESSION. So if you load a different container or close the app, we start from scratch.
-Files are no longer read in 1 byte increments, they are now read in chunks based on the file size. This has the following effects: Reading small files(BIN,TMX,SPR) is slightly slower than it could otherwise be, but still much faster than 0.8 version. Big files(CVM) are many, many times faster than in 0.8
Also the memory usage for the application has changed because of this. What this means is that programs memory usage will very briefly increase from time to time as determined by this formula:
Base memory usage in MB + (0.005 * size of file being read in MB)
Base memory usage is around 6.5MB so for the Persona 4 DATA.CVM(2862.4 MB) the memory will briefly spike to:
6.5 + (14.312)
20.812MB
This is not a lot of memory by today's standards for sure, but it is roughly a triple increase on big files, so I thought I'd mention it.
-I added a catch all solution for cases where the program gets confused and wants to read strange data or the user opens a file of valid extension(such as BIN, a reasonably common extension) that is in fact not the kind of BIN from these games and thus contains no image data. In this case the application will present the error:
"Fatal error. Unable to find TMX signature. Exit in 5 seconds."
and properly terminate itself.
Important note: This error will happen when you reach the end of CVM files since I am still having trouble catching EOF on large files. So if you are viewing a CVM and get this message, you probably hit the end of file. I am working on this.
-I cleaned up the code as much as I could: Removed unused variables, fixed spacing and indenting, improved commenting etc.
Features new to 0.8
Major changes:
You can now directly open the CVM files without having to extract them with IsoBuster first!
Note: At the moment, this can be a bit slow. When you first load a CVM expect to wait 20+ seconds for the first image to appear. This is due to the large size of the CVM file (2 Gigabytes+ for some of them) and the fact it is being scanned per byte for the TMX0 signature. However, once you reach the first image and are using S/W keys to move around, the time in between images should be less than one second. If for some reason (I.E. you opened a 2GB CVM on a slow computer) you find it loading and loading and loading (there is a progress update on screen, so as long as it keeps updating the app is not locked), you can break out of the application with ESCAPE.
File name is no longer read from the container file as that was pretty sketchy anyway. It is now generated with this formula:
Container file name + position within container file + current date
Also, a feature was implemented that will prevent you from overwriting already exported files of the same name. If you push ENTER to save a PNG and get a RED message confirmation, the filenames clashed and you did not save the file. This would only happen if A - You already saved a PNG of that same file or B - If you did something unlikely, like opened the DATA.CVM from Persona 3, exported the 5th file, then opened the DATA.CVM from Persona 4 and tried to export the 5th file. Regardless, I would suggest you move your exported PNG files to somewhere other than the application directory after each use. For instance, if you extract a bunch of stuff from Persona 3, and then are going to extract a bunch of stuff from Persona 4, move the extracted Persona 3 stuff out of the main program directory before you move on to avoid clashes.
A quick searching feature is partially implemented to speed up reading of CVM files. Any TMX you have already navigated to within the CVM has its position stored (per session only!) so it is quicker when you navigate "backwards."
Minor Changes:
Should be faster, as files are no longer read from the start each time you move further in, the position is saved. This was necessary to allow efficient reading of large CVM files.
Changed the file index variable from a regular integer to an unsigned long integer. This was part of the reason I was having trouble reading the CVM files as their size was larger than the limit of the variable.
You can no longer select directories without valid files.
Instead of displaying a partial list of files a directory contains, you now either see the message:
"This directory contains readable files." in GREEN if the directory can be opened
or
"This directory does not contain readable files." in RED if the directory cannot be opened"
Known Bug:
I don't think the progress indicator that says "Reading byte X of Y" is correct, but really it's only there to assure the user the application is doing something and not locked.
Compatability:
Confirmed working with:
Persona 3: FES (confirmed by me)
Persona 3 Portable (confirmed by taleds)
Persona 4 (confirmed by me)
Disclaimer:
You use this at YOUR OWN RISK. The author disclaims any and all liability arising from any and all uses of this application.
License:
http://myriaddreams.com/index.php?optio ... 4&Itemid=9
Credits:
PersonaRipper for his help with some programming problems and providing me the source to his and bbrchers extractors which I converted to FreeBASIC code for large parts of the program
Simon Nash for FreeBASIC PNG Library and Jean-loup Gailly and Mark Adler for zlib library
taleds for the compatibility report for P3P
Request:
So far the TMX explorer has been successfully used with Persona 3, Persona 3:FES, Persona 3 Portable, and Persona 4. If you own another game which uses the TMX format and try this application, please let me know whether it works or not at my e-mail address which is in the about section of the program, or through a PM on this message board.
Screenshot:
Enjoy!
~Blyss
Download
Use the attachment or http://myriaddreams.com/projects/applic ... plorer.zip