Join also our Discord channel! Click here.

Havok binary files

Post questions about game models here, or help out others!
HunterAP
advanced
Posts: 40
Joined: Mon Aug 10, 2015 7:03 am
Has thanked: 14 times
Been thanked: 5 times

Re: Havok binary files

Post by HunterAP » Wed May 18, 2016 3:09 am

volfin wrote:
HunterAP wrote:How did you manage to convert from the HKX bone coordinates and rotations to SMD format?
If I could find a way to batch the process, I could get full DSIII animations out.
This is a project I started last year for Alien:Isolation, it's a HKX converter that outputs SMD skeleton. It works with 2012.2.0 and 2010.2.0, I had intended to take it all the way with support for many versions, and animation output, but I simply moved on to other projects. It's written in C#, a lot of the file structure is defined at the top of Program.cs, and maybe will help people, so I'm making the code public.

https://bitbucket.org/Volfin/hkx2smd/overview
I tried building your program as both x86 and x64, normally running the 2014.1.0-r1 HKX files give some errors in the header, but if I copy-paste the 2012.2.0-r1 header over the original, the program reads everything and gets up to "class hkaskeleton signature: 0xFEC1CEDB", then after a few seconds the program crashes with the "System.OutOfMemoryException" error. Looking it up says that I don;t have enough memory, even though I have 16GB of RAM, but I'm assuming that your program is looking for a big chunk of sequential memory that I may not have if my memory usage is fragmented.

What's interesting is that I've tried this with multiple skeletons, and it doesn't matter what size the HKX file is, I still get the error.

I would fix the issue myself, but I'm not versed in C#, so I don't know how to fix it in the code myself.

If you'd like, I could just send you a HKX sample of the skeleton and animations.

User avatar
volfin
ultra-veteran
ultra-veteran
Posts: 447
Joined: Sun Jul 06, 2014 6:30 am
Has thanked: 107 times
Been thanked: 298 times

Re: Havok binary files

Post by volfin » Wed May 18, 2016 5:46 am

Well the point of me releasing it is that I have no time to mess with it anymore (haven't touched it since Alien:Isolation). I know it worked for Alien: Isolation and Skyrim, but don't guarantee anything else. The problem is HKX is very customizable so the format varies widely from game to game, and then on top of that many developers modify it in-house to do even more strange things, and some even obfuscate it purposely to prevent people reverse engineering their animations.

So likely the code will have to be tweaked for every particular game, no small job. That's why I gave the code out. :)

philgrf
ultra-n00b
Posts: 5
Joined: Mon Mar 06, 2017 6:11 pm
Been thanked: 3 times

Re: Havok binary files

Post by philgrf » Mon Mar 06, 2017 6:52 pm

As I understand, the spline compression is a technic that is animation specific.

See this article at riot game engineering
https://engineering.riotgames.com/news/ ... ation-data
So, it is likely there is no simple link between the hkx file and the animation keys.

Maybe the way hkx animation is encoded is similar to the one described in the article that is to say :
- rotation is encoded as quaternion (x,y,z,w),
- the highest value among (x,y,z,w) is discarded as x^2+y^2+z^2+w^2=1 and can thus be reconstructed from the 3 others,
- bytes need to be read according to a key (m bytes for discarded value, n bytes for remaining value1, n bytes for remaining value2, n bytes for remaining value3)

edit : this article at bitsquid describes also the technic
http://bitsquid.blogspot.fr/2009_11_01_archive.html

Best regards

Phil

SergeantJoe
mega-veteran
mega-veteran
Posts: 239
Joined: Fri Aug 30, 2013 5:39 am
Has thanked: 56 times
Been thanked: 20 times

Re: Havok binary files

Post by SergeantJoe » Fri Mar 10, 2017 6:52 pm

The SDK also has a LOT of documentation files in it, including files for all the compression types instead of just Spline. It's not available through Intel anymore but I managed to save copies. Feel free to PM me for the link.

philgrf
ultra-n00b
Posts: 5
Joined: Mon Mar 06, 2017 6:11 pm
Been thanked: 3 times

Re: Havok binary files

Post by philgrf » Sun Mar 12, 2017 11:26 am

Sergeant Joe,

I thank you for your reminding of the Intel documentation.
I have downloaded havok 2013 before Intel closed the public access.
At the time I downloaded, I couldn't make any sense of spline compression.

Now, I understand better :
After spending some time on xentax forum, I started to get acquainted with binary files ans lately with file compression.
Following your advice, I looked again at havok 2013 help files.

The game I'm interested in is using spline compression.
So, I restricted my field of research the spline compression.

There are havok default settings for spline compression :
Position : 16 bits
Rotation : 40 bits
Scale : 16 bits
Float (what for ? timeline ?) : 16 bits
Frames per block : 256

As the spline compression is highly customizable(*) in havok, this should only be considered as a reference.
At least, it gives the structure of the data.

I am a beginner at reading binary files but it looks like an interesting challenge :

- identify blocks,
- identify number of frame per blocks,
- guess bone ordering,
- guess quantization for each bone,
- translate into skeleton pose,
- translate into animation.

I need to evaluate the required time.

Best regards

Phil

*() : Even, a per bone customization is possible. The doc says :
"Judiciously setting compression parameters per bone can result in a significant reduction in output data size. One such use case is to create level of detail compression by grouping bones roughly by depth in the hierarchy; tight tolerances could be user for "major bones" such as spine and arm bones while more relaxed tolerances for "minor bones" such as hands and fingers could be used to gain additional savings. A similar use case might involve grouping bones as "upper body" and "lower body". Extremities such as hands often contain a significant number of bones though their visual impact may be limited."

BloatheadSorcerer
n00b
Posts: 15
Joined: Thu Jan 15, 2015 10:10 pm
Has thanked: 1 time

Re: Havok binary files

Post by BloatheadSorcerer » Mon Apr 03, 2017 7:40 pm

Does anyone have a working link for the script that flips the skeleton or model because when I open animation in 3dsmax the skeleton has the right animation but the model is all over the place.

zweihard
ultra-n00b
Posts: 9
Joined: Fri Mar 24, 2017 12:18 am

Re: Havok binary files

Post by zweihard » Mon Jul 31, 2017 11:52 pm

Would anyone mind helping me out with obtaining the animation data for Dark Souls III?

Regards

SergeantJoe
mega-veteran
mega-veteran
Posts: 239
Joined: Fri Aug 30, 2013 5:39 am
Has thanked: 56 times
Been thanked: 20 times

Re: Havok binary files

Post by SergeantJoe » Tue Aug 01, 2017 12:49 pm

zweihard wrote:Would anyone mind helping me out with obtaining the animation data for Dark Souls III?
Sorry, you're out of luck. Havok formats are incredibly difficult, even with all the documentation nobody has been able to figure it out.

zweihard
ultra-n00b
Posts: 9
Joined: Fri Mar 24, 2017 12:18 am

Re: Havok binary files

Post by zweihard » Tue Aug 01, 2017 9:23 pm

SergeantJoe wrote:
zweihard wrote:Would anyone mind helping me out with obtaining the animation data for Dark Souls III?
Sorry, you're out of luck. Havok formats are incredibly difficult, even with all the documentation nobody has been able to figure it out.
Reverse engineering job huh?

Ginsor
n00b
Posts: 13
Joined: Thu Nov 23, 2017 10:46 am
Has thanked: 2 times

Re: Havok binary files

Post by Ginsor » Fri Nov 24, 2017 10:25 pm

Sorry for bringing up this topic again, but Im kinda stuck with a havok file here.

I got a havok binary with the hk_2012.2.0-r1 header. However, none of the converters seem to work with it (even when renaming the header) and the original havok preview tool says something like "wrong platform".

Anyone has an idea of what is so special/wrong about that file?

hk_2012.2.0-r1 binary

SergeantJoe
mega-veteran
mega-veteran
Posts: 239
Joined: Fri Aug 30, 2013 5:39 am
Has thanked: 56 times
Been thanked: 20 times

Re: Havok binary files

Post by SergeantJoe » Tue Nov 28, 2017 12:01 pm

I honestly have no idea, and there's nobody else that can help. Sorry, you just have to give up.

Ginsor
n00b
Posts: 13
Joined: Thu Nov 23, 2017 10:46 am
Has thanked: 2 times

Re: Havok binary files

Post by Ginsor » Tue Nov 28, 2017 12:15 pm

SergeantJoe wrote:I honestly have no idea, and there's nobody else that can help. Sorry, you just have to give up.

well, there are some people that made the converters, so there should be some other help out there...in theory haha
however, thanks for looking into it!

User avatar
Loomy
beginner
Posts: 37
Joined: Sat Sep 12, 2015 2:16 am
Location: South Africa
Has thanked: 19 times
Been thanked: 6 times

Re: Havok binary files

Post by Loomy » Sat Dec 09, 2017 4:51 pm

Ginsor wrote:Sorry for bringing up this topic again, but Im kinda stuck with a havok file here.

I got a havok binary with the hk_2012.2.0-r1 header. However, none of the converters seem to work with it (even when renaming the header) and the original havok preview tool says something like "wrong platform".

Anyone has an idea of what is so special/wrong about that file?

hk_2012.2.0-r1 binary

Been a while since i tried my luck figuring this out, but just if it helps: look out for the c# project shared by the awesome (if not mistaken) SeargentJoe, (hkx2smd, and there is another one, or similar project on github, with some fiddling managed to parse some basic 2013 structs, im looking to research splinecompressed anims & bone/weight/blendweight conversion. though quite difficult to find older/recent havok src. (looking for 2011r3 & 2013+)

Just off top of my head, and old snippet idk if works:- look for this flag in ur data which specifies platform type and modify to ur build platform. also thinking might be some endianness conversion needed ( ? )

public enum hkPlatform
{
HCL_PLATFORM_WIN32 = 0x0,
HCL_PLATFORM_X64 = 0x1,
HCL_PLATFORM_MACPPC = 0x2,
HCL_PLATFORM_IOS = 0x4,
HCL_PLATFORM_MAC386 = 0x8,
HCL_PLATFORM_PS3 = 0x10,
HCL_PLATFORM_XBOX360 = 0x20,
HCL_PLATFORM_WII = 0x40,
HCL_PLATFORM_LRB = 0x80,
HCL_PLATFORM_LINUX = 0x100,
HCL_PLATFORM_NGP = 0x400,
HCL_PLATFORM_ANDROID = 0x800,
HCL_PLATFORM_CTR = 0x1000//4096
}
if u look up my old posts, io posted 010 template for havok while back, somewhere on forum, including offset of Platform flag (i didnt have much joy parsing PS3 files on win32 build)

HunterAP
advanced
Posts: 40
Joined: Mon Aug 10, 2015 7:03 am
Has thanked: 14 times
Been thanked: 5 times

Re: Havok binary files

Post by HunterAP » Sun Dec 10, 2017 3:19 am

After all this time I'm also trying to get some newer formats of HKX working. Right now, Volfin's hkt2smd is crashing because newer formats report that for a variable used in the code, there is over 1.5 billion elements in an array (AKA way too much to work with), and I'm not sure how to go about patching it.

Also, interestingly, trying to use hkt2smd on Dark Souls 1 has a similar error but in a different section.

User avatar
Loomy
beginner
Posts: 37
Joined: Sat Sep 12, 2015 2:16 am
Location: South Africa
Has thanked: 19 times
Been thanked: 6 times

Re: Havok binary files

Post by Loomy » Sat Dec 16, 2017 8:37 am

HunterAP wrote:After all this time I'm also trying to get some newer formats of HKX working. Right now, Volfin's hkt2smd is crashing because newer formats report that for a variable used in the code, there is over 1.5 billion elements in an array (AKA way too much to work with), and I'm not sure how to go about patching it.

Also, interestingly, trying to use hkt2smd on Dark Souls 1 has a similar error but in a different section.
can u post here sample of both assets ur trying to parse ? i can take a look.

Post Reply