Game Pass save support

[snoozbuster]

tl;dr - if you're here to ask for support, thumbs up this issue. If you're interested in gory technical details, read on.

I've spent some time the last few days reverse-engineering the save format that the XGP version of No Man's Sky uses. Since the code for this tool isn't open source, I can't open a PR to add support myself, so instead I'll describe the file formats as best I can in the hopes that @goatfungus can use it to add the support relatively easily.

Part 1: Microsoft's piece

Since the XGP version of NMS is a UWP app, it saves the games using some MS SDK, presumably an Xbox one to enable cloud saves. These saves are stored at

%LocalAppData%\Packages\HelloGames.NoMansSky_bs190hzg1sesy\SystemAppData\wgs\[XYZ]_29070100B936489ABCE8B9AF3980429C\

where XYZ is some long hexadecimal string which appears to be unique per person. The number after the underscore, however, appears to be the same across everyone.

Inside this directory are a lot more folders with all-hex names. At its root is a containers.index file which can be read to understand the save slots and which folders they live in.

I've been able to identify these sections of the file so far:

container index

All offsets are in reference to this particular file (since there's a lot of variable width strings in here). This file holds the save slots - for any given save folder on disk, if it's not in this index file, it won't show up in game.

Offset	Color	Type	Description
$0000	N/A	int32	I think this is a version number - it's always been 0xE for me.
$0004	dark red	int32 or int64	We'll refer to this later as numSaves; it's the number of save files (including autosaves). I'm not sure if this number is a 32-bit or 64-bit one; the extra 4 bytes were always 0 for me.
$000C	blue	wchar[int32]	An int32 denoting the string length, followed by that many 16-bit characters. It's always HelloGames.NoMansSky_bsl90hzglsesy!NoMansSky, which is the UWP package name.
$0066	N/A	?	I'm not sure what these next 12 bytes are. The first 8 are all over the place and change pretty often, and the last 4 are usually a small int32 (in this file it's 1, but I don't know if it was always that for this game).
$0074	blue	wchar[int32]	Same format as the previous string, but is always a512e4cc-80fd-4c57-a96d-15166f42a9f0. for me. It's not the same for other people though.
$00C0	N/A	?	Not sure what these 8 bytes are for. I don't think I ever saw them vary from 00 00 00 10 00 00 00 00, even across people.
$00C8	orange	wchar[int32]	This is the start of the first save. There are two copies of the same string in the same format as above which describe the save (in the format SlotNAuto or SlotNManual, for auto and manual saves respectively).
$00F5	red	wchar[int32]	This is a double-quoted representation of a int32 printed in hex (in this file, "0x8D83E8BC3970A7F"). I have a theory that this is the memory address that the save was at when it was written to disk, but I don't know what it would be used for. Sometimes it's a 0-length string, as well.
$011E	purple	byte	We'll refer to this (much) later as containerNum. It's used to lookup this save file in one of the hex-string folders we saw earlier.
$011F	N/A	int32	Not sure what this is - it's always 1. I could see it being the number of filenames to read, though.
$0124	purple	Filename	This is which folder to look in to find this save file. It's written in a special "filename format" that I'll describe in the next section.
$0134	black	?	Similar to the 12-byte section above, I don't know what this 24-byte section means. Some parts of it tend to be similar for similar files though, could be a last modified date (likely used in cloud storage conflict resolution if so). The chunks of 0s are pretty much always 0s, though. It doesn't seem to be required for reading or writing.
$014B	green	SaveSlot	The start of the next save slot. Start back with the orange strings and repeat numSaves - 1 times. There is a final null terminating byte at the end of the file.

"Filename format"

The folder name for each save is stored as [int32, int16, int16, int16, int8, int8, int8, int8, int8, int8]. Parse each of these integers (little endian), then convert them to hex strings in big endian format and join them to get the folder name for the save slot. As an example, when this is applied to the purple section highlighted above, it looks like:

BCF1BA8EE22EC944B17D14DB4EDDE929             # raw bytes
BCF1BA8E E22E C944 B17D 14 DB 4E DD E9 29    # split into int groups
8EBAF1BC 2EE2 44C9 7DB1 14 DB 4E DD E9 29    # rotated from littleE to bigE
8EBAF1BC2EE244C97DB114DB4EDDE929             # rejoined as folder name

As we can see, we end up with the second folder name from the first screenshot. Now we can open that folder to see what's in the "Slot1Auto" save.

Inside this folder we have three more files. This is where the containerNum from before comes in - we need to open container.[containerNum] in this folder to figure out which of these two holds the save file data and save file meta. (Of course, the meta is the tiny one, and the save data is the big one, so you could probably skip parsing this file and use that assumption intead.)

container contents

This file is a little confusing to me.

I won't color this one, cause it's easy enough to look at. There's two int32s - I don't know what these mean here. In all my saves, they're 4 and 2 respectively. Then, "data" as a wchar[], followed by a bunch of 0s and then two copies of the "data" filename in the Filename format described above. Immediately after those two, there's "meta" followed by more 0s and the "meta" filename (again twice).

I know that at one point I had one of these containers that only had the filename entries once each for data and meta - but none of them are like that anymore, and I don't remember if it changed the first two bytes. The filenames here might be constant length wchar[160]s as well - they do appear to be controlled by the game, because once we get to those files we are out of Microsoft-land and into the NMS part of it.

Part 2: NMS save files

The save is broken up into two pieces, the "data" and the "meta". Part 1 described how to figure out which was which, but what are these pieces?

The meta file

The meta file is extremely small:

It turns out, the game uses this to display the summary of the save file on the save select screen without needing to load the whole save data. The different pieces are:

Offset	Description
$0000	(int32) I think this is a version or magic byte - it was 0x101E in every file I opened (even from other saves I downloaded).
$0004	(int32) The difficulty displayed on the save select screen (0x01 for Normal, 0x03 for Survival, 0x05 for Permadeath, and presumably 0x07 for Ambient, although I didn't try it. As far as I'm aware, this doesn't need to match the save data itself.
$0008	There are two int64s (maybe?), one of which is likely for the play time counter and the other which is the last save date. I'm not exactly sure what format or units these are in though.

The data file

And now, the final piece: I think the "data" section is a totally normal NMS PC save file.

(truncated for length)

Parts of the JSON structure are visible with mangled key names. However, parts of this file are garbled entirely - I'm not sure if normal NMS saves have this weird psuedo-encryption applied to them as well, although I could see from other tools that something like this was at least present in some older versions of NMS. If not, it's possible that a different piece of the "meta" or container.index files could be the key - the encryption kinda looks like it's XOR-based with a key that has large chunks of 0s in it.

I was hoping I could load this file directly into NMSSE, but since it doesn't seem to support loading arbitrary files and I don't actually know what a normal NMS save looks like on disc, I couldn't try to copy to the normal NMS format and location to try it. My hope is that it's a totally normal NMS save, and this extra information about the container file formats can be used to auto-detect and load the different save slots into the editor, just like for the Steam version.

Last notes

With this information, I was able to:

• Change the containerNum in containers.index and the corresponding file suffix on disc, and the game still recognized and loaded the save
• Splice in save files from other people, or make copies of them, by copying their data/meta files to a "host" container and renaming them to replace the existing ones
  • The fact that I was able to do that with a save posted on github by another person implies there's no user-specific encryption or checksums
• Manually change the difficulty in both the save data (since it is right at the beginning and part of the unencrypted JSON) and meta to create a copy of a save on another difficulty

Interestingly, when I mixed a "meta" from one save and a "data" from another, the save select screen displayed the correct metadata for the corresponding save, but loading up the save actually just acted as if I was starting a new game (white Initialize screen). It's possible that the meta and data files are tied together in another way I don't understand (maybe that file holds the encryption key?).

Anyway, that's all I've found. Even if this doesn't help get XGP support into NMSSE, I hope it helps some other poor shmuck with a hex editor tweak their file (all I wanted when I started this was to change the difficulty of my save so I could play with a friend).

Attachments

Here are my saves for reference:
NMS_XGP_saves.zip

There's four saves in this zip:

• A permadeath autosave with 3 minutes on it
• A survival save with ~50 hours on it
  • Plus its autosave
• A copy of the survival save converted to Normal difficulty. This save was created by inserting its files into a different Normal-difficulty save that I used as a host.

[jpeaglesandkatz]

Spent a day trying to do something with these files as well.. Think you can pretty much forget it.. If they bothered to compress the files I'm sure they also encrypted them. I see references to AES etc in the loggings... Looking how well tight shut everything around store apps is and the inability to mod anything, I'm pretty sure MS closed down this things as well.. Good luck trying though.

[cengelha]

Figured out the container stuff on my own for my own save editor. The data and meta files are different from Steam and that's the reason they cannot be loaded in any editor. For Steam the data file is completely plaintext. So, the most crucial part to support the Windows Store is to decipher the data (and meta?) file into plaintext (and back)

Edit: As far as I know for the saves it's completely up to the devs what they do with it. But as far as my own research goes, it seems to be the same encryption/compression as for the PS4

[snoozbuster]

@SubsterNL since I was able to take someone else's save and load it into my game, there's no MS-specific encryption. I was actually pretty surprised by this, I was expected them to be locked down as well but it doesn't seem like that's the case.

@cengelha do you have any idea what that compression is? have you gotten ahold of any compressed PS4 saves, and do they look like the XGP ones (chunk of uncompressed text interleaved with chunks of compressed text)?

[cengelha]

@snoozbuster Unfortunately no idea. Tried a few things but I'm not expert in it... About the PS4 saves: I have a few from SaveWizard (which includes plaintext JSON) but if you look in this issue (#258) it looks like it's not the default and the output of the PS4 Savemounter contains something that looks a lot like what we have here. Another clue for me are the PS4 meta. Couldn't figure out one entry but I assume it's the compressed size of the JSON and roughly the same as for the Windows Store for a freshly started game.

[snoozbuster]

@cengelha if the meta files are the same 24-byte files that I described in the post, I think it's playtime and save date in there. I think that file is used to display the info on the save select screen (if you swap out the meta file from another save, it shows the other's save's difficulty, playtime, and save date).

Now, it might also be used in the decompression process, since the game will fail to load the save data if the meta and data don't match. but as long as they match, the game will load the save successfully, even if you got it from another container and renamed the hex files on disc to match - so there's nothing in the container files that's used on the save data itself. I'd love it if you could attach some of those PS4 saves you have - I'm definitely interested in seeing how similar they are.

[cengelha]

@snoozbuster Sorry. Forgot to mention that the meta differs on all 3 platforms.

As the PS4 saves are not mine I will ask if it's okay to post them. But it's one big file that contains a byte header with a size of about 4kb including the meta data among other. The meta data have a size of 48 byte for each save and the most interesting part here are the last save timestamp and the size if the JSON (data) and its offset in the file. After that header the JSON for each save is appended with a separator string between them.

The meta for Steam is a 104 byte file but only 56 byte are used. It contains a SpookyHash and a SHA256 of the data file. If they don't match, the save cannot be loaded. Timestamp here is the last write time of the data file itself.

Sounds reasonable what you wrote about the Windows Store meta. I will have a look after work. But even if it's a part of the decompession/decryption, there is still the question what algorithm is used...

[cengelha]

@snoozbuster Ran my tests and here's the result (converted it to an int32 array):
I don't know if the last two are int32 or int64 but both following those are always 0.

1. Fixed value. Similar to other platforms.
2. Difficulty sounds good
3. Total Play Time
4. Not the save timestamp. Might be again the size of the decompressed JSON and probably the one that prevents the loading if it doesn't match

Save timestamp is probably like Steam the last write of the file itself.

508A379C7D584F84B09CF76A443F9BAE [4126, 5, 253, 0, 929699, 0]
34FDFB7876064464B178A098331A41A4 [4126, 5, 258, 0, 931734, 0]
B98371FDCD89449091ECDF05638146BD [4126, 5, 268, 0, 931732, 0]
609674886EE8405B93A83E2B225D7FB1 [4126, 5, 272, 0, 931731, 0]
40F2E6C686954FB3828897F50783AD56 [4126, 5, 276, 0, 931735, 0]
314009A984CC4E0B9B7A2354B4E46DC6 [4126, 5, 286, 0, 932978, 0]

It's a fresh save, just ran to the ship and saved. As you see the last value doesn't grow with the play time. Don't know what the first jump in the last value was but for the second bigger jump I interacted with the emergency beacon at the crash site.

[snoozbuster]

@cengelha fascinating... if the last number is the size of the decompressed JSON, are fresh saves on Steam usually 900kb? cause that seems awfully large. that last number is definitely mysterious though... When I mixed a meta file from one save with a data file from another it wouldn't load the data file anymore. what if we mix the meta file from one save with a data file from another, but transfer the last 8 bytes from the matching meta file as well? (basically just changing the first 16 bytes that we are pretty sure we understand). if doing that successfully loads the game then that could point to that last number being used in the decompression or validation of the save data.

[cengelha]

@snoozbuster Mixing the meta data is definitely worth a try.
Last time I tried to see what a minimum file size could be it was already above 700kb and that was some updates ago and since then they added even more to it. So, 900kb is definitely possible. The reason it's so big is, that most of the structure of the file is always there. Then it gets bigger by putting stuff in lists (base building, inventories, discoveries, etc). My main save is over 4,5mb

[cengelha]

@snoozbuster Good news! Mixing the meta data works, btw, but I also got the algorithm for the compression and have the first step already running.
But at one point you might help me (I will have another look as well): I couldn't figure out where the timestamp for the last save is, so the game always shows the timestamp of the last save by the game itself. It doesn't seem to be in the files itself (including container) nor in the file attributes (Access, Write, Creation). Maybe you have an idea?

Edit: Found.

[cengelha]

In case you haven't noticed: I finished it for my editor and you can use it now with any platform you like

[runthis]

@cengelha Thank you! -- For anyone wanting the link: https://github.com/cengelha/NomNom/

[runthis]

As an aside, since it is pretty new, I recommend to anyone to ensure you back up your wgs folder. This is a common action to take on the state of decay 2 community editor "just in case".

[goatfungus]

MS Game Pass is now supported in NMSSaveEditor.

[Acru]

Just a note in regards to the container index file, "$00F5 (red) wchar[int32]" may be a timestamp, for cloud syncing purposes.