Make saves backward- and forward- compatable.

[kevingranade]

This is a really long-term issue. Currently data is pushed out to the save file by exporting the value of various enums, and some data is position-dependent, e.g. a certain point in the save file there is a list of skill values, and which values is associated with which skill depends on its order in the list. Similarly items are stored as the value of their enum, and if the enum changes between builds items will be read back from the save as a different item.

In order to make saves backward and forward compatible, all data in the save file must be labelled or positioned in a way that is stable across builds. The simplest way to do this is to label each value with a string indicating its type, but this might be prohibitive for some aspects of the save, such as map data. Since serialization is handled separately by each class, each class can be updated independently to handle this so we can slowly proceed toward supporting full save compatibility.

In addition, to make saves backwards-compatible, the code reading in the save will have to either ignore or interpolate unknown values. So with a new skill, the code would discard unknown skills, but with e.g. guns, the game might substitute a different gun that exists in the current version.

[aawilson]

It seems like handling this for real would require kind of a "migrations" file for versions that keeps note of save file format changes. One consideration for making that easier might be to think about going into a lightweight database system (like sqlite) to let it handle most of the garbage around labeling and positioning data.

[ZwodahS]

actually making it backward compatible is not important. Most player will want to play a new updated version. Making it forward compatible is more important, so new version don't break the save that the player already have.

[aawilson]

I think you are confused. Do you mean to say that rather than writing future versions of the executable to accept older versions of the save files, the save file format should be written in such a way that it can handle all versions of the executable? That would be both insanely difficult and limiting compared to the other way around, since it's impossible to see into the future to know what requirements a particular version will require of its save files, but it's (relatively) easy to look into the past to see what needs to be made up for for an old save file to work with a new executable. The only benefit of trying to design for forward-compatibility would be for someone to use a new savefile with an old executable, which is hardly an interesting use case.

[ZwodahS]

no. I probably phrase it in a wrong way. I meant that all executable should be able to handle all past save files.
It depends on how you see backward compatible. I am saying that savefile being backward compatible is not necessary, as in, the save file being compatible with older executable. However, if you see it in that the executable should be backward compatible, then yes, that is what i meant.

[aawilson]

Haha, yeah, the English language is terrible like that

[kevingranade]

Foward-compatability is nice because it means people can try out new versions with their existing saves, and then fall back to a previous version if there is some problem. There are various ways to support this, but what I'm mostly thinking of is that if a particular entity in a save is not recognized, the game substitutes a valid fallback. The key really is that future content might be lost, but the game as a whole would still be playable.

As I outlined in the original issue, it requires two things. The format of the save file must to be locked down, and the values used for entities within the save file must be stable. We can achieve the second by writing out unique strings of some kind rather than raw integer values like we do now.

I'm not super familiar with integrating programs with sqlite, but I'm skeptical that doing so will solve our labelling problems. It seems very likely that they'll just move to the internals of a SQL statement instead of the source code.

[ZwodahS]

ambiguity in language is what makes NLP hard :P.

Anyway, I doubt SQLite will do much either, since it is just a way to read/write data. Like what kevin say, it is just about how we want to handle it in code. I think forward compatibility will be useful if we are releasing daily snapshot.

But here is the problem, how can a old exe know what a string means and find a replacement for it ? Like say we want to add chicken into the game, so we may have MRE - chicken. Due to problem, we want to change all MRE - chicken to MRE - beef, but the old exe do not have MRE - chicken in the unique string, let alone knowing what that means.

[jjs2]

Here's how to do it:

Have a table of integer-string mappings...

e.g. 0: "nothing", 1: "door", 2: "door/locked", etc....

You can generate this mapping during program startup as well... every time you load a new symbolic json id for an object or creature or terrain type (or when creating the first instance of a compound object or terrain type, e.g. "arrow/fletched+field_tipped+flammable" or "bear/hibernating" or "door/locked+boarded"), you append it to the in-memory table. (e.g. a vector of strings. but you also want a hashmap for the reverse lookups...)

Then when writing the save file, you just serialize this table, and then spit out integers as before. When loading a save, first create an integer->integer mapping from the old save file's integer ids to the currently running program's integer ids for the same keys.

[kevingranade]

We're maintaining backwards compat from 0.8 until further notice, but forward compat is still a bit off. Should have made two issues.

[kevingranade]

This is just baked into the system and development as a whole. We're not totally there yet, but such a broad issue isn't productive.