[Spec] Version tag changes

[hexabits]

There are currently several issues with <version> that need to be addressed for various lib and scripting features.

Version Granularity

The primary issue is that there are more "versions" than <version> tags, when you include User Version and User Version 2 (which is actually more appropriately BS Version or BS Stream). Version should mean not just the 4 MAJ/MIN/REV/PATCH components, but a distinct NIF header for which all blocks are identical in layout.

This means spreading many existing versions across multiple elements, and adding several attributes to differentiate them.

Unique Identifier: id = <string>

Now that a <version> is a combination of multiple values, each version should have an id attribute. This ID must be unique and ideally can be used in code generation of enumerations. Thusly, a format was chosen similar to existing enumerations for NIF versions in the various projects.

ID must start with V and should only separate Version components with underscores. Currently, two underscores separate the Version and User Version when applicable. When a version is highly associated with only one game and vice versa, instead of using User Version or (User Version + BS Version), the ID can be suffixed with an abbreviation for the game, e.g. V20_2_0_7_FO4.

Note: Right now, the User Version is generally appended to the ID, but the BS Version is not. This is because multiple BS Versions are likely to be listed/grouped for one <version> and it's possible to lose/gain BS Versions as the version conditions are modified or simplified.

User Versions: user = <List[int]>

This is the User Version in the header. The default is assumed to be 0. However, User Version did not technically exist until after version 10.0.1.8. Despite this, there need not be any distinction between 0 and null for User Version. All versions shall report a User Version of 0 unless it is overridden in the attributes.

<List[int]> will be explained in Grouping of User and Bethesda versions

Bethesda Versions: bsver = <List[int]>

^{Alternative name: bsstream, named after the BSStreamHeader struct Bethesda added to the header serialization/deserialization which contains the stream version and export strings.}

The default is assumed to be 0. This is vital as it is used directly in version conditions to exclude non-Bethesda files or vice versa.

The Bethesda Version is almost entirely the only thing checked against in Bethesda's own blocks during de/serialization, and so it is actually more important than User Version, which is very rarely checked against for any NIF version.

Grouping of User and Bethesda versions

There are sets/ranges of User and Bethesda versions that are functionally equivalent to each other. They still need to be enumerated however, so this requires that user and bsver accept space-separated lists. Space-separated is the appropriate way to write lists in XML and should be maintained. (The comma-separated lists currently in nif.xml are technically not incorrect as it is actually a string used directly in code generation.)

Important: Hex-formatted integers must also be supported. The custom versions for Divinity 2 are easier to reference in hex (0x10000, 0x20000, 0x30000).

Example:

<version id="V20_2_0_7__11_7" num="20.2.0.7" user="11" bsver="27 28">Fallout 3, Fallout NV</version>
<version id="V20_2_0_7__11_8" num="20.2.0.7" user="11" bsver="30 31 32 33">Fallout 3, Fallout NV</version>
<version id="V20_2_0_7_FO3" num="20.2.0.7" user="11" bsver="34">Fallout 3, Fallout NV</version>

Custom Versions: custom = <boolean>

Some games have ignored User Version and instead created their own 4-component Version and left User Version at 0. The custom attribute set to true denotes that it is not an official Gamebryo version and has customization of one or more blocks. For versions with a user or bsver, custom="true" is implied and redundant to specify.

Even though this is listed under Version Granularity, two otherwise identical <version> but one with custom="true" and one with custom="false" should not be allowed at this time. There is generally no way to actually tell the difference by reading the header, etc. when a Custom Version uses the same 4-component Version number as an official Gamebryo Version. This does in fact happen in practice, and those custom versions are not currently supported by the XML.

Other Considerations

Aside from version granularity issues, there are other lacking version-related descriptors that the XML would benefit from having.

Supported status: supported = <boolean>

Right now this is designed to be true/false, but there is the possibility of it becoming more (tri-state, enum). The default is assumed to be true. If set to false this means that in some way or another, this version is not fully decoded and either some or all files of that version will not read/write correctly regardless of the software.

This attribute has many potential uses. Most generally, software/libs can decide to ignore unsupported versions. This is important for both code generation and runtime interpreters of the XML. There are currently 10 versions marked as unsupported and when ignored they all fairly drastically reduce the complexity of the blocks that those versions influence.

Custom file extensions ext = <List[string]>

There is no formalized association of NIF files with custom extensions and what versions use them. This will associate the custom extensions with their version(s) and this info can be used to automate the ability to acknowledge other file extensions than NIF/KF for software/libs.

Example:

<version id="V20_2_0_7_FO3" num="20.2.0.7" user="11" bsver="34" ext="rdt">Fallout 3, Fallout NV</version>
<version id="V20_2_0_7_SKY" num="20.2.0.7" user="12" bsver="83" ext="bto btr">Skyrim</version>
<version id="V20_3_0_9" num="20.3.0.9" user="0 0x10000" ext="nft item">Warhammer, Lazeska, Howling Sword, Bully SE, Divinity 2 (0x10000)</version>

Removal of hypothetical or unused versions

The listed versions should be concrete, real-world examples of NIFs found in the wild. There is at least one version that seems purely hypothetical (10.0.1.3) and doesn't even have any games listed for it.

The "demo/example" versions listed in the XML might also be better if they were dropped. For example: 20.3.0.1, 20.3.0.2, 20.3.0.3, 20.3.0.6 do not seem to actually appear in any real games. Their presence or absence do not really change anything at the moment, but upcoming features depend on keeping the number of versions low for complexity/performance/storage reasons.

Important: Versions used in attributes like vercond do not need to be in the list of versions. These versions are often purely hypothetical and are an artifact of the iterative development of Gamebryo.

Listed order must be maintained

The id attribute has use in generating enumerations and so the ordering of <version> has implicit value. The first version == 0, the second == 1 and so on.

Number of versions must stay under 65

The version count should stay under 65 at all costs, and under 54 if possible. A new feature will be encoding the applicable versions for every <add> into a 64-bit integer, i.e. a bitflag. To allow this to be used by as many languages as possible, it should stay inside of 64 bits. Ideally, it should try to stay within 53 bits, as some programming languages' underlying number type is double-precision floating point. In these languages only the integers up to 2⁵³ can be exactly represented.

Game List formatting

The element content for each <version> requires special formatting.

1. Inner element text must be comma-separated list of games
2. {{}} around a game name means this <version> is most important to that game, i.e. it should be considered the "primary" version for that game.
3. Each game must have one instance at most of {{}} around its name.
4. A <version> without any {{}} games can be considered "secondary". This is useful for determining the level of support for said version. For example, a <version> with only secondary usage might be ignored in UI for exporters or programs, or ignored for codegen.

Note: In the case of Oblivion, there are two "primary" versions.. 20.0.0.4 (10) is used for KF and 20.0.0.5 is used for NIF. Thus I changed the "game name" to {{Oblivion KF}} for 20.0.0.4 (10).

Example

Gist for Current WIP spec

For Discussion

Yes, FO3/FNV really have that many BS Versions. No, there's nothing to do about combining them into fewer versions, because the XML has been analyzed and the version splits are all made ideally. Any changes would result in a few files no longer loading correctly.

Epic Mickey 15/17 split is unconfirmed. The Divinity 2 0x10000/0x20000/0x30000 splits are unconfirmed.

There's not much for actual discussion other than my formatting choices for the id attribute. As I stated, the groupings of bsver could potentially change, either split up or lumped together. So putting a volatile set of integers into the ID seems like a poor choice. For User Versions, usually the ID just gets an identifier appended that references the game, like for Divinity 2.

The double underscore before the User Version when included is so that it doesn't look like part of the Version components.

[hexabits]

I have added definition of custom NIF file extensions to the proposed spec and updated the issue text.

Example of the changes in the WIP gist:

Current WIP revision

[hexabits]

I have also analyzed the Divinity 2 files and found that 0x10000 is functionally identical to a User Version of 0 and have merged the two, which reduced the versions by one and allowed me to simplify the ID for the custom Divinity 2 version. It now looks like:

<version id="V20_3_0_9" num="20.3.0.9" user="0 0x10000" ext="nft item">Warhammer, Lazeska, Howling Sword, Bully SE, Divinity 2 (0x10000)</version>
<version id="V20_3_0_9_DIV2" num="20.3.0.9" user="0x20000 0x30000" ext="item">Divinity 2</version>

I also actually decoded CStreamableAssetData in the process though that's off-topic. The only DIV2 file that errors is a character rig that is filled entirely with custom blocks that I will never decode. Excluding the XML files which are actually NIFs with only custom blocks holding text. They are also not within the scope of nif.xml in my opinion, as they do not contain geometry.

[hexabits]

As per the discussion in Discord, I have updated the specification for version tags to include that games listed in the element content for each <version> require special formatting.

1. Inner element text must be comma-separated list of games
2. {{}} around a game name means this <version> is most important to that game, i.e. it should be considered the "primary" version for that game.
3. Each game must have only one instance of {{}} around its name. Zero instances are allowed.
4. A <version> without any {{}} games can be considered "secondary". This is useful for determining the level of support for said version. For example, a <version> with only secondary usage might be ignored in UI for exporters or programs, or ignored for codegen.

Note: In the case of Oblivion, there are two "primary" versions.. 20.0.0.4 (10) is used for KF and 20.0.0.5 is used for NIF. Thus I changed the "game name" to {{Oblivion KF}} for 20.0.0.4 (10).

I have updated the WIP Gist in the ticket as well.

[hexabits]

Here is a visualization of what the {{}} should convey to any parser. For Python, it was quite simple to support this:

# Example Usage
class Version(object):
    # Primary Version for Game
    primary = r"(?:[{]{2})([^{,}]+)(?:[}]{2})"
    re_prim = re.compile(primary)
    game_all = r"[^{ ,}]{2}[^{,}]+[^{,}]{2}"
    re_all = re.compile(game_all)

    # __init__ not shown

    def all_games(self):
        """All games listed for this version"""
        return self.re_all.findall(self.description)

    def primary_games(self):
        """Primary games listed for this version"""
        return self.re_prim.findall(self.description)

For ver=V20_3_0_9 it would output

ver.all_games()
=> ['Bully SE', 'Warhammer', 'Lazeska', 'Howling Sword', 'Ragnarok Online 2', 'Divinity 2 (0x10000)']
ver.primary_games()
=> ['Bully SE']

This was done in alternative to creating an entire reverse mapping for game->version.