Create a migration scheme for Godot Engine resources

[fire]

Describe the project you are working on

We are making a 3d social vr online game.

Describe the problem or limitation you are having in your project

We want to keep distributed scenes updated and migrate to the latest as soon as possible. This solves making it easier to upgrade godot engine scenes and scanning for security problems.

@SaracenOne was worried about code improvements when we do a possible compatibility breakage like in godotengine/godot#65312 where an enumeration had an entry removed and then a new entry was added in the same spot.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

We want to use the original resource. Extract the mapping of property name to Variant. Run the migration by taking Map<StringName, Variant>, the two pairs of versions, and migrations. Save the properties into a MissingResource. On load the MissingResource will convert to a resource.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

1. Resource -> MissingResource-> Resource https://docs.godotengine.org/en/latest/classes/class_missingresource.html
2. For each Resource property: get_property_list(), get(), set() are migrated. original_class and is_recording_properties properties exist and are set in the MissingResource.
3. MissingResource can be checked easily for security and complexity explosions via a conversion to HashMap<StringName, Variant> and because MissingResource has less implementation to verify.
4. In HashMap format check for problems and run schema migrations. (Note that you can convert the HashMap to json in a straightforward manner.)
5. When you load a MissingResource it will auto-convert to the original class.

Reference:

• Implement missing Node & Resource placeholders godot#60597
• https://github.com/godotengine/godot/blob/f5978f4d97150d9e7a1598f9637f0f3c06d72d33/core/io/missing_resource.h
• https://github.com/bruvzg/gdsdecomp/blob/master/compat/resource_loader_compat.cpp

Contributors:

• @fire
• @nikitalita
• @lyuma

If this enhancement will not be used often, can it be worked around with a few lines of script?

We need to store the migration in the engine.

Is there a reason why this should be core and not an add-on in the asset library?

The engine developers need to create a migration for any compatibility breaking changes.

[Creta5164]

I agree that this feature is needed, my current situation is different, but I'm migrating in the following way for now.

Ignore the part about casting to Variant, it works just as well with the sprites property.

Even when I was using Unity, most of the time to do such migrations, I had to create two different types of “old” and “new” properties, migrate them to the new ones, and then replace the “new” ones with the “old” ones again after a period of time.
Personally, I think this is pretty annoying, and if the feature is built as the author says it will be, I think it will be quite useful to solve this annoyance.

[fire]

The current Godot Engine contributor consensus is to avoid creating a migration database and work on forward-compatible properties as needed.

I still favour the migration database approach.

What is a forward-compatible property?

A forward-compatible property is a pattern where we create a boolean property whose existing behaviour happens when the property is false. The new behaviour occurs when the boolean is true. So when a new instance is created, it uses the new behaviour while the old behaviour is kept when the boolean property is false.

[Creta5164]

I'm not sure I'm qualified to comment here, but if the workaround is to simply handle the old/current behavior with just one bool property, I wonder if we'd have the following issue :

• Change property to different type
• Additional properties for migration

So I'm thinking, as you suggested in your suggestion, to import the existing resource into an intangible instance called MissingResource and resolve its contents.
(I was thinking get them into varint dictionary)
Please correct me if my understanding of the concept is wrong.

I'm also curious how other people (especially add-on and core engine developers) deal with migrations in these situations, I've only been writing Godot for a short time, so I'm afraid that I may not even realize that there is a much better way to deal with these migration issues.

[fire]

Both MissingResource and MissingNode already exist.

https://docs.godotengine.org/en/stable/classes/class_missingresource.

https://docs.godotengine.org/en/stable/classes/class_missingnode.html

Although the team has yet to agree to this proposal of creating a change-by-change migration, it is the best solution, as far as I know.

This proposal is in limbo.