Improve Migration Support

[apopiak]

Teams building chains on Substrate are struggling with migrating their nodes and runtimes. We want to improve the migration experience to reduce necessary effort and time spent as well as reduce the amount of effort it takes to migrate Substrate based chains.

💔 Status Quo

Currently Parity core developers write migrations in Substrate for Kusama and Polkadot. They are then removed within one to a few weeks of being introduced (in order to keep the codebase clean). Chain developers building on Substrate either a) apply these migrations the same way in a timely fashion or b) need to fork Substrate to modify and adjust the migrations or reintroduce them if they have been removed in the meantime.

Substrate users need to go into Substrate's PR listing and filter for runtime migrations in order to find the migrations.

🛑 Current Problems

• Users are confused about migrations:
  • where to find them (different locations, formatting, buried in history)
  • when to migrate (on new Substrate release? at a fixed interval?)
  • "Will this upgrade work for my chain?"
• Users need to do lots of manual work for migrations.
  • Especially when deciding to upgrade later than Kusama the manual work compounds with the confusion.
• Users have trouble testing migrations.
• Users cannot roll back migrations.

👓 Vision

We want Substrate users to be able to migrate their chains correctly as easily as possible.

As a result, we want migrations to be:

• Correct: Migrations should leave the chain in a correct state after the migration.
  • Testable: It should be possible to determine whether they run correctly.
  • Composable: Running several migrations one after another should not corrupt state.
• Discoverable: Migrations should be discoverable when users want to apply them or use them as inspiration for their own migration.
• Documented: Migrations should list assumptions so that users can verify whether they will work for their chain.
• Reusable: Migrations (especially simple ones) should be reusable so they can be applied to many chains and don't need to be rewritten.
  • Easy and Automatic: Simple migrations should just work™ without a lot of manual integration.

❎ Non-Goals

Complicated Migrations ≠ Easy and Automatic

We cannot make complicated migrations or migrations affecting many parts of the runtime effortless. There will always be manual work necessary for these cases. We can only provide guidance.

Maintaining Old Migrations

We will not maintain old migrations. Migrations target the transition from one version to another and will thus be outdated once applied. We can only keep documentation and code archives around for (very) old migrations.

✅ Proposed Solutions

• Pallet Storage Version Unified Pallet Storage Version substrate#6888
  • The storage version of a pallet should be stored in a standardized format.
• Testing Tool
  • There should be a tool that allows running end-to-end tests of runtime migrations to determine whether they run correctly.
  • @seunlanlege is working on a test runner addressing this here.
• Migrations in Release Notes add runtime migrations to release notes/changelog substrate#6875
  • We should mention migrations in changelogs and release notes so that people are aware of them and can integrate them.
• Migration Guide (draft doc for collaboration)
  • We could have more documentation on how to migrate runtimes successfully

[apopiak]

A first idea addressing this issue would be introducing an (or several) extra repository that just contains migrations. The migrations could be crates depending on both the version they are migrating from as well as the one the are migrating to.
Runtimes could then include these crates to perform the migrations necessary.

[apopiak]

Original Text:

This issue is meant as a meta / tracking issue for Substrate's migration support.

I currently see several issues with the way we do migrations that it would be great to improve on.

1. I have the impression that we currently work off the assumption that most chains are similar to Kusama and Polkadot in how they migrate.
   a. The migrations included in Substrate only work out-of-the-box for Kusama and Polkadot. It's great that there are comments indicating assumptions (e.g. here) but the code is still Kusama/Polkadot specific.
   b. We add and remove migrations regularly. This assumes that downstream chains follow a similar migration time schedule.
   c. Sometimes we add migrations in extra migration.rs files, sometimes in the pallet's lib.rs themselves. This makes it difficult to add several past migrations together.
2. We assume that migrations don't fail. They are uni-directional and don't allow rollback.
3. There is no tooling to support creating and testing migrations. There are only ad-hoc solutions shared unofficially like grafting state onto a dev chainspec. I think we could use tooling for migration integration tests.

[apopiak]

💡Ideas for Improvement

Processes & Documentation

Ideas for adding and/or improving processes that will make migrations easier for Substrate users.

• Increase clarity and predictability of migrations by
  • mentioning how long they will stay
  • mentioning when they are removed
• Have an up-to-date list of migrations that adds context, guidance and trouble shooting.
  • MVP is the tagged PR listing
• Move migrations into a separate repo that can keep historical migrations around.
• Have a migration guide that leads through the process.

Tooling

• Have a metadata diffing tool to determine what changed between different runtimes. "Diffing the schema" so to speak.
  • A quick hacky version of this we can do easily by diffing the json represenation of the metadata.

Substrate Changes

• Allow faking validators/keys to test consensus changes.

[apopiak]

We can use docker run --network=host jacogr/polkadot-js-tools metadata <ws-old> <ws-new> (e.g. docker run --network=host jacogr/polkadot-js-tools metadata wss://rpc.polkadot.io ws://localhost:9944 to compare a local instance with the Polkadot production chain) to determine metadata changes between runtime versions.

[apopiak]

Collecting migrations in this repo with notes and examples.