Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

documentation for in-place migrations with x/upgrade module #8967

Merged
merged 62 commits into from
Apr 14, 2021
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
d4f1ffc
upgrade draft docs
technicallyty Mar 22, 2021
36d5386
upgrade draft docs2
technicallyty Mar 22, 2021
ea16761
Merge branch 'master' into ty-8940-upgrade_docs
technicallyty Mar 23, 2021
ed250a9
change to generic id
technicallyty Mar 23, 2021
3c6e6a1
Update docs/building-modules/upgrade.md
technicallyty Mar 24, 2021
32b3f92
Update docs/building-modules/README.md
technicallyty Mar 24, 2021
8aa4922
Update docs/building-modules/upgrade.md
technicallyty Mar 24, 2021
028512e
Update docs/building-modules/upgrade.md
technicallyty Mar 24, 2021
88076a3
Update docs/building-modules/upgrade.md
technicallyty Mar 24, 2021
33cbbac
Update docs/building-modules/upgrade.md
technicallyty Mar 24, 2021
3f82d6e
Update docs/building-modules/upgrade.md
technicallyty Mar 24, 2021
c7417ed
Update docs/building-modules/upgrade.md
technicallyty Mar 24, 2021
5da5e9c
Update docs/core/README.md
technicallyty Mar 24, 2021
912f802
Update docs/core/upgrade.md
technicallyty Mar 24, 2021
b178be3
Update docs/core/upgrade.md
technicallyty Mar 24, 2021
d2466af
Merge branch 'master' into ty-8940-upgrade_docs
technicallyty Mar 24, 2021
51d4a57
Merge branch 'ty-8940-upgrade_docs' of https://github.com/cosmos/cosm…
technicallyty Mar 24, 2021
f54941c
added lines for version map and consensus versions
technicallyty Mar 24, 2021
faf3553
fix headers, add synopsis tag
technicallyty Mar 24, 2021
17ea2a4
Merge branch 'master' into ty-8940-upgrade_docs
technicallyty Mar 24, 2021
aa0298a
Merge branch 'master' into ty-8940-upgrade_docs
technicallyty Mar 29, 2021
fda35b9
docs for new modules
technicallyty Mar 29, 2021
c23c775
Merge branch 'master' into ty-8940-upgrade_docs
technicallyty Mar 30, 2021
7e1c59e
fix
technicallyty Mar 30, 2021
b91e93f
remove line
technicallyty Mar 30, 2021
e3f45fe
Update docs/core/upgrade.md
technicallyty Mar 31, 2021
cf28777
Update docs/building-modules/upgrade.md
technicallyty Mar 31, 2021
63e5cfe
Merge branch 'master' into ty-8940-upgrade_docs
technicallyty Mar 31, 2021
52414d3
SetUpgradeHandler example snippet
technicallyty Mar 31, 2021
a388ee4
Merge branch 'ty-8940-upgrade_docs' of https://github.com/cosmos/cosm…
technicallyty Mar 31, 2021
188abe9
Merge branch 'master' into ty-8940-upgrade_docs
technicallyty Apr 2, 2021
5bc3bf6
-general clean up of docs
technicallyty Apr 2, 2021
e054d77
Update docs/core/upgrade.md
technicallyty Apr 6, 2021
8b1c7ad
Update docs/core/upgrade.md
technicallyty Apr 6, 2021
a63942b
Update docs/core/upgrade.md
technicallyty Apr 6, 2021
d4472a5
Update docs/core/upgrade.md
technicallyty Apr 6, 2021
47021cb
Update docs/core/upgrade.md
technicallyty Apr 6, 2021
1d6c429
Update docs/core/upgrade.md
technicallyty Apr 6, 2021
6a7d393
Update docs/core/upgrade.md
technicallyty Apr 6, 2021
7810c1b
Update docs/core/upgrade.md
technicallyty Apr 6, 2021
c217fa5
Update docs/core/upgrade.md
technicallyty Apr 6, 2021
0986348
Update docs/core/upgrade.md
technicallyty Apr 6, 2021
80260f5
Update docs/core/upgrade.md
technicallyty Apr 6, 2021
f61c7fa
Update docs/core/upgrade.md
technicallyty Apr 6, 2021
0c85af2
Merge branch 'master' into ty-8940-upgrade_docs
technicallyty Apr 6, 2021
6c70507
Apply suggestions from code review
technicallyty Apr 7, 2021
3d9e00a
fix self-reference
technicallyty Apr 7, 2021
41b2c56
Merge branch 'master' into ty-8940-upgrade_docs
technicallyty Apr 7, 2021
8ee1736
Merge branch 'master' into ty-8940-upgrade_docs
technicallyty Apr 9, 2021
68a662a
Merge branch 'ty-8940-upgrade_docs' of https://github.com/cosmos/cosm…
technicallyty Apr 9, 2021
7883af3
clearer definitions for overwriting genesis functions
technicallyty Apr 9, 2021
a77aa4c
add sync section
technicallyty Apr 9, 2021
52da394
forgot to save this
technicallyty Apr 9, 2021
5a639af
Merge branch 'master' into ty-8940-upgrade_docs
Apr 12, 2021
027eb18
Merge branch 'master' into ty-8940-upgrade_docs
technicallyty Apr 12, 2021
1895a8a
Merge branch 'ty-8940-upgrade_docs' of https://github.com/cosmos/cosm…
technicallyty Apr 12, 2021
ff40788
Merge branch 'master' into ty-8940-upgrade_docs
technicallyty Apr 13, 2021
6ea0101
update description of initgenesis modules
technicallyty Apr 13, 2021
21c8c66
Merge branch 'master' into ty-8940-upgrade_docs
technicallyty Apr 13, 2021
84b6a73
specify upgrade method
technicallyty Apr 13, 2021
85d86e1
Update docs/core/upgrade.md
technicallyty Apr 13, 2021
b1ccd62
Merge branch 'master' into ty-8940-upgrade_docs
mergify[bot] Apr 14, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/building-modules/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,4 @@ This repository contains documentation on concepts developers need to know in or
10. [Module Interfaces](./module-interfaces.md)
11. [Standard Module Structure](./structure.md)
12. [Errors](./errors.md)
13. [In-Place Store Migrations](./upgrade.md)
55 changes: 55 additions & 0 deletions docs/building-modules/upgrade.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
<!--
order: 13
-->

# In-Place Store Migrations

In-place store migrations allow your modules to smoothly transition to new versions with breaking changes. This document outlines how to build modules to take advantage of this functionality. {synopsis}
technicallyty marked this conversation as resolved.
Show resolved Hide resolved
## Pre-requisite Readings
technicallyty marked this conversation as resolved.
Show resolved Hide resolved

technicallyty marked this conversation as resolved.
Show resolved Hide resolved
- [In-Place Store Migration](../core-concepts/upgrade.md) {prereq}
## Consensus Version

In order to successfully upgrade your existing modules, your `AppModule`s must implement the function `ConsensusVersion() uint64`. The `uint64` returned will serve as the consensus version and should be hard coded by the module developer. This number will serve as a state-breaking version of each app module, so it *MUST* be incremented on each consensus-breaking change introduced by the module. The initial version *MUST* be set to 1, with the exception of situations where an upgrade introduces an entirely new module. More on that [here](#adding-new-modules-in-an-upgrade)

## Registering Migrations

To register the functionality that takes place during a module upgrade, we must register which migrations we want to take place. This takes place in the `Configurator` via the `RegisterMigration` method. The `AppModule`s have a reference to the configurator in the `RegisterServices` method. We can register a number of migrations here, however, if more than one migration script is registered, it is important they are listed in increasing order. Additionally, we must ensure there are enough migrations that will lead to the desired consensus version. For example, if we wanted to migrate to version 3 of a module, we would need to register a separate migration for both version 1 and 2 as shown below.
technicallyty marked this conversation as resolved.
Show resolved Hide resolved

```golang
func (am AppModule) RegisterServices(cfg module.Configurator) {
// --snip--
cfg.RegisterMigration(types.ModuleName, 1, func(ctx sdk.Context) error {
// Perform in-place store migrations from ConsensusVersion 1 to 2.
})
cfg.RegisterMigration(types.ModuleName, 2, func(ctx sdk.Context) error {
// Perform in-place store migrations from ConsensusVersion 2 to 3.
})
}
```

Since these migrations are functions that need access to a Keeper's store, we use a wrapper around the keepers called `Migrator`. An example of this can be found here:
technicallyty marked this conversation as resolved.
Show resolved Hide resolved

+++ https://github.com/cosmos/cosmos-sdk/blob/6ac8898fec9bd7ea2c1e5c79e0ed0c3f827beb55/x/bank/keeper/migrations.go#L8-L21

In addition to the `Migrator` wrapper, we also define our migration scripts. More on that below.

## Writing Migration Scripts

In order to define the functionality that takes place during an upgrade, we will write a migration script. Since migration scripts will manipulate legacy code, we place these functions in a `legacy/` directory. For example, if we wanted to write migration scripts for a module named `bank`, we would place the functions in `x/bank/legacy/`. We recommend the following naming convention for these functions:

```golang
// Migrating bank module from version 1 to 2
func (m Migrator) Migrate1to2(ctx sdk.Context) error {
return v043bank.MigrateStore(ctx, m.keeper.storeKey) // v043bank is package `x/bank/legacy/v043`.
}
```

If you would like to see example code of changes implemented in a migration, you can check out the code [here](https://github.com/cosmos/cosmos-sdk/blob/36f68eb9e041e20a5bb47e216ac5eb8b91f95471/x/bank/legacy/v043/store.go#L41-L62). For context, this introduced migrations of the bank store that updated addresses to be prefixed by their length in bytes as perscribed in (ADR-028)[../architecture/adr-028-public-key-addresses.md].
technicallyty marked this conversation as resolved.
Show resolved Hide resolved


## Adding New Modules In Upgrades

When introducing a new module to your application during an upgrade, your `AppModule` must begin at consensus version 0. When an upgrade is being executed, all modules with consensus versions set to 0 will be recognized as new modules. This will result in the new module's `InitGenesis` function executing. This allows new modules to setup some inital state. The result of this will set the `AppModule`s consensus version to `1`.
amaury1093 marked this conversation as resolved.
Show resolved Hide resolved

If your module does not require any initial state, you do not need to implement any `InitGenesis`. The `InitGenesis` will result in a no-op, but still upgrade your consensus version to `1` for future upgrades.
amaury1093 marked this conversation as resolved.
Show resolved Hide resolved
amaury1093 marked this conversation as resolved.
Show resolved Hide resolved
27 changes: 14 additions & 13 deletions docs/core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,19 +9,20 @@ parent:
This repository contains reference documentation on the core concepts of the Cosmos SDK.

1. [`BaseApp`](./baseapp.md)
1. [Transaction](./transactions.md)
1. [Context](./context.md)
1. [Node Client](./node.md)
1. [Store](./store.md)
1. [Encoding](./encoding.md)
1. [gRPC, REST and Tendermint Endpoints](./grpc_rest.md)
1. [Command-Line Interface](./cli.md)
1. [Events](./events.md)
1. [Telemetry](./telemetry.md)
1. [Object-Capabilities](./ocap.md)
1. [RunTx recovery middleware](./runtx_middleware.md)
1. [Simulation](./simulation.md)
1. [Protobuf documentation](./proto-docs.md)
2. [Transaction](./transactions.md)
3. [Context](./context.md)
4. [Node Client](./node.md)
5. [Store](./store.md)
6. [Encoding](./encoding.md)
7. [gRPC, REST and Tendermint Endpoints](./grpc_rest.md)
8. [Command-Line Interface](./cli.md)
9. [Events](./events.md)
10. [Telemetry](./telemetry.md)
11. [Object-Capabilities](./ocap.md)
12. [RunTx recovery middleware](./runtx_middleware.md)
13. [Simulation](./simulation.md)
14. [Protobuf documentation](./proto-docs.md)
15. [In-Place Store Migrations](./upgrade.md)

After reading about the core concepts, check the [IBC documentation](../ibc/README.md) to learn more
about the IBC core concepts and how to integrate it to you application.
technicallyty marked this conversation as resolved.
Show resolved Hide resolved
43 changes: 43 additions & 0 deletions docs/core/upgrade.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
<!--
order: 15
-->

# In-Place Store Migrations

Upgrade your app modules smoothly with custom in-place migration logic. {synopsis}
technicallyty marked this conversation as resolved.
Show resolved Hide resolved

The Cosmos SDK currently has two ways to perform upgrades. The first way is by exporting the entire application state to a JSON file using the `simd export` CLI command, making changes, and then starting a new binary with the changed JSON file as the genesis file. The second way is by performing upgrades in place, significantly decreasing the time needed to perform upgrades for chains with a larger state. The following will guide you on how to setup your application to take advantage of the second method described above.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember reading a sentence saying something like "this document describes the 2nd method", imo it's still useful to add it here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

## Enabling Upgrades

To enable your application to conform to the upgrade module's specifications, a few changes need to be made to your application.

## Genesis State
technicallyty marked this conversation as resolved.
Show resolved Hide resolved

Each app module's consensus version must be saved to state on the application's genesis. This can be done by adding the following line to the `InitChainer` method in `app.go`
technicallyty marked this conversation as resolved.
Show resolved Hide resolved

```diff
func (app *MyApp) InitChainer(ctx sdk.Context, req abci.RequestInitChain) abci.ResponseInitChain {
...
+ app.UpgradeKeeper.SetModuleVersionMap(ctx, app.mm.GetVersionMap())
...
}
```

amaury1093 marked this conversation as resolved.
Show resolved Hide resolved
### Consensus Version
The consensus version is defined on each app module by the module developer. It serves as the breaking change version of the module.

### Version Map
The version map is a mapping of module names to consensus versions. The map is persisted to state for use during in-place migrations.

## Upgrade Handlers

Upgrades utilize an `UpgradeHandler` to facilitate migrations. `UpgradeHandler`s are functions implemented by the app developer that conform to the following function signature.

```golang
type UpgradeHandler func(ctx sdk.Context, plan Plan, versionMap VersionMap) (VersionMap, error)
```

## Running Migrations

In practice, the handlers should simply call and return the values from the `app.mm.RunMigrations` function. The `RunMigrations` function should be passed the `VersionMap` from the `UpgradeHandler`. With this, the `RunMigration` function will loop through the `VersionMap`, and for any current app module who's consensus version is greater than its corresponding value in the `VersionMap`, have its migration scripts ran. To learn how to configure migration scripts, refer to (this guide)[../building-modules/upgrade.md].
amaury1093 marked this conversation as resolved.
Show resolved Hide resolved