From d4f1ffc6f98321145437e5d49abc3e5d328b9dac Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Mon, 22 Mar 2021 13:33:44 -0700 Subject: [PATCH 01/43] upgrade draft docs --- docs/building-modules/README.md | 1 + docs/core/README.md | 27 ++++++++++++++------------- 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/docs/building-modules/README.md b/docs/building-modules/README.md index 22f3ffb75e20..fc4a94a3d8f4 100644 --- a/docs/building-modules/README.md +++ b/docs/building-modules/README.md @@ -20,3 +20,4 @@ This repository contains documentation on concepts developers need to know in or 10. [Module Interfaces](./module-interfaces.md) 11. [Standard Module Structure](./structure.md) 12. [Errors](./errors.md) +13. [Upgrades](./upgrade.md) diff --git a/docs/core/README.md b/docs/core/README.md index 94a993f5a556..e5398f891310 100644 --- a/docs/core/README.md +++ b/docs/core/README.md @@ -9,19 +9,20 @@ parent: This repository contains reference documentation on the core concepts of the Cosmos SDK. 1. [`BaseApp`](./baseapp.md) -1. [Transaction](./transactions.md) -1. [Context](./context.md) -1. [Node Client](./node.md) -1. [Store](./store.md) -1. [Encoding](./encoding.md) -1. [gRPC, REST and Tendermint Endpoints](./grpc_rest.md) -1. [Command-Line Interface](./cli.md) -1. [Events](./events.md) -1. [Telemetry](./telemetry.md) -1. [Object-Capabilities](./ocap.md) -1. [RunTx recovery middleware](./runtx_middleware.md) -1. [Simulation](./simulation.md) -1. [Protobuf documentation](./proto-docs.md) +2. [Transaction](./transactions.md) +3. [Context](./context.md) +4. [Node Client](./node.md) +5. [Store](./store.md) +6. [Encoding](./encoding.md) +7. [gRPC, REST and Tendermint Endpoints](./grpc_rest.md) +8. [Command-Line Interface](./cli.md) +9. [Events](./events.md) +10. [Telemetry](./telemetry.md) +11. [Object-Capabilities](./ocap.md) +12. [RunTx recovery middleware](./runtx_middleware.md) +13. [Simulation](./simulation.md) +14. [Protobuf documentation](./proto-docs.md) +15. [Upgrades](./upgrade.md) After reading about the core concepts, check the [IBC documentation](../ibc/README.md) to learn more about the IBC core concepts and how to integrate it to you application. From 36d5386bd1c62a243fd4782a620c00f0f4e0240a Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Mon, 22 Mar 2021 13:34:05 -0700 Subject: [PATCH 02/43] upgrade draft docs2 --- docs/building-modules/upgrade.md | 42 ++++++++++++++++++++++++++++++++ docs/core/upgrade.md | 37 ++++++++++++++++++++++++++++ 2 files changed, 79 insertions(+) create mode 100644 docs/building-modules/upgrade.md create mode 100644 docs/core/upgrade.md diff --git a/docs/building-modules/upgrade.md b/docs/building-modules/upgrade.md new file mode 100644 index 000000000000..e4a2b8fbbc93 --- /dev/null +++ b/docs/building-modules/upgrade.md @@ -0,0 +1,42 @@ + + +# Upgrade + +The Upgrade module allows your modules to smoothly transition to new versions with breaking changes. This document outlines how to build modules to take advantage of this functionality. + +# Consensus Version + +In order to successfully upgrade your existing modules, your `AppModule`s must implement the function `ConsensusVersion() uint64`. The `uint64` returned will serve as the consensus version and should be hard coded by the module developer. This number will serve as a state-breaking version of each app module. The initial version *MUST* be set to 1. + +# Registering Migrations + +To register the functionality that takes place during a module upgrade, we must register which migrations we want to take place. This takes place in the `Configurator` via the `RegisterMigration` method. The `AppModule`s have a reference to the configurator in the `RegisterServices` method. We can register a number of migrations here, however, if more than one migration script is registered, it is important they are listed in increasing order. Additionally, we must ensure there are enough migrations that will lead to the desired consensus version. For example, if we wanted to migrate to version 3 of a module, we would need to register a separate migration for both version 1 and 2 as shown below. + +```golang +func (am AppModule) RegisterServices(cfg module.Configurator) { + // --snip-- + cfg.RegisterMigration(types.ModuleName, 1, func(ctx sdk.Context) error { + // Perform in-place store migrations from ConsensusVersion 1 to 2. + }) + cfg.RegisterMigration(types.ModuleName, 2, func(ctx sdk.Context) error { + // Perform in-place store migrations from ConsensusVersion 2 to 3. + }) +} +``` + +Since these migrations are functions that need access to a Keeper's store, we use a wrapper around the keepers called `Migrator`. An example of this can be found (here)[https://github.com/cosmos/cosmos-sdk/blob/master/x/bank/keeper/migrations.go]. In addition to the `Migrator` wrapper, we also define our migration scripts. More on that below. + +# Writing Migration Scripts + +In order to define the functionality that takes place during an upgrade, we will write a migration script. Since migration scripts will manipulate legacy code, we place these functions in a `legacy/` directory. For example, if we wanted to write migration scripts for a module named `bank`, we would place the functions in `x/bank/legacy/`. We recommend the following naming convention for these functions: + +```golang +// Migrating bank module from version 1 to 2 +func (m Migrator) Migrate1to2(ctx sdk.Context) error { + return v043bank.MigrateStore(ctx, m.keeper.storeKey) // v043bank is package `x/bank/legacy/v043`. +} +``` + +If you would like to see example code of changes implemented in a migration, you can check out the code [here](https://github.com/cosmos/cosmos-sdk/blob/36f68eb9e041e20a5bb47e216ac5eb8b91f95471/x/bank/legacy/v043/store.go#L41-L62). For context, this introduced migrations of the bank store that updated addresses to be prefixed by their length in bytes as perscribed in (ADR-028)[https://docs.cosmos.network/master/architecture/adr-028-public-key-addresses.html]. diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md new file mode 100644 index 000000000000..b8549029c4e7 --- /dev/null +++ b/docs/core/upgrade.md @@ -0,0 +1,37 @@ + + +# Upgrades + +Upgrade your app modules smoothly with custom in-place migration logic. + +The Cosmos SDK currently has two ways to perform upgrades. The first way is by exporting the entire application state to a JSON file, making changes, and then starting a new binary with the changed JSON file as the genesis file. The second way is by performing upgrades in place, significantly decreasing the time needed to perform upgrades for chains with a larger state. The following will guide you on how to setup your application to take advantage of the second method described above. + +# Enabling Upgrades + +To enable your application to conform to the upgrade module's specifications, a few changes need to be made to your application. + +# Genesis State + +Each app module's consensus version must be saved to state on the application's genesis. This can be done by adding the following line to the `InitChainer` method in `app.go` + +```diff +func (app *thorChainApp) InitChainer(ctx sdk.Context, req abci.RequestInitChain) abci.ResponseInitChain { + ... ++ app.UpgradeKeeper.SetModuleVersionMap(ctx, app.mm.GetVersionMap()) + ... +} +``` + +# Upgrade Handlers + +Upgrades utilize an `UpgradeHandler` to facilitate migrations. `UpgradeHandler`s are functions implemented by the app developer that conform to the following function signature. + +```golang +type UpgradeHandler func(ctx sdk.Context, plan Plan, versionMap VersionMap) (VersionMap, error) +``` + +# Running Migrations + +In practice, the handlers should simply call and return the values from the `app.mm.RunMigrations` function. The `RunMigrations` function should be passed the `VersionMap` from the `UpgradeHandler`. With this, the `RunMigration` function will loop through the `VersionMap`, and for any current app module who's consensus version is greater than its corresponding value in the `VersionMap`, have its migration scripts ran. To learn how to configure migration scripts, refer to (this guide)[../building-modules/upgrade.md]. \ No newline at end of file From ed250a9a1de6687f2f75d3574e24a61bc38dc143 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Tue, 23 Mar 2021 11:15:51 -0700 Subject: [PATCH 03/43] change to generic id --- docs/core/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index b8549029c4e7..d5f57ead3c09 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -17,7 +17,7 @@ To enable your application to conform to the upgrade module's specifications, a Each app module's consensus version must be saved to state on the application's genesis. This can be done by adding the following line to the `InitChainer` method in `app.go` ```diff -func (app *thorChainApp) InitChainer(ctx sdk.Context, req abci.RequestInitChain) abci.ResponseInitChain { +func (app *MyApp) InitChainer(ctx sdk.Context, req abci.RequestInitChain) abci.ResponseInitChain { ... + app.UpgradeKeeper.SetModuleVersionMap(ctx, app.mm.GetVersionMap()) ... From 3c6e6a127a46f10a7966a4a89fa2ef483d8c9027 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Wed, 24 Mar 2021 13:36:02 -0700 Subject: [PATCH 04/43] Update docs/building-modules/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/building-modules/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/building-modules/upgrade.md b/docs/building-modules/upgrade.md index e4a2b8fbbc93..e2c652cc1a39 100644 --- a/docs/building-modules/upgrade.md +++ b/docs/building-modules/upgrade.md @@ -2,7 +2,7 @@ order: 13 --> -# Upgrade +# In-Place Store Migrations The Upgrade module allows your modules to smoothly transition to new versions with breaking changes. This document outlines how to build modules to take advantage of this functionality. From 32b3f92807083d942e4f8f184f4a4eb0ea9ba26e Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Wed, 24 Mar 2021 13:36:35 -0700 Subject: [PATCH 05/43] Update docs/building-modules/README.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/building-modules/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/building-modules/README.md b/docs/building-modules/README.md index fc4a94a3d8f4..411c3e8d38d8 100644 --- a/docs/building-modules/README.md +++ b/docs/building-modules/README.md @@ -20,4 +20,4 @@ This repository contains documentation on concepts developers need to know in or 10. [Module Interfaces](./module-interfaces.md) 11. [Standard Module Structure](./structure.md) 12. [Errors](./errors.md) -13. [Upgrades](./upgrade.md) +13. [In-Place Store Migrations](./upgrade.md) From 8aa49220d3f642299f598a9ef89e1cccbe32a5d8 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Wed, 24 Mar 2021 13:36:49 -0700 Subject: [PATCH 06/43] Update docs/building-modules/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/building-modules/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/building-modules/upgrade.md b/docs/building-modules/upgrade.md index e2c652cc1a39..fe525b5eec1a 100644 --- a/docs/building-modules/upgrade.md +++ b/docs/building-modules/upgrade.md @@ -4,7 +4,7 @@ order: 13 # In-Place Store Migrations -The Upgrade module allows your modules to smoothly transition to new versions with breaking changes. This document outlines how to build modules to take advantage of this functionality. +In-place store migrations allow your modules to smoothly transition to new versions with breaking changes. This document outlines how to build modules to take advantage of this functionality. # Consensus Version From 028512eb723742266fd2408c904379f014f7a89d Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Wed, 24 Mar 2021 13:37:45 -0700 Subject: [PATCH 07/43] Update docs/building-modules/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/building-modules/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/building-modules/upgrade.md b/docs/building-modules/upgrade.md index fe525b5eec1a..235b9ec29d02 100644 --- a/docs/building-modules/upgrade.md +++ b/docs/building-modules/upgrade.md @@ -8,7 +8,7 @@ In-place store migrations allow your modules to smoothly transition to new versi # Consensus Version -In order to successfully upgrade your existing modules, your `AppModule`s must implement the function `ConsensusVersion() uint64`. The `uint64` returned will serve as the consensus version and should be hard coded by the module developer. This number will serve as a state-breaking version of each app module. The initial version *MUST* be set to 1. +In order to successfully upgrade your existing modules, your `AppModule`s must implement the function `ConsensusVersion() uint64`. The `uint64` returned will serve as the consensus version and should be hard coded by the module developer. This number will serve as a state-breaking version of each app module, so it *MUST* be incremented on each consensus-breaking change introduced by the module. The initial version *MUST* be set to 1. # Registering Migrations From 88076a3473eda4df0591a8110c19b4a9e75ea401 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Wed, 24 Mar 2021 13:38:10 -0700 Subject: [PATCH 08/43] Update docs/building-modules/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/building-modules/upgrade.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/building-modules/upgrade.md b/docs/building-modules/upgrade.md index 235b9ec29d02..344acb3bcc99 100644 --- a/docs/building-modules/upgrade.md +++ b/docs/building-modules/upgrade.md @@ -26,7 +26,11 @@ func (am AppModule) RegisterServices(cfg module.Configurator) { } ``` -Since these migrations are functions that need access to a Keeper's store, we use a wrapper around the keepers called `Migrator`. An example of this can be found (here)[https://github.com/cosmos/cosmos-sdk/blob/master/x/bank/keeper/migrations.go]. In addition to the `Migrator` wrapper, we also define our migration scripts. More on that below. +Since these migrations are functions that need access to a Keeper's store, we use a wrapper around the keepers called `Migrator`. An example of this can be found here: + ++++ https://github.com/cosmos/cosmos-sdk/blob/6ac8898fec9bd7ea2c1e5c79e0ed0c3f827beb55/x/bank/keeper/migrations.go#L8-L21 + +In addition to the `Migrator` wrapper, we also define our migration scripts. More on that below. # Writing Migration Scripts From 33cbbac5070f32178c6f9f196282949f2791ccd2 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Wed, 24 Mar 2021 13:38:21 -0700 Subject: [PATCH 09/43] Update docs/building-modules/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/building-modules/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/building-modules/upgrade.md b/docs/building-modules/upgrade.md index 344acb3bcc99..29e5f2dc76c1 100644 --- a/docs/building-modules/upgrade.md +++ b/docs/building-modules/upgrade.md @@ -6,7 +6,7 @@ order: 13 In-place store migrations allow your modules to smoothly transition to new versions with breaking changes. This document outlines how to build modules to take advantage of this functionality. -# Consensus Version +## Consensus Version In order to successfully upgrade your existing modules, your `AppModule`s must implement the function `ConsensusVersion() uint64`. The `uint64` returned will serve as the consensus version and should be hard coded by the module developer. This number will serve as a state-breaking version of each app module, so it *MUST* be incremented on each consensus-breaking change introduced by the module. The initial version *MUST* be set to 1. From 3f82d6e96ed952734a8d84a590691220e2d05a1b Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Wed, 24 Mar 2021 13:38:40 -0700 Subject: [PATCH 10/43] Update docs/building-modules/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/building-modules/upgrade.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/building-modules/upgrade.md b/docs/building-modules/upgrade.md index 29e5f2dc76c1..fa4bd694d402 100644 --- a/docs/building-modules/upgrade.md +++ b/docs/building-modules/upgrade.md @@ -5,7 +5,9 @@ order: 13 # In-Place Store Migrations In-place store migrations allow your modules to smoothly transition to new versions with breaking changes. This document outlines how to build modules to take advantage of this functionality. +## Pre-requisite Readings +- [In-Place Store Migration](../core-concepts/upgrade.md) {prereq} ## Consensus Version In order to successfully upgrade your existing modules, your `AppModule`s must implement the function `ConsensusVersion() uint64`. The `uint64` returned will serve as the consensus version and should be hard coded by the module developer. This number will serve as a state-breaking version of each app module, so it *MUST* be incremented on each consensus-breaking change introduced by the module. The initial version *MUST* be set to 1. From c7417ed903fe62d652828a5a8e25aca4616e73f9 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Wed, 24 Mar 2021 13:38:55 -0700 Subject: [PATCH 11/43] Update docs/building-modules/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/building-modules/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/building-modules/upgrade.md b/docs/building-modules/upgrade.md index fa4bd694d402..65f88f4531fe 100644 --- a/docs/building-modules/upgrade.md +++ b/docs/building-modules/upgrade.md @@ -45,4 +45,4 @@ func (m Migrator) Migrate1to2(ctx sdk.Context) error { } ``` -If you would like to see example code of changes implemented in a migration, you can check out the code [here](https://github.com/cosmos/cosmos-sdk/blob/36f68eb9e041e20a5bb47e216ac5eb8b91f95471/x/bank/legacy/v043/store.go#L41-L62). For context, this introduced migrations of the bank store that updated addresses to be prefixed by their length in bytes as perscribed in (ADR-028)[https://docs.cosmos.network/master/architecture/adr-028-public-key-addresses.html]. +If you would like to see example code of changes implemented in a migration, you can check out the code [here](https://github.com/cosmos/cosmos-sdk/blob/36f68eb9e041e20a5bb47e216ac5eb8b91f95471/x/bank/legacy/v043/store.go#L41-L62). For context, this introduced migrations of the bank store that updated addresses to be prefixed by their length in bytes as perscribed in (ADR-028)[../architecture/adr-028-public-key-addresses.md]. From 5da5e9c83dffbc4a8bcb5dc990eb454652533d22 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Wed, 24 Mar 2021 13:39:11 -0700 Subject: [PATCH 12/43] Update docs/core/README.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/README.md b/docs/core/README.md index e5398f891310..e3e9d308fe92 100644 --- a/docs/core/README.md +++ b/docs/core/README.md @@ -22,7 +22,7 @@ This repository contains reference documentation on the core concepts of the Cos 12. [RunTx recovery middleware](./runtx_middleware.md) 13. [Simulation](./simulation.md) 14. [Protobuf documentation](./proto-docs.md) -15. [Upgrades](./upgrade.md) +15. [In-Place Store Migrations](./upgrade.md) After reading about the core concepts, check the [IBC documentation](../ibc/README.md) to learn more about the IBC core concepts and how to integrate it to you application. From 912f8024acd60f4dc4d62e487dbd2529bff291d9 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Wed, 24 Mar 2021 13:41:14 -0700 Subject: [PATCH 13/43] Update docs/core/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/upgrade.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index d5f57ead3c09..1c9672218fc1 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -2,7 +2,7 @@ order: 15 --> -# Upgrades +# In-Place Store Migrations Upgrade your app modules smoothly with custom in-place migration logic. @@ -34,4 +34,4 @@ type UpgradeHandler func(ctx sdk.Context, plan Plan, versionMap VersionMap) (Ver # Running Migrations -In practice, the handlers should simply call and return the values from the `app.mm.RunMigrations` function. The `RunMigrations` function should be passed the `VersionMap` from the `UpgradeHandler`. With this, the `RunMigration` function will loop through the `VersionMap`, and for any current app module who's consensus version is greater than its corresponding value in the `VersionMap`, have its migration scripts ran. To learn how to configure migration scripts, refer to (this guide)[../building-modules/upgrade.md]. \ No newline at end of file +In practice, the handlers should simply call and return the values from the `app.mm.RunMigrations` function. The `RunMigrations` function should be passed the `VersionMap` from the `UpgradeHandler`. With this, the `RunMigration` function will loop through the `VersionMap`, and for any current app module who's consensus version is greater than its corresponding value in the `VersionMap`, have its migration scripts ran. To learn how to configure migration scripts, refer to (this guide)[../building-modules/upgrade.md]. From b178be3b8adc62691f680f0d73fd6c67480532c4 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Wed, 24 Mar 2021 13:41:44 -0700 Subject: [PATCH 14/43] Update docs/core/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 1c9672218fc1..2c4fc4ab59d5 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -6,7 +6,7 @@ order: 15 Upgrade your app modules smoothly with custom in-place migration logic. -The Cosmos SDK currently has two ways to perform upgrades. The first way is by exporting the entire application state to a JSON file, making changes, and then starting a new binary with the changed JSON file as the genesis file. The second way is by performing upgrades in place, significantly decreasing the time needed to perform upgrades for chains with a larger state. The following will guide you on how to setup your application to take advantage of the second method described above. +The Cosmos SDK currently has two ways to perform upgrades. The first way is by exporting the entire application state to a JSON file using the `simd export` CLI command, making changes, and then starting a new binary with the changed JSON file as the genesis file. The second way is by performing upgrades in place, significantly decreasing the time needed to perform upgrades for chains with a larger state. The following will guide you on how to setup your application to take advantage of the second method described above. # Enabling Upgrades From f54941cb1f42886c22b8a84d5ea3eac9405b8e84 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Wed, 24 Mar 2021 14:29:07 -0700 Subject: [PATCH 15/43] added lines for version map and consensus versions --- docs/core/upgrade.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 2c4fc4ab59d5..84ee743ddfec 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -4,15 +4,15 @@ order: 15 # In-Place Store Migrations -Upgrade your app modules smoothly with custom in-place migration logic. +Upgrade your app modules smoothly with custom in-place migration logic. {synopsis} The Cosmos SDK currently has two ways to perform upgrades. The first way is by exporting the entire application state to a JSON file using the `simd export` CLI command, making changes, and then starting a new binary with the changed JSON file as the genesis file. The second way is by performing upgrades in place, significantly decreasing the time needed to perform upgrades for chains with a larger state. The following will guide you on how to setup your application to take advantage of the second method described above. -# Enabling Upgrades +## Enabling Upgrades To enable your application to conform to the upgrade module's specifications, a few changes need to be made to your application. -# Genesis State +## Genesis State Each app module's consensus version must be saved to state on the application's genesis. This can be done by adding the following line to the `InitChainer` method in `app.go` @@ -24,7 +24,13 @@ func (app *MyApp) InitChainer(ctx sdk.Context, req abci.RequestInitChain) abci.R } ``` -# Upgrade Handlers +### Consensus Version +The consensus version is defined on each app module. This is a `uint64` that tracks breaking changes of each module for migrations. + +### Version Map +The version map is a mapping of module names to consensus versions. The map is persisted to state for use during in-place migrations. + +## Upgrade Handlers Upgrades utilize an `UpgradeHandler` to facilitate migrations. `UpgradeHandler`s are functions implemented by the app developer that conform to the following function signature. @@ -32,6 +38,6 @@ Upgrades utilize an `UpgradeHandler` to facilitate migrations. `UpgradeHandler`s type UpgradeHandler func(ctx sdk.Context, plan Plan, versionMap VersionMap) (VersionMap, error) ``` -# Running Migrations +## Running Migrations In practice, the handlers should simply call and return the values from the `app.mm.RunMigrations` function. The `RunMigrations` function should be passed the `VersionMap` from the `UpgradeHandler`. With this, the `RunMigration` function will loop through the `VersionMap`, and for any current app module who's consensus version is greater than its corresponding value in the `VersionMap`, have its migration scripts ran. To learn how to configure migration scripts, refer to (this guide)[../building-modules/upgrade.md]. From faf3553214a34bff88c197c0428d93c82242d702 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Wed, 24 Mar 2021 14:30:46 -0700 Subject: [PATCH 16/43] fix headers, add synopsis tag --- docs/building-modules/upgrade.md | 6 +++--- docs/core/upgrade.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/building-modules/upgrade.md b/docs/building-modules/upgrade.md index 65f88f4531fe..3b4fb28196fd 100644 --- a/docs/building-modules/upgrade.md +++ b/docs/building-modules/upgrade.md @@ -4,7 +4,7 @@ order: 13 # In-Place Store Migrations -In-place store migrations allow your modules to smoothly transition to new versions with breaking changes. This document outlines how to build modules to take advantage of this functionality. +In-place store migrations allow your modules to smoothly transition to new versions with breaking changes. This document outlines how to build modules to take advantage of this functionality. {synopsis} ## Pre-requisite Readings - [In-Place Store Migration](../core-concepts/upgrade.md) {prereq} @@ -12,7 +12,7 @@ In-place store migrations allow your modules to smoothly transition to new versi In order to successfully upgrade your existing modules, your `AppModule`s must implement the function `ConsensusVersion() uint64`. The `uint64` returned will serve as the consensus version and should be hard coded by the module developer. This number will serve as a state-breaking version of each app module, so it *MUST* be incremented on each consensus-breaking change introduced by the module. The initial version *MUST* be set to 1. -# Registering Migrations +## Registering Migrations To register the functionality that takes place during a module upgrade, we must register which migrations we want to take place. This takes place in the `Configurator` via the `RegisterMigration` method. The `AppModule`s have a reference to the configurator in the `RegisterServices` method. We can register a number of migrations here, however, if more than one migration script is registered, it is important they are listed in increasing order. Additionally, we must ensure there are enough migrations that will lead to the desired consensus version. For example, if we wanted to migrate to version 3 of a module, we would need to register a separate migration for both version 1 and 2 as shown below. @@ -34,7 +34,7 @@ Since these migrations are functions that need access to a Keeper's store, we us In addition to the `Migrator` wrapper, we also define our migration scripts. More on that below. -# Writing Migration Scripts +## Writing Migration Scripts In order to define the functionality that takes place during an upgrade, we will write a migration script. Since migration scripts will manipulate legacy code, we place these functions in a `legacy/` directory. For example, if we wanted to write migration scripts for a module named `bank`, we would place the functions in `x/bank/legacy/`. We recommend the following naming convention for these functions: diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 84ee743ddfec..6e89a3ca486a 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -25,7 +25,7 @@ func (app *MyApp) InitChainer(ctx sdk.Context, req abci.RequestInitChain) abci.R ``` ### Consensus Version -The consensus version is defined on each app module. This is a `uint64` that tracks breaking changes of each module for migrations. +The consensus version is defined on each app module by the module developer. It serves as the breaking change version of the module. ### Version Map The version map is a mapping of module names to consensus versions. The map is persisted to state for use during in-place migrations. From fda35b9e01ea3c6b5181dea413a228b2c2d54482 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Mon, 29 Mar 2021 14:29:41 -0700 Subject: [PATCH 17/43] docs for new modules --- docs/building-modules/upgrade.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/building-modules/upgrade.md b/docs/building-modules/upgrade.md index 3b4fb28196fd..9cb8ea1be6c7 100644 --- a/docs/building-modules/upgrade.md +++ b/docs/building-modules/upgrade.md @@ -10,7 +10,7 @@ In-place store migrations allow your modules to smoothly transition to new versi - [In-Place Store Migration](../core-concepts/upgrade.md) {prereq} ## Consensus Version -In order to successfully upgrade your existing modules, your `AppModule`s must implement the function `ConsensusVersion() uint64`. The `uint64` returned will serve as the consensus version and should be hard coded by the module developer. This number will serve as a state-breaking version of each app module, so it *MUST* be incremented on each consensus-breaking change introduced by the module. The initial version *MUST* be set to 1. +In order to successfully upgrade your existing modules, your `AppModule`s must implement the function `ConsensusVersion() uint64`. The `uint64` returned will serve as the consensus version and should be hard coded by the module developer. This number will serve as a state-breaking version of each app module, so it *MUST* be incremented on each consensus-breaking change introduced by the module. The initial version *MUST* be set to 1, with the exception of situations where an upgrade introduces an entirely new module. More on that [here](#adding-new-modules-in-an-upgrade) ## Registering Migrations @@ -46,3 +46,10 @@ func (m Migrator) Migrate1to2(ctx sdk.Context) error { ``` If you would like to see example code of changes implemented in a migration, you can check out the code [here](https://github.com/cosmos/cosmos-sdk/blob/36f68eb9e041e20a5bb47e216ac5eb8b91f95471/x/bank/legacy/v043/store.go#L41-L62). For context, this introduced migrations of the bank store that updated addresses to be prefixed by their length in bytes as perscribed in (ADR-028)[../architecture/adr-028-public-key-addresses.md]. + + +## Adding New Modules In Upgrades + +When introducing a new module to your application during an upgrade, your `AppModule` must begin at consensus version 0. When an upgrade is being executed, all modules with consensus versions set to 0 will be recognized as new modules. This will result in the new module's `InitGenesis` function executing. This allows new modules to setup some inital state. The result of this will set the `AppModule`s consensus version to `1`. + +If your module does not require any initial state, you do not need to implement any `InitGenesis`. The `InitGenesis` will result in a no-op, but still upgrade your consensus version to `1` for future upgrades. \ No newline at end of file From 7e1c59e8fd2714c258f1f8ba035d102efd6712b4 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Tue, 30 Mar 2021 10:47:48 -0700 Subject: [PATCH 18/43] fix --- docs/building-modules/upgrade.md | 7 ------- docs/core/upgrade.md | 33 ++++++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+), 7 deletions(-) diff --git a/docs/building-modules/upgrade.md b/docs/building-modules/upgrade.md index 9cb8ea1be6c7..85d4c5cec673 100644 --- a/docs/building-modules/upgrade.md +++ b/docs/building-modules/upgrade.md @@ -46,10 +46,3 @@ func (m Migrator) Migrate1to2(ctx sdk.Context) error { ``` If you would like to see example code of changes implemented in a migration, you can check out the code [here](https://github.com/cosmos/cosmos-sdk/blob/36f68eb9e041e20a5bb47e216ac5eb8b91f95471/x/bank/legacy/v043/store.go#L41-L62). For context, this introduced migrations of the bank store that updated addresses to be prefixed by their length in bytes as perscribed in (ADR-028)[../architecture/adr-028-public-key-addresses.md]. - - -## Adding New Modules In Upgrades - -When introducing a new module to your application during an upgrade, your `AppModule` must begin at consensus version 0. When an upgrade is being executed, all modules with consensus versions set to 0 will be recognized as new modules. This will result in the new module's `InitGenesis` function executing. This allows new modules to setup some inital state. The result of this will set the `AppModule`s consensus version to `1`. - -If your module does not require any initial state, you do not need to implement any `InitGenesis`. The `InitGenesis` will result in a no-op, but still upgrade your consensus version to `1` for future upgrades. \ No newline at end of file diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 6e89a3ca486a..622ffe87d27f 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -41,3 +41,36 @@ type UpgradeHandler func(ctx sdk.Context, plan Plan, versionMap VersionMap) (Ver ## Running Migrations In practice, the handlers should simply call and return the values from the `app.mm.RunMigrations` function. The `RunMigrations` function should be passed the `VersionMap` from the `UpgradeHandler`. With this, the `RunMigration` function will loop through the `VersionMap`, and for any current app module who's consensus version is greater than its corresponding value in the `VersionMap`, have its migration scripts ran. To learn how to configure migration scripts, refer to (this guide)[../building-modules/upgrade.md]. + +## Adding New Modules In Upgrades + +New modules can be introduced to the application during an upgrade. The SDK recognizes new modules during upgrades and will call the corresponding module's `DefaultGenesis` function to setup the its initial state. This can be skipped if the module does not require any inital state. + +If you wish to overwrite the default behavior of running InitGenesis during an upgrade for new modules, make sure to pass the latest `ConsensusVersion` of the new module into the returned `module.VersionMap`. This will then skip running InitGenesis for the module: + +```go +// Foo is a new module being introduced +// in this upgrade plan +import foo "github.com/my/module/foo" + +app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) { + // We make sure to set foo's version to the latest ConsensusVersion in the VersionMap. + // This will skip running InitGenesis on Foo + vm["foo"] = foo.AppModule{}.ConsensusVersion() + + return app.mm.RunMigrations(ctx, vm) +}) +``` + +Using the same method, you can also run InitGenesis on your new module with a custom genesis state: + +```go +app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) { + vm["foo"] = foo.AppModule{}.ConsensusVersion() + + // Run custom InitGenesis for foo + app.mm["foo"].InitGenesis(ctx, app.appCodec, myCustomGenesisState) + + return app.mm.RunMigrations(ctx, vm) +}) +``` \ No newline at end of file From b91e93ff2a8274dcef9eafe3149cc136342e075e Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Tue, 30 Mar 2021 10:52:06 -0700 Subject: [PATCH 19/43] remove line --- docs/building-modules/upgrade.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/building-modules/upgrade.md b/docs/building-modules/upgrade.md index 85d4c5cec673..78f6af1a6492 100644 --- a/docs/building-modules/upgrade.md +++ b/docs/building-modules/upgrade.md @@ -10,8 +10,7 @@ In-place store migrations allow your modules to smoothly transition to new versi - [In-Place Store Migration](../core-concepts/upgrade.md) {prereq} ## Consensus Version -In order to successfully upgrade your existing modules, your `AppModule`s must implement the function `ConsensusVersion() uint64`. The `uint64` returned will serve as the consensus version and should be hard coded by the module developer. This number will serve as a state-breaking version of each app module, so it *MUST* be incremented on each consensus-breaking change introduced by the module. The initial version *MUST* be set to 1, with the exception of situations where an upgrade introduces an entirely new module. More on that [here](#adding-new-modules-in-an-upgrade) - +In order to successfully upgrade your existing modules, your `AppModule`s must implement the function `ConsensusVersion() uint64`. The `uint64` returned will serve as the consensus version and should be hard coded by the module developer. This number will serve as a state-breaking version of each app module, so it *MUST* be incremented on each consensus-breaking change introduced by the module. The initial version *MUST* be set to 1. ## Registering Migrations To register the functionality that takes place during a module upgrade, we must register which migrations we want to take place. This takes place in the `Configurator` via the `RegisterMigration` method. The `AppModule`s have a reference to the configurator in the `RegisterServices` method. We can register a number of migrations here, however, if more than one migration script is registered, it is important they are listed in increasing order. Additionally, we must ensure there are enough migrations that will lead to the desired consensus version. For example, if we wanted to migrate to version 3 of a module, we would need to register a separate migration for both version 1 and 2 as shown below. From e3f45fe33df55ee5586ce695014e88064a85373b Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Wed, 31 Mar 2021 09:00:52 -0700 Subject: [PATCH 20/43] Update docs/core/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/upgrade.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 622ffe87d27f..2f21b4edcdcf 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -44,7 +44,7 @@ In practice, the handlers should simply call and return the values from the `app ## Adding New Modules In Upgrades -New modules can be introduced to the application during an upgrade. The SDK recognizes new modules during upgrades and will call the corresponding module's `DefaultGenesis` function to setup the its initial state. This can be skipped if the module does not require any inital state. +New modules can be introduced to the application during an upgrade. The SDK recognizes new modules during upgrades and will call the corresponding module's `InitGenesis` function to setup the its initial state. This can be skipped if the module does not require any inital state. If you wish to overwrite the default behavior of running InitGenesis during an upgrade for new modules, make sure to pass the latest `ConsensusVersion` of the new module into the returned `module.VersionMap`. This will then skip running InitGenesis for the module: @@ -73,4 +73,4 @@ app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgrad return app.mm.RunMigrations(ctx, vm) }) -``` \ No newline at end of file +``` From cf287779c9b8d5e8027e7d5ba6c3a03526f3cdbb Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Wed, 31 Mar 2021 09:01:20 -0700 Subject: [PATCH 21/43] Update docs/building-modules/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/building-modules/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/building-modules/upgrade.md b/docs/building-modules/upgrade.md index 78f6af1a6492..59d017407804 100644 --- a/docs/building-modules/upgrade.md +++ b/docs/building-modules/upgrade.md @@ -44,4 +44,4 @@ func (m Migrator) Migrate1to2(ctx sdk.Context) error { } ``` -If you would like to see example code of changes implemented in a migration, you can check out the code [here](https://github.com/cosmos/cosmos-sdk/blob/36f68eb9e041e20a5bb47e216ac5eb8b91f95471/x/bank/legacy/v043/store.go#L41-L62). For context, this introduced migrations of the bank store that updated addresses to be prefixed by their length in bytes as perscribed in (ADR-028)[../architecture/adr-028-public-key-addresses.md]. +If you would like to see example code of changes implemented in a migration, you can check out the code [here](https://github.com/cosmos/cosmos-sdk/blob/36f68eb9e041e20a5bb47e216ac5eb8b91f95471/x/bank/legacy/v043/store.go#L41-L62). For context, this introduced migrations of the bank store that updated addresses to be prefixed by their length in bytes as perscribed in [ADR-028](../architecture/adr-028-public-key-addresses.md). From 52414d3658c828cc43be4792055e59298ae7a566 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Wed, 31 Mar 2021 10:13:59 -0700 Subject: [PATCH 22/43] SetUpgradeHandler example snippet --- docs/core/upgrade.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 622ffe87d27f..e7312e2f2c2c 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -42,6 +42,21 @@ type UpgradeHandler func(ctx sdk.Context, plan Plan, versionMap VersionMap) (Ver In practice, the handlers should simply call and return the values from the `app.mm.RunMigrations` function. The `RunMigrations` function should be passed the `VersionMap` from the `UpgradeHandler`. With this, the `RunMigration` function will loop through the `VersionMap`, and for any current app module who's consensus version is greater than its corresponding value in the `VersionMap`, have its migration scripts ran. To learn how to configure migration scripts, refer to (this guide)[../building-modules/upgrade.md]. +When upgrades are executed, they refer to the functionality described in an upgrade handler. All upgrade handlers should describe the logic needed for the upgrade plan, and end with returning the values from a call to `app.RunMigrations(ctx, vm)`. This will return the updated `VersionMap` to be saved to the upgrade module's store. + +```golang +app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) { + + // ... + // do upgrade logic + // ... + + // RunMigrations returns the VersionMap + // with the updated module ConsensusVersions + return app.RunMigrations(ctx, vm) +}) +``` + ## Adding New Modules In Upgrades New modules can be introduced to the application during an upgrade. The SDK recognizes new modules during upgrades and will call the corresponding module's `DefaultGenesis` function to setup the its initial state. This can be skipped if the module does not require any inital state. From 5bc3bf6292d30591187c661b8c81109bebcba115 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Fri, 2 Apr 2021 15:29:02 -0700 Subject: [PATCH 23/43] -general clean up of docs -include PR #9007 information --- docs/building-modules/upgrade.md | 9 ++++--- docs/core/upgrade.md | 41 +++++++++++++++++++------------- 2 files changed, 28 insertions(+), 22 deletions(-) diff --git a/docs/building-modules/upgrade.md b/docs/building-modules/upgrade.md index 59d017407804..ad8ae3dadca2 100644 --- a/docs/building-modules/upgrade.md +++ b/docs/building-modules/upgrade.md @@ -10,7 +10,8 @@ In-place store migrations allow your modules to smoothly transition to new versi - [In-Place Store Migration](../core-concepts/upgrade.md) {prereq} ## Consensus Version -In order to successfully upgrade your existing modules, your `AppModule`s must implement the function `ConsensusVersion() uint64`. The `uint64` returned will serve as the consensus version and should be hard coded by the module developer. This number will serve as a state-breaking version of each app module, so it *MUST* be incremented on each consensus-breaking change introduced by the module. The initial version *MUST* be set to 1. +In order to successfully upgrade your existing modules, your `AppModule`s must implement the function `ConsensusVersion() uint64`. The versions should be hard coded by the module developer. Consensus versions serve as a state-breaking versions of app modules, and will be incremented when the module is upgraded. The initial version *MUST* be set to 1. + ## Registering Migrations To register the functionality that takes place during a module upgrade, we must register which migrations we want to take place. This takes place in the `Configurator` via the `RegisterMigration` method. The `AppModule`s have a reference to the configurator in the `RegisterServices` method. We can register a number of migrations here, however, if more than one migration script is registered, it is important they are listed in increasing order. Additionally, we must ensure there are enough migrations that will lead to the desired consensus version. For example, if we wanted to migrate to version 3 of a module, we would need to register a separate migration for both version 1 and 2 as shown below. @@ -31,11 +32,9 @@ Since these migrations are functions that need access to a Keeper's store, we us +++ https://github.com/cosmos/cosmos-sdk/blob/6ac8898fec9bd7ea2c1e5c79e0ed0c3f827beb55/x/bank/keeper/migrations.go#L8-L21 -In addition to the `Migrator` wrapper, we also define our migration scripts. More on that below. - ## Writing Migration Scripts -In order to define the functionality that takes place during an upgrade, we will write a migration script. Since migration scripts will manipulate legacy code, we place these functions in a `legacy/` directory. For example, if we wanted to write migration scripts for a module named `bank`, we would place the functions in `x/bank/legacy/`. We recommend the following naming convention for these functions: +In order to define the functionality that takes place during an upgrade, we will write a migration script. Since migration scripts will manipulate legacy code, we place these functions in a `legacy/` directory. For example, if we wanted to write migration scripts for the bank module, we would place the functions in `x/bank/legacy/`. We recommend the following naming convention for these functions: ```golang // Migrating bank module from version 1 to 2 @@ -44,4 +43,4 @@ func (m Migrator) Migrate1to2(ctx sdk.Context) error { } ``` -If you would like to see example code of changes implemented in a migration, you can check out the code [here](https://github.com/cosmos/cosmos-sdk/blob/36f68eb9e041e20a5bb47e216ac5eb8b91f95471/x/bank/legacy/v043/store.go#L41-L62). For context, this introduced migrations of the bank store that updated addresses to be prefixed by their length in bytes as perscribed in [ADR-028](../architecture/adr-028-public-key-addresses.md). +If you would like to see example code of changes implemented in a migration, you can check out the code [here](https://github.com/cosmos/cosmos-sdk/blob/36f68eb9e041e20a5bb47e216ac5eb8b91f95471/x/bank/legacy/v043/store.go#L41-L62). For context, this introduced migrations of the bank store that updated addresses to be prefixed by their length in bytes as outlined in [ADR-028](../architecture/adr-028-public-key-addresses.md). diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index a61161a1c113..938192b7f340 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -6,11 +6,7 @@ order: 15 Upgrade your app modules smoothly with custom in-place migration logic. {synopsis} -The Cosmos SDK currently has two ways to perform upgrades. The first way is by exporting the entire application state to a JSON file using the `simd export` CLI command, making changes, and then starting a new binary with the changed JSON file as the genesis file. The second way is by performing upgrades in place, significantly decreasing the time needed to perform upgrades for chains with a larger state. The following will guide you on how to setup your application to take advantage of the second method described above. - -## Enabling Upgrades - -To enable your application to conform to the upgrade module's specifications, a few changes need to be made to your application. +The Cosmos SDK currently has two ways to perform upgrades. The first way is by exporting the entire application state to a JSON file using the `export` CLI command, making changes, and then starting a new binary with the changed JSON file as the genesis file. The second way is by performing upgrades in place, significantly decreasing the time needed to perform upgrades for chains with a larger state. The following guide will provide you with the necessary information in order to setup your application to take advantage of in-place upgrades. ## Genesis State @@ -24,27 +20,34 @@ func (app *MyApp) InitChainer(ctx sdk.Context, req abci.RequestInitChain) abci.R } ``` +Using this information, the SDK will be able to detect when modules with newer versions are introduced to the app. + ### Consensus Version -The consensus version is defined on each app module by the module developer. It serves as the breaking change version of the module. +The consensus version is defined on each app module by the module developer. It serves as the breaking change version of the module. This number is how the SDK identifies which modules to upgrade. For example, if the bank module was version 2, and an upgrade introduces bank module 3, the SDK will know to upgrade the bank module. ### Version Map -The version map is a mapping of module names to consensus versions. The map is persisted to state for use during in-place migrations. +The version map is a mapping of module names to consensus versions. The map is persisted to state for use during in-place migrations. When migrations finish, the updated version map is persisted to state. ## Upgrade Handlers -Upgrades utilize an `UpgradeHandler` to facilitate migrations. `UpgradeHandler`s are functions implemented by the app developer that conform to the following function signature. +Upgrades utilize an `UpgradeHandler` to facilitate migrations. `UpgradeHandler`s are functions implemented by the app developer that conform to the following function signature. These functions utiltize a `VersionMap` containing all the module versions to determine which modules need upgrading. ```golang -type UpgradeHandler func(ctx sdk.Context, plan Plan, versionMap VersionMap) (VersionMap, error) +type UpgradeHandler func(ctx sdk.Context, plan Plan, fromVM VersionMap) (VersionMap, error) ``` -## Running Migrations +Inside these functions, you should perform any upgrade logic you wish to include in the provided `plan`. All upgrade handler functions should end with the following line of code: -In practice, the handlers should simply call and return the values from the `app.mm.RunMigrations` function. The `RunMigrations` function should be passed the `VersionMap` from the `UpgradeHandler`. With this, the `RunMigration` function will loop through the `VersionMap`, and for any current app module who's consensus version is greater than its corresponding value in the `VersionMap`, have its migration scripts ran. To learn how to configure migration scripts, refer to (this guide)[../building-modules/upgrade.md]. +```golang + return app.mm.RunMigrations(ctx, cfg, fromVM) +``` -When upgrades are executed, they refer to the functionality described in an upgrade handler. All upgrade handlers should describe the logic needed for the upgrade plan, and end with returning the values from a call to `app.RunMigrations(ctx, vm)`. This will return the updated `VersionMap` to be saved to the upgrade module's store. +## Running Migrations + +Migrations are run inside of an `UpgradeHandler` via `app.RunMigrations(ctx, cfg, vm)`. As described above, `UpgradeHandler`s are functions which describe the functionality to occur during an upgrade. The `RunMigration` function will loop through the `VersionMap` argument, and run the migration scripts for any versions that are less than the new binary's app module versions. Once the migrations are finished, a new `VersionMap` will be returned to persist the upgraded module versions to state. ```golang +cfg := module.NewConfigurator(...) app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) { // ... @@ -57,11 +60,13 @@ app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgrad }) ``` +To learn more about configuring migration scripts, refer to this (guide)[../building-modules/upgrade.md]. + ## Adding New Modules In Upgrades -New modules can be introduced to the application during an upgrade. The SDK recognizes new modules during upgrades and will call the corresponding module's `InitGenesis` function to setup the its initial state. This can be skipped if the module does not require any inital state. +Entirely new modules can be introduced to the application during an upgrade. The SDK recognizes new modules during upgrades and will call the corresponding module's `InitGenesis` function to setup its initial state. This can be skipped if the module does not require any inital state. Otherwise, it is important to implement `InitGenesis` for new modules to successfully upgrade your application without error. -If you wish to overwrite the default behavior of running InitGenesis during an upgrade for new modules, make sure to pass the latest `ConsensusVersion` of the new module into the returned `module.VersionMap`. This will then skip running InitGenesis for the module: +In the scenario where your application does not need any inital state via `InitGenesis`, you must take extra steps to ensure `InitGenesis` is skipped to avoid errors. To do so, you simply need to update the value of the module version in the `VersionMap` in the `UpgradeHandler`. ```go // Foo is a new module being introduced @@ -73,19 +78,21 @@ app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgrad // This will skip running InitGenesis on Foo vm["foo"] = foo.AppModule{}.ConsensusVersion() - return app.mm.RunMigrations(ctx, vm) + return app.mm.RunMigrations(ctx, cfg, vm) }) ``` -Using the same method, you can also run InitGenesis on your new module with a custom genesis state: +Using a similar method, you can also run InitGenesis on your new module with a custom genesis state: ```go +import foo "github.com/my/module/foo" + app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) { vm["foo"] = foo.AppModule{}.ConsensusVersion() // Run custom InitGenesis for foo app.mm["foo"].InitGenesis(ctx, app.appCodec, myCustomGenesisState) - return app.mm.RunMigrations(ctx, vm) + return app.mm.RunMigrations(ctx, cfg, vm) }) ``` From e054d77391271c68547f9a48a9ea6faa84fd6038 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Tue, 6 Apr 2021 08:54:50 -0700 Subject: [PATCH 24/43] Update docs/core/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/upgrade.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 938192b7f340..5fedb827dd2b 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -4,6 +4,10 @@ order: 15 # In-Place Store Migrations +::: warning +Please make sure you read this whole document and fully understand in-place store migrations before running them on a live chain. +::: + Upgrade your app modules smoothly with custom in-place migration logic. {synopsis} The Cosmos SDK currently has two ways to perform upgrades. The first way is by exporting the entire application state to a JSON file using the `export` CLI command, making changes, and then starting a new binary with the changed JSON file as the genesis file. The second way is by performing upgrades in place, significantly decreasing the time needed to perform upgrades for chains with a larger state. The following guide will provide you with the necessary information in order to setup your application to take advantage of in-place upgrades. From 8b1c7adeafeaee4fd6dc97b23ab0a677c78d1091 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Tue, 6 Apr 2021 08:58:00 -0700 Subject: [PATCH 25/43] Update docs/core/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 5fedb827dd2b..181a8b2a8928 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -10,7 +10,7 @@ Please make sure you read this whole document and fully understand in-place stor Upgrade your app modules smoothly with custom in-place migration logic. {synopsis} -The Cosmos SDK currently has two ways to perform upgrades. The first way is by exporting the entire application state to a JSON file using the `export` CLI command, making changes, and then starting a new binary with the changed JSON file as the genesis file. The second way is by performing upgrades in place, significantly decreasing the time needed to perform upgrades for chains with a larger state. The following guide will provide you with the necessary information in order to setup your application to take advantage of in-place upgrades. +The Cosmos SDK currently has two methods to perform upgrades. The first method is by exporting the entire application state to a JSON file using the `export` CLI command, making changes, and then starting a new binary with the changed JSON file as the genesis file. More details on this method can be found in the [chain upgrade guide](../migrations/chain-upgrade-guide-040.md#upgrade-procedure). The second method, introduced in v0.43, works by performing upgrades in place, significantly decreasing the time needed to perform upgrades for chains with a larger state. The following guide will provide you with the necessary information in order to setup your application to take advantage of in-place upgrades. ## Genesis State From a63942b24a96005b9b207d85b09737f4cedd7cdd Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Tue, 6 Apr 2021 08:58:27 -0700 Subject: [PATCH 26/43] Update docs/core/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 181a8b2a8928..24675fac549f 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -64,7 +64,7 @@ app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgrad }) ``` -To learn more about configuring migration scripts, refer to this (guide)[../building-modules/upgrade.md]. +To learn more about configuring migration scripts for your modules, refer to this [guide](../building-modules/upgrade.md). ## Adding New Modules In Upgrades From d4472a5cf4451724007e14d3c2995159692c3ac5 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Tue, 6 Apr 2021 09:04:46 -0700 Subject: [PATCH 27/43] Update docs/core/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/upgrade.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 24675fac549f..7ba867d1d541 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -12,6 +12,12 @@ Upgrade your app modules smoothly with custom in-place migration logic. {synopsi The Cosmos SDK currently has two methods to perform upgrades. The first method is by exporting the entire application state to a JSON file using the `export` CLI command, making changes, and then starting a new binary with the changed JSON file as the genesis file. More details on this method can be found in the [chain upgrade guide](../migrations/chain-upgrade-guide-040.md#upgrade-procedure). The second method, introduced in v0.43, works by performing upgrades in place, significantly decreasing the time needed to perform upgrades for chains with a larger state. The following guide will provide you with the necessary information in order to setup your application to take advantage of in-place upgrades. +## Tracking Module Versions + +Each module gets assigned a consensus version by the module developer, which serves as the breaking change version of the module. The SDK keeps track of all modules' consensus versions in the x/upgrade's `VersionMap` store. During an upgrade, the SDK calculates the difference between the old `VersionMap` stored in state and the new `VersionMap`. For each difference found, the SDK will run module-specific migrations and increment the respective consensus version of each upgraded module. + +The next paragraphs detail each component of the in-place store migration process, and gives instructions on how to update your app to take advantage of this functionality. + ## Genesis State Each app module's consensus version must be saved to state on the application's genesis. This can be done by adding the following line to the `InitChainer` method in `app.go` From 47021cb6b97e9d45df6b084b8e778d30b742cb91 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Tue, 6 Apr 2021 09:05:23 -0700 Subject: [PATCH 28/43] Update docs/core/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 7ba867d1d541..f946df072670 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -20,7 +20,7 @@ The next paragraphs detail each component of the in-place store migration proces ## Genesis State -Each app module's consensus version must be saved to state on the application's genesis. This can be done by adding the following line to the `InitChainer` method in `app.go` +When starting a new chain, each module's consensus version must be saved to state during the application's genesis. This can be done by adding the following line to the `InitChainer` method in `app.go` ```diff func (app *MyApp) InitChainer(ctx sdk.Context, req abci.RequestInitChain) abci.ResponseInitChain { From 1d6c42994691847bdcaa4540d7ac4d6c14ac9452 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Tue, 6 Apr 2021 09:07:58 -0700 Subject: [PATCH 29/43] Update docs/core/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index f946df072670..d338a3c6f272 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -33,7 +33,7 @@ func (app *MyApp) InitChainer(ctx sdk.Context, req abci.RequestInitChain) abci.R Using this information, the SDK will be able to detect when modules with newer versions are introduced to the app. ### Consensus Version -The consensus version is defined on each app module by the module developer. It serves as the breaking change version of the module. This number is how the SDK identifies which modules to upgrade. For example, if the bank module was version 2, and an upgrade introduces bank module 3, the SDK will know to upgrade the bank module. +The consensus version is defined on each app module by the module developer. It serves as the breaking change version of the module. The consensus version helps to inform the SDK on which modules need to be upgraded. For example, if the bank module was version 2 and an upgrade introduces bank module 3, the SDK will know to upgrade the bank module and run its "version 2 to 3" migration script. ### Version Map The version map is a mapping of module names to consensus versions. The map is persisted to state for use during in-place migrations. When migrations finish, the updated version map is persisted to state. From 6a7d393844ae139b970a8c3134e361c001f5e2ff Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Tue, 6 Apr 2021 09:08:46 -0700 Subject: [PATCH 30/43] Update docs/core/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index d338a3c6f272..0b2c9aab2f21 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -36,7 +36,7 @@ Using this information, the SDK will be able to detect when modules with newer v The consensus version is defined on each app module by the module developer. It serves as the breaking change version of the module. The consensus version helps to inform the SDK on which modules need to be upgraded. For example, if the bank module was version 2 and an upgrade introduces bank module 3, the SDK will know to upgrade the bank module and run its "version 2 to 3" migration script. ### Version Map -The version map is a mapping of module names to consensus versions. The map is persisted to state for use during in-place migrations. When migrations finish, the updated version map is persisted to state. +The version map is a mapping of module names to consensus versions. The map is persisted to x/upgrade's state for use during in-place migrations. When migrations finish, the updated version map is persisted to state. ## Upgrade Handlers From 7810c1b239bb4e89ee3030d3167d5353abd525ac Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Tue, 6 Apr 2021 09:10:17 -0700 Subject: [PATCH 31/43] Update docs/core/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 0b2c9aab2f21..6fb9ec3d130f 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -40,7 +40,7 @@ The version map is a mapping of module names to consensus versions. The map is p ## Upgrade Handlers -Upgrades utilize an `UpgradeHandler` to facilitate migrations. `UpgradeHandler`s are functions implemented by the app developer that conform to the following function signature. These functions utiltize a `VersionMap` containing all the module versions to determine which modules need upgrading. +Upgrades utilize an `UpgradeHandler` to facilitate migrations. `UpgradeHandler`s are functions implemented by the app developer that conform to the following function signature. These functions retrieve the `VersionMap` from x/upgrade's state, and return the new `VersionMap` to be stored in x/upgrade after the upgrade. The diff between the two `VersionMap`s determines which modules need upgrading. ```golang type UpgradeHandler func(ctx sdk.Context, plan Plan, fromVM VersionMap) (VersionMap, error) From c217fa58ac1b1534ff198df0e30d72866dcd126d Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Tue, 6 Apr 2021 09:10:27 -0700 Subject: [PATCH 32/43] Update docs/core/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 6fb9ec3d130f..e5f21433d112 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -54,7 +54,7 @@ Inside these functions, you should perform any upgrade logic you wish to include ## Running Migrations -Migrations are run inside of an `UpgradeHandler` via `app.RunMigrations(ctx, cfg, vm)`. As described above, `UpgradeHandler`s are functions which describe the functionality to occur during an upgrade. The `RunMigration` function will loop through the `VersionMap` argument, and run the migration scripts for any versions that are less than the new binary's app module versions. Once the migrations are finished, a new `VersionMap` will be returned to persist the upgraded module versions to state. +Migrations are run inside of an `UpgradeHandler` via `app.mm.RunMigrations(ctx, cfg, vm)`. As described above, `UpgradeHandler`s are functions which describe the functionality to occur during an upgrade. The `RunMigration` function will loop through the `VersionMap` argument, and run the migration scripts for any versions that are less than the new binary's app module versions. Once the migrations are finished, a new `VersionMap` will be returned to persist the upgraded module versions to state. ```golang cfg := module.NewConfigurator(...) From 09863484e1feeaf79613b09ebf11dd1f989bd48a Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Tue, 6 Apr 2021 09:10:35 -0700 Subject: [PATCH 33/43] Update docs/core/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index e5f21433d112..f8d1c5ef17a9 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -66,7 +66,7 @@ app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgrad // RunMigrations returns the VersionMap // with the updated module ConsensusVersions - return app.RunMigrations(ctx, vm) + return app.mm.RunMigrations(ctx, vm) }) ``` From 80260f5455afbc3e9553b1752967f6f74c461e4e Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Tue, 6 Apr 2021 09:11:55 -0700 Subject: [PATCH 34/43] Update docs/core/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index f8d1c5ef17a9..cd5714cbff66 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -74,7 +74,7 @@ To learn more about configuring migration scripts for your modules, refer to thi ## Adding New Modules In Upgrades -Entirely new modules can be introduced to the application during an upgrade. The SDK recognizes new modules during upgrades and will call the corresponding module's `InitGenesis` function to setup its initial state. This can be skipped if the module does not require any inital state. Otherwise, it is important to implement `InitGenesis` for new modules to successfully upgrade your application without error. +Entirely new modules can be introduced to the application during an upgrade. The SDK recognizes new modules during upgrades because their consensus version in the `fromVM` `VersionMap` is 0. In this case, `RunMigrations` will call the corresponding module's `InitGenesis` function to setup its initial state. This can be skipped if the module does not require any inital state. Otherwise, it is important to implement `InitGenesis` for new modules to successfully upgrade your application without error. In the scenario where your application does not need any inital state via `InitGenesis`, you must take extra steps to ensure `InitGenesis` is skipped to avoid errors. To do so, you simply need to update the value of the module version in the `VersionMap` in the `UpgradeHandler`. From f61c7fa827d931484d941035bfedd466c9f5afc2 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Tue, 6 Apr 2021 09:12:16 -0700 Subject: [PATCH 35/43] Update docs/core/upgrade.md Co-authored-by: Amaury <1293565+amaurym@users.noreply.github.com> --- docs/core/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index cd5714cbff66..5632dac568e1 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -92,7 +92,7 @@ app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgrad }) ``` -Using a similar method, you can also run InitGenesis on your new module with a custom genesis state: +Using a similar method, you can also run `InitGenesis` on your new module with a custom genesis state: ```go import foo "github.com/my/module/foo" From 6c705071b561d68dc5b12416bb3c8e946accd8ff Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Wed, 7 Apr 2021 14:03:50 -0700 Subject: [PATCH 36/43] Apply suggestions from code review Co-authored-by: Barrie Byron --- docs/building-modules/upgrade.md | 23 ++++++++++++++++------- docs/core/README.md | 2 +- docs/core/upgrade.md | 31 +++++++++++++++++-------------- 3 files changed, 34 insertions(+), 22 deletions(-) diff --git a/docs/building-modules/upgrade.md b/docs/building-modules/upgrade.md index ad8ae3dadca2..47d776b4d0ac 100644 --- a/docs/building-modules/upgrade.md +++ b/docs/building-modules/upgrade.md @@ -4,17 +4,26 @@ order: 13 # In-Place Store Migrations -In-place store migrations allow your modules to smoothly transition to new versions with breaking changes. This document outlines how to build modules to take advantage of this functionality. {synopsis} -## Pre-requisite Readings +In-place store migrations allow your modules to upgrade to new versions that include breaking changes. This document outlines how to build modules to take advantage of this functionality. {synopsis} +## Prerequisite Readings - [In-Place Store Migration](../core-concepts/upgrade.md) {prereq} ## Consensus Version -In order to successfully upgrade your existing modules, your `AppModule`s must implement the function `ConsensusVersion() uint64`. The versions should be hard coded by the module developer. Consensus versions serve as a state-breaking versions of app modules, and will be incremented when the module is upgraded. The initial version *MUST* be set to 1. +Successful upgrades of existing modules require your `AppModule` to implement the function `ConsensusVersion() uint64`. + +- The versions must be hard-coded by the module developer. +- The initial version **must** be set to 1. + +Consensus versions serve as state-breaking versions of app modules and are incremented when the module is upgraded. ## Registering Migrations -To register the functionality that takes place during a module upgrade, we must register which migrations we want to take place. This takes place in the `Configurator` via the `RegisterMigration` method. The `AppModule`s have a reference to the configurator in the `RegisterServices` method. We can register a number of migrations here, however, if more than one migration script is registered, it is important they are listed in increasing order. Additionally, we must ensure there are enough migrations that will lead to the desired consensus version. For example, if we wanted to migrate to version 3 of a module, we would need to register a separate migration for both version 1 and 2 as shown below. +To register the functionality that takes place during a module upgrade, you must register which migrations we want to take place. + +Migration registration takes place in the `Configurator` using the `RegisterMigration` method. The `AppModule` reference to the configurator is in the `RegisterServices` method. + +You can register one or more migrations. If you register more than one migration script, list the migrations in increasing order and ensure there are enough migrations that lead to the desired consensus version. For example, to migrate to version 3 of a module, register separate migrations for version 1 and version 2 as shown in the following example: ```golang func (am AppModule) RegisterServices(cfg module.Configurator) { @@ -28,13 +37,13 @@ func (am AppModule) RegisterServices(cfg module.Configurator) { } ``` -Since these migrations are functions that need access to a Keeper's store, we use a wrapper around the keepers called `Migrator`. An example of this can be found here: +Since these migrations are functions that need access to a Keeper's store, use a wrapper around the keepers called `Migrator` as shown in this example: +++ https://github.com/cosmos/cosmos-sdk/blob/6ac8898fec9bd7ea2c1e5c79e0ed0c3f827beb55/x/bank/keeper/migrations.go#L8-L21 ## Writing Migration Scripts -In order to define the functionality that takes place during an upgrade, we will write a migration script. Since migration scripts will manipulate legacy code, we place these functions in a `legacy/` directory. For example, if we wanted to write migration scripts for the bank module, we would place the functions in `x/bank/legacy/`. We recommend the following naming convention for these functions: +To define the functionality that takes place during an upgrade, write a migration script. Since migration scripts manipulate legacy code, place these functions in a `legacy/` directory. For example, to write migration scripts for the bank module, place the functions in `x/bank/legacy/`. Use the recommended naming convention for these functions. For example, `v043bank` is the script that migrates this legacy package `x/bank/legacy/v043: ```golang // Migrating bank module from version 1 to 2 @@ -43,4 +52,4 @@ func (m Migrator) Migrate1to2(ctx sdk.Context) error { } ``` -If you would like to see example code of changes implemented in a migration, you can check out the code [here](https://github.com/cosmos/cosmos-sdk/blob/36f68eb9e041e20a5bb47e216ac5eb8b91f95471/x/bank/legacy/v043/store.go#L41-L62). For context, this introduced migrations of the bank store that updated addresses to be prefixed by their length in bytes as outlined in [ADR-028](../architecture/adr-028-public-key-addresses.md). +To see example code of changes that were implemented in a migration of balance keys, check out the [func migrateBalanceKeys](https://github.com/cosmos/cosmos-sdk/blob/36f68eb9e041e20a5bb47e216ac5eb8b91f95471/x/bank/legacy/v043/store.go#L41-L62) code. For context, this code introduced migrations of the bank store that updated addresses to be prefixed by their length in bytes as outlined in [ADR-028](../architecture/adr-028-public-key-addresses.md). diff --git a/docs/core/README.md b/docs/core/README.md index e3e9d308fe92..a6853b41802a 100644 --- a/docs/core/README.md +++ b/docs/core/README.md @@ -25,4 +25,4 @@ This repository contains reference documentation on the core concepts of the Cos 15. [In-Place Store Migrations](./upgrade.md) After reading about the core concepts, check the [IBC documentation](../ibc/README.md) to learn more -about the IBC core concepts and how to integrate it to you application. +about the IBC core concepts and how to integrate IBC in your application. diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 5632dac568e1..39dc0cdcecb5 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -5,22 +5,25 @@ order: 15 # In-Place Store Migrations ::: warning -Please make sure you read this whole document and fully understand in-place store migrations before running them on a live chain. +Read and understand all of the in-place store migration documentation before you run a migration on a live chain. ::: Upgrade your app modules smoothly with custom in-place migration logic. {synopsis} -The Cosmos SDK currently has two methods to perform upgrades. The first method is by exporting the entire application state to a JSON file using the `export` CLI command, making changes, and then starting a new binary with the changed JSON file as the genesis file. More details on this method can be found in the [chain upgrade guide](../migrations/chain-upgrade-guide-040.md#upgrade-procedure). The second method, introduced in v0.43, works by performing upgrades in place, significantly decreasing the time needed to perform upgrades for chains with a larger state. The following guide will provide you with the necessary information in order to setup your application to take advantage of in-place upgrades. +The Cosmos SDK uses two methods to perform upgrades. + +- Exporting the entire application state to a JSON file using the `export` CLI command, making changes, and then starting a new binary with the changed JSON file as the genesis file. See the [Chain Upgrade Guide](../migrations/chain-upgrade-guide-040.md#upgrade-procedure). + +- Version v0.43 and later can perform upgrades in place to significantly decrease the upgrade time for chains with a larger state. Use the [In-Place Store Migrations](./upgrade.md) guide to set up your application to take advantage of in-place upgrades. ## Tracking Module Versions -Each module gets assigned a consensus version by the module developer, which serves as the breaking change version of the module. The SDK keeps track of all modules' consensus versions in the x/upgrade's `VersionMap` store. During an upgrade, the SDK calculates the difference between the old `VersionMap` stored in state and the new `VersionMap`. For each difference found, the SDK will run module-specific migrations and increment the respective consensus version of each upgraded module. +Each module gets assigned a consensus version by the module developer. The consensus version serves as the breaking change version of the module. The SDK keeps track of all module consensus versions in the x/upgrade `VersionMap` store. During an upgrade, the difference between the old `VersionMap` stored in state and the new `VersionMap` is calculated by the Cosmos SDK. For each identified difference, the module-specific migrations are run and the respective consensus version of each upgraded module is incremented. -The next paragraphs detail each component of the in-place store migration process, and gives instructions on how to update your app to take advantage of this functionality. ## Genesis State -When starting a new chain, each module's consensus version must be saved to state during the application's genesis. This can be done by adding the following line to the `InitChainer` method in `app.go` +When starting a new chain, the consensus version of each module must be saved to state during the application's genesis. To save the consensus version, add the following line to the `InitChainer` method in `app.go`: ```diff func (app *MyApp) InitChainer(ctx sdk.Context, req abci.RequestInitChain) abci.ResponseInitChain { @@ -30,23 +33,23 @@ func (app *MyApp) InitChainer(ctx sdk.Context, req abci.RequestInitChain) abci.R } ``` -Using this information, the SDK will be able to detect when modules with newer versions are introduced to the app. +This information is used by the Cosmos SDK to detect when modules with newer versions are introduced to the app. ### Consensus Version -The consensus version is defined on each app module by the module developer. It serves as the breaking change version of the module. The consensus version helps to inform the SDK on which modules need to be upgraded. For example, if the bank module was version 2 and an upgrade introduces bank module 3, the SDK will know to upgrade the bank module and run its "version 2 to 3" migration script. +The consensus version is defined on each app module by the module developer and serves as the breaking change version of the module. The consensus version informs the SDK on which modules need to be upgraded. For example, if the bank module was version 2 and an upgrade introduces bank module 3, the SDK upgrades the bank module and runs the "version 2 to 3" migration script. ### Version Map The version map is a mapping of module names to consensus versions. The map is persisted to x/upgrade's state for use during in-place migrations. When migrations finish, the updated version map is persisted to state. ## Upgrade Handlers -Upgrades utilize an `UpgradeHandler` to facilitate migrations. `UpgradeHandler`s are functions implemented by the app developer that conform to the following function signature. These functions retrieve the `VersionMap` from x/upgrade's state, and return the new `VersionMap` to be stored in x/upgrade after the upgrade. The diff between the two `VersionMap`s determines which modules need upgrading. +Upgrades use an `UpgradeHandler` to facilitate migrations. The `UpgradeHandler` functions implemented by the app developer must conform to the following function signature. These functions retrieve the `VersionMap` from x/upgrade's state and return the new `VersionMap` to be stored in x/upgrade after the upgrade. The diff between the two `VersionMap`s determines which modules need upgrading. ```golang type UpgradeHandler func(ctx sdk.Context, plan Plan, fromVM VersionMap) (VersionMap, error) ``` -Inside these functions, you should perform any upgrade logic you wish to include in the provided `plan`. All upgrade handler functions should end with the following line of code: +Inside these functions, you must perform any upgrade logic to include in the provided `plan`. All upgrade handler functions must end with the following line of code: ```golang return app.mm.RunMigrations(ctx, cfg, fromVM) @@ -54,7 +57,7 @@ Inside these functions, you should perform any upgrade logic you wish to include ## Running Migrations -Migrations are run inside of an `UpgradeHandler` via `app.mm.RunMigrations(ctx, cfg, vm)`. As described above, `UpgradeHandler`s are functions which describe the functionality to occur during an upgrade. The `RunMigration` function will loop through the `VersionMap` argument, and run the migration scripts for any versions that are less than the new binary's app module versions. Once the migrations are finished, a new `VersionMap` will be returned to persist the upgraded module versions to state. +Migrations are run inside of an `UpgradeHandler` using `app.mm.RunMigrations(ctx, cfg, vm)`. The `UpgradeHandler` functions describe the functionality to occur during an upgrade. The `RunMigration` function loops through the `VersionMap` argument and runs the migration scripts for all versions that are less than the versions of the new binary app module. After the migrations are finished, a new `VersionMap` is returned to persist the upgraded module versions to state. ```golang cfg := module.NewConfigurator(...) @@ -70,13 +73,13 @@ app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgrad }) ``` -To learn more about configuring migration scripts for your modules, refer to this [guide](../building-modules/upgrade.md). +To learn more about configuring migration scripts for your modules, see the [Migration Upgrade Guide](../building-modules/upgrade.md). -## Adding New Modules In Upgrades +## Adding New Modules During Upgrades -Entirely new modules can be introduced to the application during an upgrade. The SDK recognizes new modules during upgrades because their consensus version in the `fromVM` `VersionMap` is 0. In this case, `RunMigrations` will call the corresponding module's `InitGenesis` function to setup its initial state. This can be skipped if the module does not require any inital state. Otherwise, it is important to implement `InitGenesis` for new modules to successfully upgrade your application without error. +You can introduce entirely new modules to the application during an upgrade. New modules are recognized because their consensus version in the `fromVM` `VersionMap` is 0. In this case, `RunMigrations` calls the `InitGenesis` function from the corresponding module to set up its initial state. If the module requires an initial state, run `InitGenesis` to ensure new modules can successfully upgrade your application without error. -In the scenario where your application does not need any inital state via `InitGenesis`, you must take extra steps to ensure `InitGenesis` is skipped to avoid errors. To do so, you simply need to update the value of the module version in the `VersionMap` in the `UpgradeHandler`. +If the new module does not require an initial state, you can skip running `InitGenesis`. Instead, manually update the value of the module version in the `VersionMap` in `UpgradeHandler`. ```go // Foo is a new module being introduced From 3d9e00a1b4005bd2ca7ef5a9016173592b8e820e Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Wed, 7 Apr 2021 14:24:38 -0700 Subject: [PATCH 37/43] fix self-reference --- docs/core/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 39dc0cdcecb5..15acf2ec88c8 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -14,7 +14,7 @@ The Cosmos SDK uses two methods to perform upgrades. - Exporting the entire application state to a JSON file using the `export` CLI command, making changes, and then starting a new binary with the changed JSON file as the genesis file. See the [Chain Upgrade Guide](../migrations/chain-upgrade-guide-040.md#upgrade-procedure). -- Version v0.43 and later can perform upgrades in place to significantly decrease the upgrade time for chains with a larger state. Use the [In-Place Store Migrations](./upgrade.md) guide to set up your application to take advantage of in-place upgrades. +- Version v0.43 and later can perform upgrades in place to significantly decrease the upgrade time for chains with a larger state. Use the [Migration Upgrade Guide](../building-modules/upgrade.md) guide to set up your application modules to take advantage of in-place upgrades. ## Tracking Module Versions From 7883af3bee69ef1232c670271e4254ba2515c3f6 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Fri, 9 Apr 2021 14:55:08 -0700 Subject: [PATCH 38/43] clearer definitions for overwriting genesis functions --- docs/core/upgrade.md | 33 ++++++++++++++++++++++----------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 15acf2ec88c8..67622e8968d4 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -77,35 +77,46 @@ To learn more about configuring migration scripts for your modules, see the [Mig ## Adding New Modules During Upgrades -You can introduce entirely new modules to the application during an upgrade. New modules are recognized because their consensus version in the `fromVM` `VersionMap` is 0. In this case, `RunMigrations` calls the `InitGenesis` function from the corresponding module to set up its initial state. If the module requires an initial state, run `InitGenesis` to ensure new modules can successfully upgrade your application without error. +You can introduce entirely new modules to the application during an upgrade. New modules are recognized because their consensus version in the `fromVM` `VersionMap` is 0. In this case, `RunMigrations` calls the `InitGenesis` function from the corresponding module to set up its initial state. -If the new module does not require an initial state, you can skip running `InitGenesis`. Instead, manually update the value of the module version in the `VersionMap` in `UpgradeHandler`. +## Overwriting Genesis Functions + +The Cosmos SDK offers modules that the application developer can import in their app. These modules often have an `InitGenesis` function already defined. + +You can write your own `InitGenesis` function for an imported module. To do this, manually trigger your custom genesis function in the upgrade handler. + +::: warning +You MUST manually set the consensus version in the version map passed to the `UpgradeHandler` function. Without this, the SDK will run the Module's existing `InitGenesis` code even if you triggered your custom function in the `UpgradeHandler`. +::: ```go -// Foo is a new module being introduced -// in this upgrade plan import foo "github.com/my/module/foo" app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) { - // We make sure to set foo's version to the latest ConsensusVersion in the VersionMap. - // This will skip running InitGenesis on Foo + + // Register the consensus version in the version map + // to avoid the SDK from triggering the default + // InitGenesis function. vm["foo"] = foo.AppModule{}.ConsensusVersion() + // Run custom InitGenesis for foo + app.mm["foo"].InitGenesis(ctx, app.appCodec, myCustomGenesisState) + return app.mm.RunMigrations(ctx, cfg, vm) }) ``` -Using a similar method, you can also run `InitGenesis` on your new module with a custom genesis state: +If you do not have a custom genesis function and want to skip the module's default genesis function, you can simply register the module with the version map in the `UpgradeHandler` as shown in the example: ```go import foo "github.com/my/module/foo" app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) { + + // Set foo's version to the latest ConsensusVersion in the VersionMap. + // This will skip running InitGenesis on Foo vm["foo"] = foo.AppModule{}.ConsensusVersion() - // Run custom InitGenesis for foo - app.mm["foo"].InitGenesis(ctx, app.appCodec, myCustomGenesisState) - return app.mm.RunMigrations(ctx, cfg, vm) }) -``` +``` \ No newline at end of file From a77aa4c0caf75b897f6ec7f000cb8bba90404655 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Fri, 9 Apr 2021 15:36:54 -0700 Subject: [PATCH 39/43] add sync section --- docs/core/upgrade.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 67622e8968d4..b320fc8d624a 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -119,4 +119,12 @@ app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgrad return app.mm.RunMigrations(ctx, cfg, vm) }) -``` \ No newline at end of file +``` + +## Syncing a Full Node to an Upgraded Blockchain + +You can sync a full node to an existing blockchain that has been upgraded using Cosmovisor + +In order to successfully sync, you must start with the initial binary that the blockchain started with at genesis. Cosmovisor will handle downloading downloading and switching to the binaries associated with each sequential upgrade. + +To learn more about Cosmovisor, see the [Cosmovisor Quick Start](../run-node/cosmovisor.md). \ No newline at end of file From 52da394c90e6921fbd3f9a3695c15ac9fc9ca8dc Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Fri, 9 Apr 2021 15:49:45 -0700 Subject: [PATCH 40/43] forgot to save this --- docs/core/upgrade.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index b320fc8d624a..f8db22ce0851 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -123,8 +123,8 @@ app.UpgradeKeeper.SetUpgradeHandler("my-plan", func(ctx sdk.Context, plan upgrad ## Syncing a Full Node to an Upgraded Blockchain -You can sync a full node to an existing blockchain that has been upgraded using Cosmovisor +You can sync a full node to an existing blockchain which has been upgraded using Cosmovisor -In order to successfully sync, you must start with the initial binary that the blockchain started with at genesis. Cosmovisor will handle downloading downloading and switching to the binaries associated with each sequential upgrade. +In order to successfully sync, you must start with the initial binary that the blockchain started with at genesis. Cosmovisor will handle downloading and switching to the binaries associated with each sequential upgrade. To learn more about Cosmovisor, see the [Cosmovisor Quick Start](../run-node/cosmovisor.md). \ No newline at end of file From 6ea0101e4be94fc385e046d2c5f16c3035de155a Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Tue, 13 Apr 2021 12:38:54 -0700 Subject: [PATCH 41/43] update description of initgenesis modules --- docs/core/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index f8db22ce0851..77dc2d944794 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -77,7 +77,7 @@ To learn more about configuring migration scripts for your modules, see the [Mig ## Adding New Modules During Upgrades -You can introduce entirely new modules to the application during an upgrade. New modules are recognized because their consensus version in the `fromVM` `VersionMap` is 0. In this case, `RunMigrations` calls the `InitGenesis` function from the corresponding module to set up its initial state. +You can introduce entirely new modules to the application during an upgrade. New modules are recognized because they have not yet been registered in `x/upgrade`'s `VersionMap` store. In this case, `RunMigrations` calls the `InitGenesis` function from the corresponding module to set up its initial state. ## Overwriting Genesis Functions From 84b6a73251c063adf7d389d7a43ec1f563e6a5d7 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+tytech3@users.noreply.github.com> Date: Tue, 13 Apr 2021 12:48:05 -0700 Subject: [PATCH 42/43] specify upgrade method --- docs/core/upgrade.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 77dc2d944794..13ef1609b988 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -15,6 +15,9 @@ The Cosmos SDK uses two methods to perform upgrades. - Exporting the entire application state to a JSON file using the `export` CLI command, making changes, and then starting a new binary with the changed JSON file as the genesis file. See the [Chain Upgrade Guide](../migrations/chain-upgrade-guide-040.md#upgrade-procedure). - Version v0.43 and later can perform upgrades in place to significantly decrease the upgrade time for chains with a larger state. Use the [Migration Upgrade Guide](../building-modules/upgrade.md) guide to set up your application modules to take advantage of in-place upgrades. + + +This document outlines the second method described above. ## Tracking Module Versions From 85d86e1b1aa074c5aeb625a1fa2e9012d2249134 Mon Sep 17 00:00:00 2001 From: technicallyty <48813565+technicallyty@users.noreply.github.com> Date: Tue, 13 Apr 2021 13:36:28 -0700 Subject: [PATCH 43/43] Update docs/core/upgrade.md Co-authored-by: Barrie Byron --- docs/core/upgrade.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/core/upgrade.md b/docs/core/upgrade.md index 13ef1609b988..3b5331ee3a68 100644 --- a/docs/core/upgrade.md +++ b/docs/core/upgrade.md @@ -17,7 +17,7 @@ The Cosmos SDK uses two methods to perform upgrades. - Version v0.43 and later can perform upgrades in place to significantly decrease the upgrade time for chains with a larger state. Use the [Migration Upgrade Guide](../building-modules/upgrade.md) guide to set up your application modules to take advantage of in-place upgrades. -This document outlines the second method described above. +This document provides steps to use the In-Place Store Migrations upgrade method. ## Tracking Module Versions @@ -130,4 +130,4 @@ You can sync a full node to an existing blockchain which has been upgraded using In order to successfully sync, you must start with the initial binary that the blockchain started with at genesis. Cosmovisor will handle downloading and switching to the binaries associated with each sequential upgrade. -To learn more about Cosmovisor, see the [Cosmovisor Quick Start](../run-node/cosmovisor.md). \ No newline at end of file +To learn more about Cosmovisor, see the [Cosmovisor Quick Start](../run-node/cosmovisor.md).