feat: introduce PreBlock #17421

mmsqe · 2023-08-17T04:58:23Z

to runs before begin blocker, and be more generic than hardcode with migration
allowed to modify consensus parameters and the changes are visible to the following state machine logics
add missing refresh for finalizeBlockState.ctx, since only ctx within beginBlock scope was changed, it did not update the ConsensusParams of finalizeBlockState.ctx which is used in endBlock and now it does
add ConsensusParamsChanged check to avoid refreshing ConsensusParams of ctx every time

Description

Context

The Original Problem

When upgrading to sdk 0.47, the storage format for consensus parameters changed, but in the migration block, ctx.ConsensusParams() is always nil, because it fails to load the old format using new code, it's supposed to be migrated by the x/upgrade module first, but unfortunately, the migration happens in BeginBlocker handler, which runs after the ctx is initialized.
When we try to solve this, we find the x/upgrade module can't modify the context to make the consensus parameters visible for the other modules, the context is passed by value, and sdk team want to keep it that way, that's good for isolations between modules.
The first alternative solution introduced a MigrateModuleManager, which only includes the x/upgrade module right now, and baseapp will run their BeginBlockers before the other modules, and reload context's consensus parameters in between, but I think it's too hacky, and suggested this new lifecycle method.

`PreBlocker`

There are two semantics around the new lifecycle method:

It runs before the BeginBlocker of all modules
It can modify consensus parameters in storage, and signal the caller through the return value.

When it returns ConsensusParamsChanged=true, the caller must refresh the consensus parameter in the deliver context:

app.deliverState.ctx = app.deliverState.ctx.WithConsensusParams(app.GetConsensusParams())

The new ctx must be passed to all the other lifecycle methods.

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

confirmed the correct type prefix in the PR title
confirmed ! in the type prefix if API or client breaking change
confirmed all author checklist items have been addressed
reviewed state machine logic
reviewed API design and naming
reviewed documentation is accurate
reviewed tests and test coverage
manually tested (if applicable)

types/module/module.go

julienrbrt

Not super a fan personally of this new notion of pre-begin blocker.

With PreCommitter, PrepareCheckStaters, I feel like it makes the module API more complicated.

What was the issue with the previous implementation? And why does it need to be generalized? What kind of module other than upgrade could take advantage of this?

julienrbrt · 2023-08-17T06:52:08Z

CHANGELOG.md

@@ -55,6 +55,8 @@ Ref: https://keepachangelog.com/en/1.0.0/
 * (version) [#17096](https://github.com/cosmos/cosmos-sdk/pull/17096) Improve `getSDKVersion()` to handle module replacements
 * (x/staking) [#17164](https://github.com/cosmos/cosmos-sdk/pull/17164) Add `BondedTokensAndPubKeyByConsAddr` to the keeper to enable vote extension verification.
 * (x/genutil) [#17296](https://github.com/cosmos/cosmos-sdk/pull/17296) Add `MigrateHandler` to allow reuse migrate genesis related function.
+* (types) [#17421](https://github.com/cosmos/cosmos-sdk/pull/17421) Replace `RunMigrationBeginBlock` with `PreBeginBlock`, which runs before begin blocker other modules, and allows to modify consensus parameters, and the changes are visible to the following state machine logics.


It was never released so no changelog needed, best to edit the other changelog and add this PR number next to it.

julienrbrt · 2023-08-17T07:11:52Z

types/module/module.go

@@ -222,6 +222,11 @@ type HasConsensusVersion interface {
 	ConsensusVersion() uint64
 }

+type PreBeginBlockAppModule interface {
+	AppModule


These should be on core/appmodule and named Has<...>. We shouldn't introduce module related interface in types/module unless we need an sdk dependency.

yihuang · 2023-08-17T07:30:52Z

Not super a fan personally of this new notion of pre-begin blocker.

With PreCommitter, PrepareCheckStaters, I feel like it makes the module API more complicated.

What was the issue with the previous implementation? And why does it need to be generalized? What kind of module other than upgrade could take advantage of this?

the main issue with the previous implementation is it treat some begin blockers different from the other begin blockers, but they are all named as BeginBlock, that could be confusing.

baseapp/baseapp.go

* to runs before begin blocker, and be more generic than hardcode with migration * allowed to modify consensus parameters and the changes are visible to the following state machine logics

core/appmodule/module.go

x/upgrade/abci.go

simapp/app.go

runtime/app.go

alexanderbez · 2023-08-17T17:21:21Z

Can we get some more context in the PR description on the what and more importantly the why? :)

mmsqe · 2023-08-18T01:16:21Z

Can we get some more context in the PR description on the what and more importantly the why? :)

Just add two issues I see with previous change, and update title since this PR does bring behaviour change.

yihuang · 2023-08-18T01:51:23Z

Can we get some more context in the PR description on the what and more importantly the why? :)

Just add two issues I see with previous change, and update title since this PR does bring behaviour change.

I just added a section about the original issue and the alternative solutions.

tac0turtle · 2023-09-13T09:24:11Z

@mmsqe could you resolve conflicts? lets see about getting this merged

tac0turtle

LGTM, we can do it after the fact, a short adr would be good on this feature so we have it documented

x/upgrade/module.go

baseapp/baseapp.go

docs/architecture/adr-068-preblock.md

github-actions · 2023-09-13T11:37:36Z

baseapp/abci.go

 		}
 	}

+	if err := app.preBlock(); err != nil {
+		return nil, err
+	}
+


Change potentially affects state.

Call sequence:

(*github.com/cosmos/cosmos-sdk/baseapp.BaseApp).FinalizeBlock (baseapp/abci.go:669)

github-actions · 2023-09-13T11:37:37Z

baseapp/baseapp.go

+			ctx = ctx.WithConsensusParams(app.GetConsensusParams(ctx))
+			// GasMeter must be set after we get a context with updated consensus params.
+			gasMeter := app.getBlockGasMeter(ctx)
+			ctx = ctx.WithBlockGasMeter(gasMeter)
+			app.finalizeBlockState.ctx = ctx
+		}
+	}


Change potentially affects state.

Call sequence:

(*github.com/cosmos/cosmos-sdk/baseapp.BaseApp).beginBlock (baseapp/baseapp.go:685) (*github.com/cosmos/cosmos-sdk/baseapp.BaseApp).FinalizeBlock (baseapp/baseapp.go:669)

docs/architecture/adr-068-preblock.md

(cherry picked from commit 4eb0185) # Conflicts: # CHANGELOG.md # UPGRADING.md # docs/docs/build/building-apps/03-app-upgrade.md # docs/docs/build/building-modules/01-module-manager.md # docs/docs/develop/advanced/00-baseapp.md # runtime/builder.go # simapp/go.mod # simapp/go.sum # simapp/gomod2nix.toml # tests/go.mod # tests/go.sum # tests/starship/tests/go.mod # tests/starship/tests/go.sum # testutil/mock/types_mock_appmodule.go # types/module/module.go # types/module/module_test.go # x/feegrant/go.mod # x/upgrade/abci.go # x/upgrade/abci_test.go # x/upgrade/go.mod # x/upgrade/go.sum

julienrbrt · 2023-09-13T12:37:08Z

@mmsqe, are you up to submit this PR https://github.com/mmsqe/cosmos-sdk/pull/2/files now to main?

Co-authored-by: mmsqe <mavis@crypto.com> Co-authored-by: Julien Robert <julien@rbrt.fr>

github-actions bot added the C:x/upgrade label Aug 17, 2023

mmsqe force-pushed the pre-begin branch from 9924e4f to 5fc4797 Compare August 17, 2023 05:11

github-advanced-security bot found potential problems Aug 17, 2023

View reviewed changes

types/module/module.go Dismissed Show dismissed Hide dismissed

julienrbrt reviewed Aug 17, 2023

View reviewed changes

tac0turtle reviewed Aug 17, 2023

View reviewed changes

baseapp/baseapp.go Outdated Show resolved Hide resolved

replace RunMigrationBeginBlock with PreBeginBlock

9944fe1

* to runs before begin blocker, and be more generic than hardcode with migration * allowed to modify consensus parameters and the changes are visible to the following state machine logics

mmsqe force-pushed the pre-begin branch 3 times, most recently from 25002c4 to 8c0cccd Compare August 17, 2023 12:05

fix test

97437d2

mmsqe force-pushed the pre-begin branch from 8c0cccd to 97437d2 Compare August 17, 2023 12:24

julienrbrt reviewed Aug 17, 2023

View reviewed changes

core/appmodule/module.go Outdated Show resolved Hide resolved

mmsqe added 2 commits August 17, 2023 20:46

update doc

5df47ca

Merge branch 'main' into pre-begin

4cb3ead

mmsqe marked this pull request as ready for review August 17, 2023 12:48

mmsqe requested a review from a team as a code owner August 17, 2023 12:48

julienrbrt self-assigned this Aug 17, 2023

yihuang reviewed Aug 17, 2023

View reviewed changes

x/upgrade/abci.go Outdated Show resolved Hide resolved

kocubinski reviewed Aug 17, 2023

View reviewed changes

simapp/app.go Show resolved Hide resolved

fix doc

875aef9

mmsqe changed the title ~~refactor: replace RunMigrationBeginBlock with PreBeginBlock~~ refactor: replace RunMigrationBeginBlock with PreBlock Aug 17, 2023

add test

fa4d36c

mmsqe force-pushed the pre-begin branch from a6ed2ad to fa4d36c Compare August 17, 2023 13:47

mmsqe commented Aug 17, 2023

View reviewed changes

runtime/app.go Outdated Show resolved Hide resolved

Apply suggestions from code review

8f901a2

mmsqe changed the title ~~refactor: replace RunMigrationBeginBlock with PreBlock~~ fix: replace RunMigrationBeginBlock with PreBlock Aug 18, 2023

tac0turtle assigned alexanderbez and likhita-809 Sep 11, 2023

Merge branch 'main' into pre-begin

7a7dbaa

github-actions bot added C:x/feegrant C:x/nft C:x/circuit labels Sep 13, 2023

tac0turtle approved these changes Sep 13, 2023

View reviewed changes

x/upgrade/module.go Outdated Show resolved Hide resolved

tac0turtle reviewed Sep 13, 2023

View reviewed changes

baseapp/baseapp.go Outdated Show resolved Hide resolved

mmsqe added 2 commits September 13, 2023 19:07

Apply suggestions from code review

766c82d

Merge branch 'main' into pre-begin

95a19e0

tac0turtle reviewed Sep 13, 2023

View reviewed changes

docs/architecture/adr-068-preblock.md Show resolved Hide resolved

tac0turtle enabled auto-merge September 13, 2023 11:18

Merge branch 'main' into pre-begin

e55d9e5

github-actions bot reviewed Sep 13, 2023

View reviewed changes

julienrbrt reviewed Sep 13, 2023

View reviewed changes

docs/architecture/adr-068-preblock.md Show resolved Hide resolved

Apply suggestions from code review

88b1801

auto-merge was automatically disabled September 13, 2023 11:50
Head branch was pushed to by a user without write access

julienrbrt enabled auto-merge September 13, 2023 11:51

julienrbrt added this pull request to the merge queue Sep 13, 2023

Merged via the queue into cosmos:main with commit 4eb0185 Sep 13, 2023
39 of 45 checks passed

mergify bot mentioned this pull request Sep 13, 2023

feat: introduce PreBlock (backport #17421) #17712

Merged

mmsqe mentioned this pull request Sep 13, 2023

refactor!: reimplement PreFinalizeBlockHook as PreBlocker #17713

Merged

20 tasks

julienrbrt added a commit that referenced this pull request Sep 13, 2023

feat: introduce PreBlock (backport #17421) (#17712)

4c083c6

Co-authored-by: mmsqe <mavis@crypto.com> Co-authored-by: Julien Robert <julien@rbrt.fr>

coderabbitai bot mentioned this pull request Apr 22, 2024

chore: sync main changelogs with v0.47.11, v0.50.6, modules and x/tx #20125

Merged

12 tasks

poorphd mentioned this pull request Jun 26, 2024

upgradetypes.ModuleName has been mistakenly added to SetOrderBeginBlockers code-423n4/2024-05-canto-findings#21

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: introduce PreBlock #17421

feat: introduce PreBlock #17421

mmsqe commented Aug 17, 2023 •

edited by yihuang

Loading

julienrbrt left a comment •

edited

Loading

julienrbrt Aug 17, 2023 •

edited

Loading

julienrbrt Aug 17, 2023 •

edited

Loading

yihuang commented Aug 17, 2023

alexanderbez commented Aug 17, 2023

mmsqe commented Aug 18, 2023

yihuang commented Aug 18, 2023

tac0turtle commented Sep 13, 2023

tac0turtle left a comment

github-actions bot Sep 13, 2023

github-actions bot Sep 13, 2023

julienrbrt commented Sep 13, 2023

feat: introduce PreBlock #17421

feat: introduce PreBlock #17421

Conversation

mmsqe commented Aug 17, 2023 • edited by yihuang Loading

Description

Context

The Original Problem

PreBlocker

Author Checklist

Reviewers Checklist

julienrbrt left a comment • edited Loading

Choose a reason for hiding this comment

julienrbrt Aug 17, 2023 • edited Loading

Choose a reason for hiding this comment

julienrbrt Aug 17, 2023 • edited Loading

Choose a reason for hiding this comment

yihuang commented Aug 17, 2023

alexanderbez commented Aug 17, 2023

mmsqe commented Aug 18, 2023

yihuang commented Aug 18, 2023

tac0turtle commented Sep 13, 2023

tac0turtle left a comment

Choose a reason for hiding this comment

github-actions bot Sep 13, 2023

Choose a reason for hiding this comment

github-actions bot Sep 13, 2023

Choose a reason for hiding this comment

julienrbrt commented Sep 13, 2023

mmsqe commented Aug 17, 2023 •

edited by yihuang

Loading

`PreBlocker`

julienrbrt left a comment •

edited

Loading

julienrbrt Aug 17, 2023 •

edited

Loading

julienrbrt Aug 17, 2023 •

edited

Loading