feat: Add `PrepareCheckState` and `Precommit` callbacks #14860

prettymuchbryce · 2023-01-31T22:30:57Z

Description

This PR introduces optional PrepareCheckState and Precommit callbacks. It allows modules to optionally include these methods.

These callbacks are similar to the existing callbacks for BeginBlock and EndBlock that modules may register, but slightly simpler they accept no arguments besides an sdk.Context, and they have no return values.

Precommit - Invoked during BaseApp.Commit() before the deliverState is written to the rootMultiStore. It is invoked with the deliverState. This could be useful for data/metrics pipelines that want to flush data to some external source, immediately before the deliverState for the block is written, and before any TransientStore within the deliverState has been reset.
PrepareCheckState - Invoked during BaseApp.Commit() after the deliverState has been written to the rootMultiStore, the deliverState has been emptied/reset, and the checkState has been set with the new block header. Invoked with the checkState of the next block. dYdX has introduced this functionality into our own fork as a way to prepare the checkState with some data immediately after the block has been committed.

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

confirmed the correct type prefix in the PR title
confirmed ! in the type prefix if API or client breaking change
confirmed all author checklist items have been addressed
reviewed state machine logic
reviewed API design and naming
reviewed documentation is accurate
reviewed tests and test coverage
manually tested (if applicable)

github-actions · 2023-01-31T22:31:53Z

baseapp/abci.go

 	// empty/reset the deliver state
 	app.deliverState = nil

+	if app.commiter != nil {
+		app.commiter(app.checkState.ctx)
+	}
+


Change potentially affects state.

Call sequence:

(*github.com/cosmos/cosmos-sdk/baseapp.BaseApp).Commit (baseapp/abci.go:426)

facundomedica · 2023-02-06T13:20:53Z

Hello! I have some questions:

What kind of data are you putting into checkState? Can this be done in PrepareProposal?
What modules would do during Commit? They are going to be setting the data in checkState?
Can you point out some use cases for this?

prettymuchbryce · 2023-02-07T19:52:48Z

Hey @facundomedica. Great questions!

What kind of data are you putting into checkState?

Currently we are putting data related to our orderbook in the checkState. This is because in our design, validators match orders intra-block rather than during EndBlock/Commit. After the block has been committed, each validator needs to "unwind" their own orders in order to account for any matches included in the block by the proposer. This results in changes to the checkState.

Can this be done in PrepareProposal?

This can't be done in PrepareProposal as PrepareProposal is only called by the proposer, and not all validators.

What modules would do during Commit? They are going to be setting the data in checkState?

Yes. Currently modules can set data in the checkState. This data will then be present for transactions during CheckTx after the block has been committed.

Can you point out some use cases for this?

I can't think of any other use-cases other than our own. Our own use-case may also be unique given that we perform a high number of state writes during CheckTx for our orderbook. AFAIK the default modules don't do this, and the only state writes that occur during CheckTx are ones done in the AnteHandlers, like updating the sequence number, and deducting fees.

It's very possible that this may be something unique to our own use-case, and if that is true then we can continue to maintain a lightweight fork that includes this change for the time being.

yihuang · 2023-02-07T22:46:14Z

Is something like PrepareCheckState more explicit here, with Commit I would think modules can commit other db together with the main db.
By put stuff in checkState do you mean transient store?

prettymuchbryce · 2023-02-07T22:54:24Z

Is something like PrepareCheckState more explicit here, with Commit I would think modules can commit other db together with the main db.

Yes I agree with your thinking here. It's not clear based on the current implementation that the state within the context is the checkState rather than the uncommitted deliverState. Renaming the callback from Commit to PrepareCheckState would help us clarify the intention of the callback, and would self-document the state that will be present in the context.

By put stuff in checkState do you mean transient store?

No this is not a TransientStore. Within our Commit callback, we modify the CacheMultiStore that has been branched for the checkState. This is the same CacheMultiStore that is present in the context during CheckTx.

alexanderbez · 2023-02-09T15:16:13Z

So what I gather from this as the gist is that you want a way for all validators to modify the CheckTx state prior to the next block being called, as opposed to it being set to the resulting state after Commit as it is today?

If so, I don't immediately see anything wrong with this and I'm not opposed. I think clear and clean APIs with sufficient documentation would be enough for me to give a green light.

I don't want to encourage teams to maintain their own forks when it's not necessary :)

itsdevbear · 2023-02-17T03:32:21Z

@alexanderbez +1 here, this would be helpful for our evm

alexanderbez · 2023-02-17T14:52:51Z

Yeah sounds good. Feel free to push it over the finish line 👍

itsdevbear · 2023-02-18T15:31:34Z

@alexanderbez would you be open to moving the hook to BEFORE the multistore is committed.

Our use case actually needs access to the transient store from the block.

Pros and cons to both I guess.

Unless we add PreCommit() PostCommit()?

Seems like @prettymuchbryce needs it after the commit though. So maybe this needs a broader discussion

alexanderbez · 2023-02-21T20:20:46Z

@itsdevbear can you further elaborate on your use-case? The original proposal was to modify CheckTx after commitment. How does yours differ?

prettymuchbryce · 2023-02-23T15:23:28Z

@itsdevbear We recently had a similar requirement in our codebase as well, so we decided to add a second callback for PreCommit which was implemented by my colleague @dydxwill. Can you take a look here and let me know if it matches your thinking?

I wonder if there would be appetite to include both of these upstream. We can help clean things up and move both callbacks into this PR.

I think clear and clean APIs with sufficient documentation would be enough for me to give a green light.

@alexanderbez I'm curious what other documentation we should be thinking about other than just the Godoc comments. For example this section of the documentation, or other places I may be missing.

itsdevbear · 2023-02-26T17:08:12Z

Our use case is to dump data from the previous block to an off chain store for indexing. Right now we are doing it in end block, which works but it's technically possible for the previous block to error out.

Doing it on commit feels much safer. But we can't do it before the ctx is reset. Since we were thinking storing this data in a transient store during the block and dumping to an off chain disk at the end.

yihuang · 2023-02-27T11:46:16Z

Our use case is to dump data from the previous block to an off chain store for indexing. Right now we are doing it in end block, which works but it's technically possible for the previous block to error out.

even if it error out, it'll be replayed eventually, so probably not a big deal?

alexanderbez · 2023-02-27T14:40:31Z

@itsdevbear if it's for indexing, why don't you just use the state streamer. That's what it was designed for.

@prettymuchbryce I would say any baseapp or ABCI docs should have mention of this 👍

itsdevbear · 2023-02-27T14:47:10Z

yeah @alexanderbez that was the other thing we were thinking. Will investigate that path as well.

Ty ser

either way I think pre and post still useful here.

tac0turtle · 2023-03-02T10:20:40Z

@itsdevbear @prettymuchbryce would you be willing to come on the community call next week to discuss this feature sync so we can move this forward

itsdevbear · 2023-03-02T14:20:01Z

@tacoturtle definitely down, we can also chat about: #14660 it's a relatively a critical for us one for us.

Can you send me the info / send a cal recurring cal invite?

prettymuchbryce · 2023-03-02T14:31:14Z

I am down. Can you send me an invite?

alexanderbez · 2023-03-20T19:20:21Z

@prettymuchbryce are you open to adding the post-hook in this PR as you did with the pre-hook, so we can achieve the needs of both teams? Happy to review and approve.

types/module/module.go

github-actions · 2023-03-22T21:19:34Z

baseapp/abci.go

 	// Commit. Use the header from this latest block.
 	app.setState(runTxModeCheck, header)

+	if app.precommiter != nil {
+		app.precommiter(app.deliverState.ctx)
+	}
+
 	// empty/reset the deliver state
 	app.deliverState = nil



Change potentially affects state.

Call sequence:

(*github.com/cosmos/cosmos-sdk/baseapp.BaseApp).Commit (baseapp/abci.go:452)

telemetry/wrapper.go

prettymuchbryce · 2023-03-29T21:38:51Z

baseapp/abci.go

@@ -456,14 +456,18 @@ func (app *BaseApp) Commit() abci.ResponseCommit {
 	header := app.deliverState.ctx.BlockHeader()
 	retainHeight := app.GetBlockRetentionHeight(header.Height)

+	if app.precommiter != nil {


@itsdevbear Was curious if this is where you envisioned this being invoked. At this point, your TransientStore will not have been cleared yet.

This makes sense to me. If we are passing in the ctx without nuking it, I don't think it really matters where it happens.

prettymuchbryce · 2023-03-30T17:24:46Z

As far as I can tell the remaining CI failures in this PR are not related to my changes. Let me know if I am missing something.

julienrbrt

lgtm!

As far as I can tell the remaining CI failures in this PR are not related to my changes. Let me know if I am missing something.

Yes, these CI failures are unrelated.

runtime/app.go

baseapp/options.go

itsdevbear · 2023-04-18T13:14:47Z

@prettymuchbryce we've been using this branch, has been good so far.

facundomedica

LGTM. This might change a bit with the implementation of abci 1.0, but I don't foresee big changes given that Commit() will remain pretty much the same iirc

tac0turtle

utACK. we can do a follow up pr to add docs

itsdevbear · 2023-04-23T16:05:34Z

⚡️🎉

julienrbrt · 2023-04-24T09:44:28Z

Tagged cosmossdk.io/api v0.4.1 with this.

itsdevbear · 2023-04-24T10:54:31Z

Pulled sdk main into polaris, it works great.

Thanks @prettymuchbryce for pushing this across.

github-actions bot reviewed Jan 31, 2023

View reviewed changes

julienrbrt assigned JeancarloBarrios and facundomedica Feb 1, 2023

itsdevbear mentioned this pull request Feb 17, 2023

feat(sdk): Committer berachain/polaris#230

Closed

Add Commit callback

cda4b81

prettymuchbryce force-pushed the prettymuchbryce/add-commiter branch from 5066667 to cda4b81 Compare March 22, 2023 19:10

julienrbrt reviewed Mar 22, 2023

View reviewed changes

types/module/module.go Outdated Show resolved Hide resolved

prettymuchbryce and others added 2 commits March 22, 2023 17:05

Add tests for Commiter change (#5)

862b4d8

add precommit callback (#7)

1d72615

github-actions bot reviewed Mar 22, 2023

View reviewed changes

lint

74ef20a

prettymuchbryce commented Mar 22, 2023

View reviewed changes

telemetry/wrapper.go Outdated Show resolved Hide resolved

Update mod file with api

eb40e33

prettymuchbryce commented Mar 29, 2023

View reviewed changes

prettymuchbryce added 5 commits March 30, 2023 11:45

Update NewManagerFromMap

ea68fee

Merge branch 'main' into prettymuchbryce/add-commiter

caa2cde

Add api to tests package go.mod

a0616d0

Merge branch 'main' into prettymuchbryce/add-commiter

22095a4

Add api to tests package go.mod

5393018

julienrbrt approved these changes Apr 4, 2023

View reviewed changes

julienrbrt self-assigned this Apr 4, 2023

calbera reviewed Apr 4, 2023

View reviewed changes

runtime/app.go Show resolved Hide resolved

calbera reviewed Apr 4, 2023

View reviewed changes

runtime/app.go Show resolved Hide resolved

calbera reviewed Apr 4, 2023

View reviewed changes

baseapp/options.go Outdated Show resolved Hide resolved

prettymuchbryce added 3 commits April 5, 2023 08:26

Set callbacks during Load

f20ee96

Fix comment

cf4674e

Merge branch 'main' into prettymuchbryce/add-commiter

d40920f

itsdevbear mentioned this pull request Apr 5, 2023

feat(cosmos): Use Precommiter hook to store end of block state berachain/polaris#540

Merged

Merge branch 'main' into prettymuchbryce/add-commiter

bd73f81

facundomedica approved these changes Apr 22, 2023

View reviewed changes

tac0turtle approved these changes Apr 23, 2023

View reviewed changes

tac0turtle added this pull request to the merge queue Apr 23, 2023

tac0turtle mentioned this pull request Apr 23, 2023

[Follow-up]: Prepare states #15914

Closed

Merged via the queue into cosmos:main with commit 93d64cc Apr 23, 2023

tac0turtle mentioned this pull request Apr 25, 2023

chore: move precommit & preparecheckstate to core api #15923

Merged

19 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add `PrepareCheckState` and `Precommit` callbacks #14860

feat: Add `PrepareCheckState` and `Precommit` callbacks #14860

prettymuchbryce commented Jan 31, 2023 •

edited

Loading

github-actions bot Jan 31, 2023

facundomedica commented Feb 6, 2023

prettymuchbryce commented Feb 7, 2023

yihuang commented Feb 7, 2023

prettymuchbryce commented Feb 7, 2023

alexanderbez commented Feb 9, 2023

itsdevbear commented Feb 17, 2023

alexanderbez commented Feb 17, 2023

itsdevbear commented Feb 18, 2023 •

edited

Loading

alexanderbez commented Feb 21, 2023

prettymuchbryce commented Feb 23, 2023

itsdevbear commented Feb 26, 2023

yihuang commented Feb 27, 2023

alexanderbez commented Feb 27, 2023

itsdevbear commented Feb 27, 2023

tac0turtle commented Mar 2, 2023

itsdevbear commented Mar 2, 2023

prettymuchbryce commented Mar 2, 2023

alexanderbez commented Mar 20, 2023

github-actions bot Mar 22, 2023

prettymuchbryce Mar 29, 2023

itsdevbear Mar 30, 2023

prettymuchbryce commented Mar 30, 2023

julienrbrt left a comment •

edited

Loading

itsdevbear commented Apr 18, 2023

facundomedica left a comment

tac0turtle left a comment

itsdevbear commented Apr 23, 2023

julienrbrt commented Apr 24, 2023 •

edited

Loading

itsdevbear commented Apr 24, 2023

feat: Add PrepareCheckState and Precommit callbacks #14860

feat: Add PrepareCheckState and Precommit callbacks #14860

Conversation

prettymuchbryce commented Jan 31, 2023 • edited Loading

Description

Author Checklist

Reviewers Checklist

github-actions bot Jan 31, 2023

Choose a reason for hiding this comment

facundomedica commented Feb 6, 2023

prettymuchbryce commented Feb 7, 2023

yihuang commented Feb 7, 2023

prettymuchbryce commented Feb 7, 2023

alexanderbez commented Feb 9, 2023

itsdevbear commented Feb 17, 2023

alexanderbez commented Feb 17, 2023

itsdevbear commented Feb 18, 2023 • edited Loading

alexanderbez commented Feb 21, 2023

prettymuchbryce commented Feb 23, 2023

itsdevbear commented Feb 26, 2023

yihuang commented Feb 27, 2023

alexanderbez commented Feb 27, 2023

itsdevbear commented Feb 27, 2023

tac0turtle commented Mar 2, 2023

itsdevbear commented Mar 2, 2023

prettymuchbryce commented Mar 2, 2023

alexanderbez commented Mar 20, 2023

github-actions bot Mar 22, 2023

Choose a reason for hiding this comment

prettymuchbryce Mar 29, 2023

Choose a reason for hiding this comment

itsdevbear Mar 30, 2023

Choose a reason for hiding this comment

prettymuchbryce commented Mar 30, 2023

julienrbrt left a comment • edited Loading

Choose a reason for hiding this comment

itsdevbear commented Apr 18, 2023

facundomedica left a comment

Choose a reason for hiding this comment

tac0turtle left a comment

Choose a reason for hiding this comment

itsdevbear commented Apr 23, 2023

julienrbrt commented Apr 24, 2023 • edited Loading

itsdevbear commented Apr 24, 2023

feat: Add `PrepareCheckState` and `Precommit` callbacks #14860

feat: Add `PrepareCheckState` and `Precommit` callbacks #14860

prettymuchbryce commented Jan 31, 2023 •

edited

Loading

itsdevbear commented Feb 18, 2023 •

edited

Loading

julienrbrt left a comment •

edited

Loading

julienrbrt commented Apr 24, 2023 •

edited

Loading