fix: state listener observe writes at wrong time (backport #13516) #14137

mergify · 2022-12-02T14:44:13Z

This is an automatic backport of pull request #13516 done by Mergify.
Cherry-pick of 1f91ee2 has failed:

On branch mergify/bp/release/v0.47.x/pr-13516
Your branch is up to date with 'origin/release/v0.47.x'.

You are currently cherry-picking commit 1f91ee2ee.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   api/cosmos/base/store/v1beta1/listening.pulsar.go
	modified:   baseapp/abci.go
	modified:   docs/architecture/adr-038-state-listening.md
	modified:   proto/cosmos/base/store/v1beta1/listening.proto
	modified:   server/config/config.go
	modified:   server/config/toml.go
	modified:   simapp/app.go
	modified:   simapp/app_v2.go
	modified:   store/cachekv/store.go
	modified:   store/cachemulti/store.go
	modified:   store/dbadapter/store.go
	modified:   store/dbadapter/store_test.go
	modified:   store/gaskv/store.go
	modified:   store/gaskv/store_test.go
	modified:   store/iavl/store.go
	modified:   store/iavl/store_test.go
	modified:   store/listenkv/store.go
	modified:   store/listenkv/store_test.go
	modified:   store/mem/mem_test.go
	modified:   store/mem/store.go
	modified:   store/prefix/store.go
	modified:   store/prefix/store_test.go
	modified:   store/rootmulti/store.go
	modified:   store/rootmulti/store_test.go
	modified:   store/streaming/constructor_test.go
	modified:   store/streaming/file/README.md
	modified:   store/streaming/file/service_test.go
	modified:   store/tracekv/store.go
	modified:   store/tracekv/store_test.go
	modified:   store/types/listening.go
	modified:   store/types/listening.pb.go
	modified:   store/types/store.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   CHANGELOG.md
	both modified:   baseapp/streaming.go
	both modified:   store/streaming/constructor.go
	both modified:   store/streaming/file/service.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

@Mergifyio refresh will re-evaluate the rules
@Mergifyio rebase will rebase this PR on its base branch
@Mergifyio update will merge the base branch into this PR
@Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

look at your merge queues
generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.com

* fix: state listener observe writes at wrong time Closes: #13457 Currently state listener is notified when the cache store write, which happens in commit event only, which breaks the current design. The solution (as discussed in the issue) is to listen state writes on rootmulti store only. It also changes the file streamer to output single data file for the writes in the whole block, since we can't distinguish writes from different stage of abci events. It adds new config items for file streamer: - streamers.file.output-metadata - streamers.file.stop-node-on-error - streamers.file.fsync * synchronous abci call, and format doc * fix comment * update file streamer readme and fix typos * typo * fix: state listener observe writes at wrong time Closes: #13457 Currently state listener is notified when the cache store write, which happens in commit event only, which breaks the current design. The solution (as discussed in the issue) is to listen state writes on rootmulti store only. It also changes the file streamer to output single data file for the writes in the whole block, since we can't distinguish writes from different stage of abci events. It adds new config items for file streamer: - streamers.file.output-metadata - streamers.file.stop-node-on-error - streamers.file.fsync synchronous abci call, and format doc fix comment update file streamer readme and fix typos typo * improve UX of file streamer, make it immediately usable after enabled - set default value to write_dir. - make write_dir based on home directory by default. - auto-create the directory if not exists. * get homePage from opts Co-authored-by: Marko <marbar3778@yahoo.com> (cherry picked from commit 1f91ee2) # Conflicts: # CHANGELOG.md # baseapp/streaming.go # store/streaming/constructor.go # store/streaming/file/service.go

baseapp/abci.go

@@ -202,7 +202,7 @@
 	// call the hooks with the BeginBlock messages
 	for _, streamingListener := range app.abciListeners {
 		if err := streamingListener.ListenBeginBlock(app.deliverState.ctx, req, res); err != nil {
-			app.logger.Error("BeginBlock listening hook failed", "height", req.Header.Height, "err", err)
+			panic(fmt.Errorf("BeginBlock listening hook failed, height: %d, err: %w", req.Header.Height, err))


baseapp/abci.go

@@ -227,7 +227,7 @@
 	// call the streaming service hooks with the EndBlock messages
 	for _, streamingListener := range app.abciListeners {
 		if err := streamingListener.ListenEndBlock(app.deliverState.ctx, req, res); err != nil {
-			app.logger.Error("EndBlock listening hook failed", "height", req.Height, "err", err)
+			panic(fmt.Errorf("EndBlock listening hook failed, height: %d, err: %w", req.Height, err))
