Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error on replay with wrong app hash calculation (block #914381) #197

Closed
cyborgshead opened this issue Jan 25, 2019 · 4 comments
Closed

Error on replay with wrong app hash calculation (block #914381) #197

cyborgshead opened this issue Jan 25, 2019 · 4 comments
Labels
Type: Bug Something isn't working Type: Specification

Comments

@cyborgshead
Copy link
Member

Current Behavior

Cyberd validator node stack on block 914381 with error on replay and wrong app hash

Expected Behavior

Continuing of sync and get online validation

Reproduction

Ubuntu 18.04.01 LTS/NVIDIA-SMI 415.27/Driver Version: 415.27/CUDA Version: 10.0
i7-7700K @ 4.20GHz/ 32GB RAM / MSI GTX 1070 8GB / 512 SSD

Log

panic: Panicked questionably: Failed to process committed block (914381:9F818FD2149C6CC6E52AF7A4932734D09732BDDEAB629D699B8888BDC0CD1C4E): Wrong Block.Header.AppHash.  Expected 64D2D5DF2A679423AACB803DFAF79D8BA3BB224919BD5FC034A491C9075E8283, got 292590BC3F2700FBF3C47178DD37638118A92E720AE03C5FA0D546D1C7B29388

goroutine 529363 [running]:
github.com/tendermint/tendermint/libs/common.PanicQ(0xe2bc20, 0xc00d5b8520)
	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/libs/common/errors.go:245 +0xb8
github.com/tendermint/tendermint/blockchain.(*BlockchainReactor).poolRoutine(0xc0000f6680)
	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/blockchain/reactor.go:341 +0x1033
created by github.com/tendermint/tendermint/blockchain.(*BlockchainReactor).OnStart
	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/blockchain/reactor.go:118 +0x89
ERROR: Error during handshake: Error on replay: Wrong Block.Header.AppHash.  Expected 64D2D5DF2A679423AACB803DFAF79D8BA3BB224919BD5FC034A491C9075E8283, got 292590BC3F2700FBF3C47178DD37638118A92E720AE03C5FA0D546D1C7B29388
ERROR: Error during handshake: Error on replay: Wrong Block.Header.AppHash.  Expected 64D2D5DF2A679423AACB803DFAF79D8BA3BB224919BD5FC034A491C9075E8283, got 292590BC3F2700FBF3C47178DD37638118A92E720AE03C5FA0D546D1C7B29388
ERROR: Error during handshake: Error on replay: Wrong Block.Header.AppHash.  Expected 64D2D5DF2A679423AACB803DFAF79D8BA3BB224919BD5FC034A491C9075E8283, got 292590BC3F2700FBF3C47178DD37638118A92E720AE03C5FA0D5

Additional Information
@cyborgshead cyborgshead added Type: Bug Something isn't working help wanted labels Jan 25, 2019
@hleb-albau
Copy link
Contributor

@litvintech yeah, reproduced. Currently, we don't know exactly bug, need more investigation.
But, already knowing that this happens on soft fork block. Seems, some hidden bug on index loading after reload.

@hleb-albau hleb-albau mentioned this issue Jan 27, 2019
Closed
10 tasks
@hleb-albau
Copy link
Contributor

@litvintech @xhipster @arturalbov Bad news, there is a bug, that causes all full nodes joining network right now to panic at 914381. Bug related, to our wrong assumption of stake update in cosmos-sdk. We miss user stake update for jailing validators and calculate the wrong rank using wrong user stake. There is no easy way to eliminate it for current testnet. So, I suggest focussing on fixing the bug and moving to #205.
There is no easy way to track bounded stake in cosmos right now, so maybe for 0.1.1 we should change the rank calculation to honor only NotBounded user stake, and investigate ways to archive bounded +unboded stake for next testnets.

@hleb-albau hleb-albau mentioned this issue Jan 27, 2019
Closed
@cyborgshead
Copy link
Member Author

On Mars I have this


I[28016-01-28|10:23:21.552] Starting ABCI with Tendermint                module=main 


I[28016-01-28|10:23:21.584] Loading mem state                            module=main 


I[28016-01-28|10:23:41.161] App loaded                                   module=main time=19.576980759s


I[28016-01-28|10:23:41.323] Rank calculated                              module=main time=154.813899ms links=38 cids=45


panic: version 914360 was already saved to different hash 48C6928913C3A60C0DDC36B803E622128ABBD807DF8DFACFB80B3BD742A4E002 (existing hash D7AE39ADE93FE78AC5BDE6E6956C5C026D8B79532A79F337B398419B5C8CEAAC)




goroutine 1 [running]:


github.com/cosmos/cosmos-sdk/store.(*iavlStore).Commit(0xc00018cc40, 0xfb2b5f, 0x5, 0x20, 0x20)


	/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.29.0/store/iavlstore.go:75 +0x1dd


github.com/cosmos/cosmos-sdk/store.commitStores(0xdf3b8, 0xc00020d620, 0x7020103, 0x0, 0xffffffffffffffff, 0xc007a5d928)


	/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.29.0/store/rootmultistore.go:488 +0x177


github.com/cosmos/cosmos-sdk/store.(*rootMultiStore).Commit(0xc0004bd490, 0xc0085acdb0, 0xc0040ba6c0, 0xd, 0x8)


	/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.29.0/store/rootmultistore.go:197 +0x47


github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc000b46800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)


	/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.29.0/baseapp/baseapp.go:814 +0xea


github.com/cybercongress/cyberd/app.(*CyberdApp).Commit(0xc000202500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)


	/sources/app/app.go:544 +0x50


github.com/tendermint/tendermint/abci/client.(*localClient).CommitSync(0xc0082d6180, 0x0, 0x0, 0x0)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/abci/client/local_client.go:215 +0xc0


github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0xc00093e8c0, 0xc003428fa0, 0x121c340, 0xc00093e8c0)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/proxy/app_conn.go:81 +0x33


github.com/tendermint/tendermint/state.ExecCommitBlock(0x121c340, 0xc00093e8c0, 0xc00082a3c0, 0x12176c0, 0xc003428fa0, 0xc0085f03f0, 0x1220b80, 0xc002f00fa0, 0x14, 0x400000, ...)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/state/execution.go:522 +0x27e


github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlocks(0xc003374840, 0x8, 0x0, 0xc007b644a0, 0x6, 0xc007b644ab, 0x5, 0xdf3ca, 0x45, 0xc0083f2600, ...)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/consensus/replay.go:409 +0x244


github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(0xc003374840, 0x8, 0x0, 0xc007b644a0, 0x6, 0xc007b644ab, 0x5, 0xdf3ca, 0x45, 0xc0083f2600, ...)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/consensus/replay.go:347 +0x880


github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc003374840, 0x1221a00, 0xc008398230, 0x2, 0x12176c0)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/consensus/replay.go:253 +0x516


github.com/tendermint/tendermint/node.NewNode(0xc0001599e0, 0x1217880, 0xc002fb14a0, 0xc0026bdac0, 0x120bbc0, 0xc004b1ee40, 0xc0026bdce0, 0x11262d0, 0xc0026bdcf0, 0x12176c0, ...)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/node/node.go:204 +0x6cf


github.com/cosmos/cosmos-sdk/server.startInProcess(0xc000199ac0, 0x1126ad0, 0x1d, 0x0, 0x0)


	/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.29.0/server/start.go:118 +0x3f1


github.com/cosmos/cosmos-sdk/server.StartCmd.func1(0xc000b44d80, 0x1bc8300, 0x0, 0x0, 0x0, 0x0)


	/root/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.29.0/server/start.go:40 +0xab


github.com/spf13/cobra.(*Command).execute(0xc000b44d80, 0x1bc8300, 0x0, 0x0, 0xc000b44d80, 0x1bc8300)


	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.1/command.go:698 +0x47a


github.com/spf13/cobra.(*Command).ExecuteC(0xc0000d2d80, 0xc000199d80, 0xc000495e40, 0xc2773d)


	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.1/command.go:783 +0x2dc


github.com/spf13/cobra.(*Command).Execute(0xc0000d2d80, 0xc26ff2, 0xc000187610)


	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.1/command.go:736 +0x2b


github.com/tendermint/tendermint/libs/cli.Executor.Execute(0xc0000d2d80, 0x1126ef8, 0x3, 0xc000b26e50)


	/root/go/pkg/mod/github.com/tendermint/tendermint@v0.27.4/libs/cli/setup.go:89 +0x4e


main.main()


	/sources/daemon/main.go:63 +0x53d

hleb-albau added a commit that referenced this issue Jan 28, 2019
@hleb-albau
#197 Error on replay with wrong app hash calculation
168ca21
arturalbov pushed a commit that referenced this issue Jan 29, 2019
@hleb-albau @arturalbov
#197 Error on replay with wrong app hash calculation
f4d4872
hleb-albau added a commit that referenced this issue Jan 31, 2019
@hleb-albau
#197 Error on replay with wrong app hash calculation
e4233fd
@hleb-albau
Copy link
Contributor

Fixed in 0.1.1 network.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: Bug Something isn't working Type: Specification
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cyborgshead @hleb-albau