Apphash failure - height: 6241714 - expected A630501E2FBAC8846FCB5A6530550ECFC204B527B9061CB2DFB2A1333335F103 got D448D7022BE2B1850DF10E4E3537D1730759594D5C260F2751BD7D7F8970C32D

[tuky191]

Hello,

Our nodes have failed few hours ago with an apphash failure (see below)

I was able to recover them by using your 4.1.4 tag, changing the upgrade handler and get past startup error.

Regardless, Can you confirm the current code version / commit id?

Looking at node_info of some of the validators - it shows commit 8e48e6189160e59bbd086716225dad2a228ab486, but this commit is not present in your white-whale-core repository

Can you please advise?

{"level":"error","module":"server","module":"consensus","height":6241714,"round":0,"err":"wrong Block.Header.AppHash.  Expected A630501E2FBAC8846FCB5A6530550ECFC204B527B9061CB2DFB2A1333335F103, got D448D7022BE2B1850DF10E4E3537D1730759594D5C260F2751BD7D7F8970C32D","time":"2024-04-02T22:50:14Z","message":"prevote step: consensus deems this block invalid; prevoting nil"}
2024-04-02 22:50:14,054 DEBG 'cosmovisor' stdout output:
{"level":"error","module":"server","module":"consensus","height":6241714,"round":0,"err":"wrong Block.Header.AppHash.  Expected A630501E2FBAC8846FCB5A6530550ECFC204B527B9061CB2DFB2A1333335F103, got D448D7022BE2B1850DF10E4E3537D1730759594D5C260F2751BD7D7F8970C32D","time":"2024-04-02T22:50:14Z","message":"prevote step: consensus deems this block invalid; prevoting nil"}

{"level":"info","module":"server","module":"consensus","dur":1000,"height":6241714,"round":0,"step":"RoundStepPrevoteWait","time":"2024-04-02T22:50:15Z","message":"Timed out"}
2024-04-02 22:50:15,348 DEBG 'cosmovisor' stdout output:
{"level":"info","module":"server","module":"consensus","dur":1000,"height":6241714,"round":0,"step":"RoundStepPrevoteWait","time":"2024-04-02T22:50:15Z","message":"Timed out"}

{"level":"info","module":"server","module":"consensus","dur":1000,"height":6241714,"round":0,"step":"RoundStepPrecommitWait","time":"2024-04-02T22:50:16Z","message":"Timed out"}
2024-04-02 22:50:16,506 DEBG 'cosmovisor' stdout output:
{"level":"info","module":"server","module":"consensus","dur":1000,"height":6241714,"round":0,"step":"RoundStepPrecommitWait","time":"2024-04-02T22:50:16Z","message":"Timed out"}

{"level":"info","module":"server","module":"consensus","height":6241714,"round":1,"proposer":"813162AB19B250C7FA4FDAE349B065817B7125F6","time":"2024-04-02T22:50:16Z","message":"resetting proposal info"}
2024-04-02 22:50:16,506 DEBG 'cosmovisor' stdout output:
{"level":"info","module":"server","module":"consensus","height":6241714,"round":1,"proposer":"813162AB19B250C7FA4FDAE349B065817B7125F6","time":"2024-04-02T22:50:16Z","message":"resetting proposal info"}

{"level":"info","module":"server","module":"consensus","proposal":"Proposal{6241714/1 (75F423B7002706A4CCFB307DFF731EAA2A8746A3AA0AC1341B954F7F498A76AB:1:3CA6A3429E76, -1) 47204A594CD2 @ 2024-04-02T22:50:16.456710221Z}","proposer":"813162AB19B250C7FA4FDAE349B065817B7125F6","time":"2024-04-02T22:50:16Z","message":"received proposal"}
2024-04-02 22:50:16,835 DEBG 'cosmovisor' stdout output:
{"level":"info","module":"server","module":"consensus","proposal":"Proposal{6241714/1 (75F423B7002706A4CCFB307DFF731EAA2A8746A3AA0AC1341B954F7F498A76AB:1:3CA6A3429E76, -1) 47204A594CD2 @ 2024-04-02T22:50:16.456710221Z}","proposer":"813162AB19B250C7FA4FDAE349B065817B7125F6","time":"2024-04-02T22:50:16Z","message":"received proposal"}

{"level":"info","module":"server","module":"consensus","height":6241714,"hash":"75F423B7002706A4CCFB307DFF731EAA2A8746A3AA0AC1341B954F7F498A76AB","time":"2024-04-02T22:50:16Z","message":"received complete proposal block"}
2024-04-02 22:50:16,835 DEBG 'cosmovisor' stdout output:
{"level":"info","module":"server","module":"consensus","height":6241714,"hash":"75F423B7002706A4CCFB307DFF731EAA2A8746A3AA0AC1341B954F7F498A76AB","time":"2024-04-02T22:50:16Z","message":"received complete proposal block"}

{"level":"error","module":"server","module":"consensus","height":6241714,"round":1,"err":"wrong Block.Header.AppHash.  Expected A630501E2FBAC8846FCB5A6530550ECFC204B527B9061CB2DFB2A1333335F103, got D448D7022BE2B1850DF10E4E3537D1730759594D5C260F2751BD7D7F8970C32D","time":"2024-04-02T22:50:16Z","message":"prevote step: consensus deems this block invalid; prevoting nil"}
2024-04-02 22:50:16,835 DEBG 'cosmovisor' stdout output:
{"level":"error","module":"server","module":"consensus","height":6241714,"round":1,"err":"wrong Block.Header.AppHash.  Expected A630501E2FBAC8846FCB5A6530550ECFC204B527B9061CB2DFB2A1333335F103, got D448D7022BE2B1850DF10E4E3537D1730759594D5C260F2751BD7D7F8970C32D","time":"2024-04-02T22:50:16Z","message":"prevote step: consensus deems this block invalid; prevoting nil"}

{"level":"error","module":"server","module":"consensus","err":"precommit step; +2/3 prevoted for an invalid block: wrong Block.Header.AppHash.  Expected A630501E2FBAC8846FCB5A6530550ECFC204B527B9061CB2DFB2A1333335F103, got D448D7022BE2B1850DF10E4E3537D1730759594D5C260F2751BD7D7F8970C32D","stack":"goroutine 1015 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x5e\ngit.luolix.top/cometbft/cometbft/consensus.(*State).receiveRoutine.func2()\n\tgit.luolix.top/cometbft/cometbft@v0.37.4/consensus/state.go:736 +0x46\npanic({0x2e1d500?, 0xc0157e3380?})\n\truntime/panic.go:914 +0x21f\ngit.luolix.top/cometbft/cometbft/consensus.(*State).enterPrecommit(0xc001020000, 0x5f3db2, 0x1)\n\tgit.luolix.top/cometbft/cometbft@v0.37.4/consensus/state.go:1463 +0x15d9\ngit.luolix.top/cometbft/cometbft/consensus.(*State).addVote(0xc001020000, 0xc02ca9e820, {0xc0018840c0, 0x28})\n\tgit.luolix.top/cometbft/cometbft@v0.37.4/consensus/state.go:2195 +0x140f\ngit.luolix.top/cometbft/cometbft/consensus.(*State).tryAddVote(0xc001020000, 0xc02ca9e820, {0xc0018840c0?, 0xc02833bc08?})\n\tgit.luolix.top/cometbft/cometbft@v0.37.4/consensus/state.go:2012 +0x26\ngit.luolix.top/cometbft/cometbft/consensus.(*State).handleMsg(0xc001020000, {{0x4144660, 0xc02ac49538}, {0xc0018840c0, 0x28}})\n\tgit.luolix.top/cometbft/cometbft@v0.37.4/consensus/state.go:865 +0x3ce\ngit.luolix.top/cometbft/cometbft/consensus.(*State).receiveRoutine(0xc001020000, 0x0)\n\tgit.luolix.top/cometbft/cometbft@v0.37.4/consensus/state.go:772 +0x3d1\ncreated by github.com/cometbft/cometbft/consensus.(*State).OnStart in goroutine 268\n\tgit.luolix.top/cometbft/cometbft@v0.37.4/consensus/state.go:383 +0x10c\n","time":"2024-04-02T22:50:17Z","message":"CONSENSUS FAILURE!!!"}
2024-04-02 22:50:17,342 DEBG 'cosmovisor' stdout output:
{"level":"error","module":"server","module":"consensus","err":"precommit step; +2/3 prevoted for an invalid block: wrong Block.Header.AppHash.  Expected A630501E2FBAC8846FCB5A6530550ECFC204B527B9061CB2DFB2A1333335F103, got D448D7022BE2B1850DF10E4E3537D1730759594D5C260F2751BD7D7F8970C32D","stack":"goroutine 1015 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x5e\ngit.luolix.top/cometbft/cometbft/consensus.(*State).receiveRoutine.func2()\n\tgit.luolix.top/cometbft/cometbft@v0.37.4/consensus/state.go:736 +0x46\npanic({0x2e1d500?, 0xc0157e3380?})\n\truntime/panic.go:914 +0x21f\ngit.luolix.top/cometbft/cometbft/consensus.(*State).enterPrecommit(0xc001020000, 0x5f3db2, 0x1)\n\tgit.luolix.top/cometbft/cometbft@v0.37.4/consensus/state.go:1463 +0x15d9\ngit.luolix.top/cometbft/cometbft/consensus.(*State).addVote(0xc001020000, 0xc02ca9e820, {0xc0018840c0, 0x28})\n\tgit.luolix.top/cometbft/cometbft@v0.37.4/consensus/state.go:2195 +0x140f\ngit.luolix.top/cometbft/cometbft/consensus.(*State).tryAddVote(0xc001020000, 0xc02ca9e820, {0xc0018840c0?, 0xc02833bc08?})\n\tgit.luolix.top/cometbft/cometbft@v0.37.4/consensus/state.go:2012 +0x26\ngit.luolix.top/cometbft/cometbft/consensus.(*State).handleMsg(0xc001020000, {{0x4144660, 0xc02ac49538}, {0xc0018840c0, 0x28}})\n\tgit.luolix.top/cometbft/cometbft@v0.37.4/consensus/state.go:865 +0x3ce\ngit.luolix.top/cometbft/cometbft/consensus.(*State).receiveRoutine(0xc001020000, 0x0)\n\tgit.luolix.top/cometbft/cometbft@v0.37.4/consensus/state.go:772 +0x3d1\ncreated by github.com/cometbft/cometbft/consensus.(*State).OnStart in goroutine 268\n\tgit.luolix.top/cometbft/cometbft@v0.37.4/consensus/state.go:383 +0x10c\n","time":"2024-04-02T22:50:17Z","message":"CONSENSUS FAILURE!!!"}

[Sen-Com]

Hello please join the White Whale Discord and open a ticket there