Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wrong Block.Header.AppHash error. Osmosis 12.1.0 #3017

Closed
dzmitrykliapkou opened this issue Oct 17, 2022 · 4 comments
Closed

wrong Block.Header.AppHash error. Osmosis 12.1.0 #3017

dzmitrykliapkou opened this issue Oct 17, 2022 · 4 comments

Comments

@dzmitrykliapkou
Copy link

System information

Osmosis version: osmolabs/osmosis:12.2.0 docker image
OS & Version: Amazon Linux 2

Expected behaviour

Node works properly

Actual behaviour

We've been running Osmosis archive node for several months. After 12.1.0 upgrade our node stopped importing new blocks with error:

osmosis-mainnet  | panic: +2/3 committed an invalid block: wrong Block.Header.AppHash.  Expected E7BA913FA027AE9ADCD65BF3A36EF82E70ABD77B18E495594CE0515F506B85C2, got AC9224688AFC4294ECE8EE0A0C035B8D0B75029693D15A96F7B4027BED0AB3E9

We tried to update addrbook file and genesis file, but it has not helped. The same issue was with 12.2.0 version

Steps to reproduce the behaviour

  1. Download archive snapshot and unpack it into /data directory
  2. Download genesis etc.
  3. Run osmosis node inside docker container using the following docker-compose.yml:
version: "3"

services:
  osmosis-node:
    container_name: osmosis-mainnet
    image: osmolabs/osmosis:12.1.0
    command:
      - start
      - --moniker
      - osmosis-main
      - --pruning
      - nothing
      - --x-crisis-skip-assert-invariants
      - --rpc.laddr
      - tcp://0.0.0.0:26657
    restart: always
    ports:
      - "26656:26656"
      - "26657:26657"
      - "26658:26658"
      - "9090:9090" # GRPC
      - "1317:1317" # for rest-server
    volumes:
      - /root/config:/osmosis/.osmosisd/config
      - /data:/osmosis/.osmosisd/data
      - /root/wasm:/osmosis/.osmosisd/wasm

Logs

osmosis-mainnet  | 8:14AM INF Replay: Vote blockID={"hash":"C1D3DB500CE6C7CF79959A1FF3E5DCB89B93D3EDB89A1BC5DC0BBB677114D8F0","parts":{"hash":"254B9FCBF2BE8EA514C99B1DD43BBF12D9069479F3114F26D91FC4D0876F3848","total":1}} height=6287087 module=consensus peer=31d2c86f7957e2db91297e54c3b0456ea06c2250 round=207 type=2
osmosis-mainnet  | 8:14AM INF Replay: Vote blockID={"hash":"C1D3DB500CE6C7CF79959A1FF3E5DCB89B93D3EDB89A1BC5DC0BBB677114D8F0","parts":{"hash":"254B9FCBF2BE8EA514C99B1DD43BBF12D9069479F3114F26D91FC4D0876F3848","total":1}} height=6287087 module=consensus peer=31d2c86f7957e2db91297e54c3b0456ea06c2250 round=207 type=2
osmosis-mainnet  | 8:14AM INF Replay: Vote blockID={"hash":"C1D3DB500CE6C7CF79959A1FF3E5DCB89B93D3EDB89A1BC5DC0BBB677114D8F0","parts":{"hash":"254B9FCBF2BE8EA514C99B1DD43BBF12D9069479F3114F26D91FC4D0876F3848","total":1}} height=6287087 module=consensus peer=31d2c86f7957e2db91297e54c3b0456ea06c2250 round=207 type=2
osmosis-mainnet  | 8:14AM INF Replay: Vote blockID={"hash":"C1D3DB500CE6C7CF79959A1FF3E5DCB89B93D3EDB89A1BC5DC0BBB677114D8F0","parts":{"hash":"254B9FCBF2BE8EA514C99B1DD43BBF12D9069479F3114F26D91FC4D0876F3848","total":1}} height=6287087 module=consensus peer=31d2c86f7957e2db91297e54c3b0456ea06c2250 round=207 type=2
osmosis-mainnet  | 8:14AM INF Replay: Vote blockID={"hash":"C1D3DB500CE6C7CF79959A1FF3E5DCB89B93D3EDB89A1BC5DC0BBB677114D8F0","parts":{"hash":"254B9FCBF2BE8EA514C99B1DD43BBF12D9069479F3114F26D91FC4D0876F3848","total":1}} height=6287087 module=consensus peer=31d2c86f7957e2db91297e54c3b0456ea06c2250 round=207 type=2
osmosis-mainnet  | 8:14AM INF Replay: Vote blockID={"hash":"C1D3DB500CE6C7CF79959A1FF3E5DCB89B93D3EDB89A1BC5DC0BBB677114D8F0","parts":{"hash":"254B9FCBF2BE8EA514C99B1DD43BBF12D9069479F3114F26D91FC4D0876F3848","total":1}} height=6287087 module=consensus peer=31d2c86f7957e2db91297e54c3b0456ea06c2250 round=207 type=2
osmosis-mainnet  | 8:14AM INF Replay: BlockPart height=6287087 module=consensus peer=31d2c86f7957e2db91297e54c3b0456ea06c2250 round=0
osmosis-mainnet  | 8:14AM INF received complete proposal block hash=C1D3DB500CE6C7CF79959A1FF3E5DCB89B93D3EDB89A1BC5DC0BBB677114D8F0 height=6287087 module=consensus
osmosis-mainnet  | panic: +2/3 committed an invalid block: wrong Block.Header.AppHash.  Expected E7BA913FA027AE9ADCD65BF3A36EF82E70ABD77B18E495594CE0515F506B85C2, got AC9224688AFC4294ECE8EE0A0C035B8D0B75029693D15A96F7B4027BED0AB3E9
osmosis-mainnet  |
osmosis-mainnet  | goroutine 1 [running]:
osmosis-mainnet  | github.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc00017f180, 0x5feeef)
osmosis-mainnet  |      github.com/tendermint/tendermint@v0.34.21/consensus/state.go:1599 +0x1105
osmosis-mainnet  | github.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc00017f180, 0x5feeef)
osmosis-mainnet  |      github.com/tendermint/tendermint@v0.34.21/consensus/state.go:1568 +0x2ff
osmosis-mainnet  | github.com/tendermint/tendermint/consensus.(*State).handleCompleteProposal(0xc00017f180, 0x0?)
osmosis-mainnet  |      github.com/tendermint/tendermint@v0.34.21/consensus/state.go:1947 +0x399
osmosis-mainnet  | github.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc00017f180, {{0x2b353a0?, 0xc05614efd8?}, {0xc8ab7dd8f0?, 0xc0013e5878?}})
osmosis-mainnet  |      github.com/tendermint/tendermint@v0.34.21/consensus/state.go:837 +0x1c5
osmosis-mainnet  | github.com/tendermint/tendermint/consensus.(*State).readReplayMessage(0xc00017f180, 0x253d484?, {0x0?, 0x0?})
osmosis-mainnet  |      github.com/tendermint/tendermint@v0.34.21/consensus/replay.go:81 +0x87f
osmosis-mainnet  | github.com/tendermint/tendermint/consensus.(*State).catchupReplay(0xc00017f180, 0x5feeef)
osmosis-mainnet  |      github.com/tendermint/tendermint@v0.34.21/consensus/replay.go:160 +0x71d
osmosis-mainnet  | github.com/tendermint/tendermint/consensus.(*State).OnStart(0xc00017f180)
osmosis-mainnet  |      github.com/tendermint/tendermint@v0.34.21/consensus/state.go:324 +0x19b
osmosis-mainnet  | github.com/tendermint/tendermint/libs/service.(*BaseService).Start(0xc00017f180)
osmosis-mainnet  |      github.com/tendermint/tendermint@v0.34.21/libs/service/service.go:144 +0x2e9
osmosis-mainnet  | github.com/tendermint/tendermint/consensus.(*Reactor).OnStart(0xc001152510)
osmosis-mainnet  |      github.com/tendermint/tendermint@v0.34.21/consensus/reactor.go:86 +0x145
osmosis-mainnet  | github.com/tendermint/tendermint/libs/service.(*BaseService).Start(0xc001152510)
osmosis-mainnet  |      github.com/tendermint/tendermint@v0.34.21/libs/service/service.go:144 +0x2e9
osmosis-mainnet  | github.com/tendermint/tendermint/p2p.(*Switch).OnStart(0xc0011ef9e0)
osmosis-mainnet  |      github.com/tendermint/tendermint@v0.34.21/p2p/switch.go:227 +0x9c
osmosis-mainnet  | github.com/tendermint/tendermint/libs/service.(*BaseService).Start(0xc0011ef9e0)
osmosis-mainnet  |      github.com/tendermint/tendermint@v0.34.21/libs/service/service.go:144 +0x2e9
osmosis-mainnet  | github.com/tendermint/tendermint/node.(*Node).OnStart(0xc000272840)
osmosis-mainnet  |      github.com/tendermint/tendermint@v0.34.21/node/node.go:979 +0x38b
osmosis-mainnet  | github.com/tendermint/tendermint/libs/service.(*BaseService).Start(0xc000272840)
osmosis-mainnet  |      github.com/tendermint/tendermint@v0.34.21/libs/service/service.go:144 +0x2e9
osmosis-mainnet  | github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x2b698f8, 0xc0012fb8f0}, {0x0, 0x0}, {0x2b57440, 0xc0011a58f0}, ...}, ...)
osmosis-mainnet  |      github.com/cosmos/cosmos-sdk@v0.46.1/server/start.go:279 +0x815
osmosis-mainnet  | github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc001208000?, {0xc00117bc00?, 0x0?, 0x7?})
osmosis-mainnet  |      github.com/cosmos/cosmos-sdk@v0.46.1/server/start.go:128 +0x193
osmosis-mainnet  | github.com/spf13/cobra.(*Command).execute(0xc001208000, {0xc00117bb90, 0x7, 0x7})
osmosis-mainnet  |      github.com/spf13/cobra@v1.5.0/command.go:872 +0x694
osmosis-mainnet  | github.com/spf13/cobra.(*Command).ExecuteC(0xc00113ef00)
osmosis-mainnet  |      github.com/spf13/cobra@v1.5.0/command.go:990 +0x3b4
osmosis-mainnet  | github.com/spf13/cobra.(*Command).Execute(...)
osmosis-mainnet  |      github.com/spf13/cobra@v1.5.0/command.go:918
osmosis-mainnet  | github.com/spf13/cobra.(*Command).ExecuteContext(...)
osmosis-mainnet  |      github.com/spf13/cobra@v1.5.0/command.go:911
osmosis-mainnet  | github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x0?, {0xc000292348, 0x12})
osmosis-mainnet  |      github.com/cosmos/cosmos-sdk@v0.46.1/server/cmd/execute.go:36 +0x1eb
osmosis-mainnet  | main.main()
osmosis-mainnet  |      github.com/osmosis-labs/osmosis/v12/cmd/osmosisd/main.go:16 +0x31
osmosis-mainnet exited with code 2
osmosis-mainnet  | 8:14AM INF starting ABCI with Tendermint
@osmo-bot osmo-bot moved this to Needs Review 🔍 in Osmosis Chain Development Oct 17, 2022
@ValarDragon
Copy link
Member

So what happened here is there was a network non-determinism error on v12.0.0 at that exact block height, which got fixed in v12.1.0 (and fix is in v12.2.0)

I suspect what happened is you switched the binary at that height, so still had the same error.

@dzmitrykliapkou
Copy link
Author

Is there any way to "roll back" the database? Rather than downloading 7TB once again

@ValarDragon
Copy link
Member

There unfortunately isn't right now :(

Trying to support getting cosmos/iavl#548 pushed through, which would make this easy to do in the future

@czarcas7ic
Copy link
Member

I am going to close this issue, but please re open if this needs further assistance

Repository owner moved this from Needs Review 🔍 to Done ✅ in Osmosis Chain Development Nov 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Osmosis Chain Development
Archived in project
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ValarDragon @czarcas7ic @dzmitrykliapkou