-
Notifications
You must be signed in to change notification settings - Fork 12
Troubleshooting
This page details commonly encountered errors and possible resolutions.
{"level":"info","ts":1728330119.1452067,"logger":"kwild.cometbft","caller":"p2p/switch.go:714","msg":"Ignoring inbound connection: error while adding peer","module":"p2p","err":"duplicate ID<02b2dd22472c7b8f539bf2bee472ef03d76d0aa2>","id":"02b2dd22472c7b8f539bf2bee472ef03d76d0aa2"}
There are two node instances running with the same private key.
Stop the duplicate instance. Make a new one with a different private key.
If may be necessary to either change the IP address of the new node, or on the other nodes that were rejecting inbound, delete abci/addrbook.json
Possible bug. The node made change to state that were different to what the validators who approved the block got.
Check your kwil version. Stop, drop the postgres data, restart to being replaying blocks.
DETAILS NEEDED -- SHOW LOG WITH "expected" MESSAGE
- check node's peers
- check other nodes' block counts with RPC
- ensure more than 2/3 of validators are online.
timed out waiting for tx to be included in a block
Usually this is with the broadcast mode that waits for it to be included in a block.
Either keep checking with utils query-tx, or blocks are not being mined.
When attempting to use statesync to sync a new node:
"msg":"failed to fetch and verify app hash","module":"statesync","err":"verify from #20401 to #20402 failed: old header has expired at 2024-10-03 14:19:34.658537651 +0000 UTC (now: 2024-10-07 07:32:29.50737627 +0000 UTC m=+24.376991152)"
Edit config.toml with chain.p2p.statesync.trust_period = 10000h
found conflicting vote from ourselves; did you unsafe_reset a validator?
"msg_type":"*consensus.VoteMessage","err":"conflicting votes from validator B04C9C60030FB70E55C706605A727AA5AC2700DD""
Two possible causes:
- You have two validators running with the same private key.
- If this is halting the entire chain, the majority of validators committed to a block, then deployed a change. This will necessitate a block rollback and rebuild of state. See https://github.com/kwilteam/kwil-db/wiki/Node-Recovery#drop-only-the-postgresql-database
error during handshake: error on replay: app block height (22496) is higher than core (22495)
higher up:
"msg":"Preparing ABCI application at height 22496, appHash 4a4a9e97147db6ced1bdd783f31f7d38b8528b3a143437ad35e303702c888dcd"}
The appHeight is what is reported from the postgresql (app state) tables, while the other two are from the cometbft data that was rolled back.
When postgresql database is deleted/re-created empty, it will start appHeight at 0, then reapply all blocks fairly quickly to rebuild state from the blockchain data.
During statesync, you may see
"caller":"light/client.go:604","msg":"Can't verify","module":"statesync","module":"light","err":"verify from #21421 to #21422 failed: old header has expired at 2024-10-03 16:11:21.252976076 +0000 UTC (now: 2024-10-07 07:34:08.596417791 +0000 UTC m=+19.144896087)"
This is caused by a "trust period" that is too short. Either pick new trusted RPC servers for the chain.statsync.rpc_servers setting, or increase chain.statsync.trust_period to something very large such as "10000h".
If you are a validator replaying blocks or having just performed a recovery procedure, you may see:
found conflicting vote from ourselves; did you unsafe_reset a validator
In most cases this may be ignored, and it should go away once the validator has reached the chain's best block height.
It is common to see commit is for a block we do not know about accompanying this message, and it is also not a problem if the node gets to the latest block successfully.
"msg":"propose step; failed signing proposal","module":"consensus","height":22496,"round":0,"err":"height regression: height regression. Got 22496, last height 22497"
This usually indicates you did a rollback but did not delete the signing folder.