Synchronisation failed "retrieved hash chain is invalid"

[bloodyroar]

Hi guys,

I setted up my wallet a few days ago and I didn't get that ethernet had to download all the blocks first, before working properly. I shut down my PC and now, after restarting my wallet it won't download at all... Standing at 0% all the time. My logs prompt me this all the time: rror: invalid receipt root hash (remote: 5fef6fae9510d239d366d15c2ccdd68e0fdd20dcdab530587ba5fbf1c7d97c6f local: 37e43c2d6a2d56f508033bc154229c66957a151d7b3348f8b94cec20476a9a45)

WARN [05-06|11:46:36] Synchronisation failed, dropping peer peer=b7f0c173c382e4df err="retrieved hash chain is invalid" WARN [05-06|12:32:45] Synchronisation failed, dropping peer peer=da5861f09f35f408 err="retrieved hash chain is invalid" WARN [05-06|12:34:07] Synchronisation failed, retrying err="block download canceled (requested)" ERROR[05-06|12:34:35] Unhandled trie error: Missing trie node 7999fe2a6da24f391bbd2a185a6873e616070104318356056eee5bd31b0490d0 ERROR[05-06|12:34:35] Unhandled trie error: Missing trie node 7999fe2a6da24f391bbd2a185a6873e616070104318356056eee5bd31b0490d0 ERROR[05-06|12:34:35] Unhandled trie error: Missing trie node 7999fe2a6da24f391bbd2a185a6873e616070104318356056eee5bd31b0490d0 ERROR[05-06|12:34:35] Unhandled trie error: Missing trie node 7999fe2a6da24f391bbd2a185a6873e616070104318356056eee5bd31b0490d0 ERROR[05-06|12:34:35] Unhandled trie error: Missing trie node 7999fe2a6da24f391bbd2a185a6873e616070104318356056eee5bd31b0490d0 ERROR[05-06|12:34:35] Unhandled trie error: Missing trie node 7999fe2a6da24f391bbd2a185a6873e616070104318356056eee5bd31b0490d0 ERROR[05-06|12:34:35] Unhandled trie error: Missing trie node 7999fe2a6da24f391bbd2a185a6873e616070104318356056eee5bd31b0490d0 ERROR[05-06|12:34:36] Unhandled trie error: Missing trie node 7999fe2a6da24f391bbd2a185a6873e616070104318356056eee5bd31b0490d0 ERROR[05-06|12:34:36] Unhandled trie error: Missing trie node 7999fe2a6da24f391bbd2a185a6873e616070104318356056eee5bd31b0490d0 ERROR[05-06|12:34:36]

BAD BLOCK

Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Engine: ethash}

Number: 1465140 Hash: 0xfdbac16bfe74d866e1b0f74d88b0eb9b2c93d6d4ba8d2fe04803006f83a50e40 receipt{med=295b3e37ed34aeef9539782310a0d0ff3253ba4a2a3167c3b741fcaae5b256f8 cgas=21000 bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 logs=[]} receipt{med=2208891aa6f943a0841b3485451f731e812d957f8b41a49e082a7e090318a587 cgas=42000 bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 logs=[]} receipt{med=0e1e621013ad50d03de28bfc7bb8e1dec4eb61b9c8970caf3fc59451f6672b92 cgas=63000 bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 logs=[]} receipt{med=3087bc324d27e21fc5bdd36ca5f8e9cfdda8835c01250a7fbe87034ae9ee23ce cgas=84000 bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 logs=[]} receipt{med=3aa24f734a44a95e03e8e65710ea82d4bf6f2bea914b47e006bb6254a5323358 cgas=105000 bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 logs=[]} receipt{med=82d0b68313684feb6226e3d3c17a9664d3adf215ec8f56b4da734e818ecae0c5 cgas=127423 bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000040001000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000080400000000000000000 logs=[log: 7fc77bed597bf36d7dcecaf9d1c0d825b8acf85b [e1fffcc4923d04b559f4d29a8bfc6cda04eb5b0d3c460751c2402c5c5cc9109c] 000000000000000000000000ea674fdde714fd979de3edf0f56aa9716b898ec80000000000000000000000000000000000000000000000000160a3e2b8323552 ef14b7d518b85eb2da16d28738db83f8e86a0b3eeb9a01c3b1efecaa7d022607 5 fdbac16bfe74d866e1b0f74d88b0eb9b2c93d6d4ba8d2fe04803006f83a50e40 0]} receipt{med=e69b092263aa9969ce9d5d138572b7b2dacd2807b0026e5ca360db40405cbb05 cgas=148423 bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 logs=[]} receipt{med=c91321451572427d1f8b7ebe1abfad0d3606a118c6631077a28c787396d09964 cgas=169423 bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 logs=[]} receipt{med=5a3c1c64f675d65cc0c2d6ff2967a48c65037c9e9f4a35b0ea0c7e3470aa6083 cgas=190423 bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 logs=[]} receipt{med=43a7668b8387a70787620764148a0e9f4cd86e05293e38d20b8314d101799e7e cgas=211423 bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 logs=[]} receipt{med=329bf6ccc2c45d6fe4a265fe9a9942e0fdabdecc5b26da02c05993fb50733295 cgas=232423 bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 logs=[]} receipt{med=296fe7474dc16db8e7192d224459a8148d3b6d2463f7fd6a1fa0bf11696f9b6e cgas=253423 bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 logs=[]} receipt{med=6f9ddde56ae5b7f71f1991aaca77e8c02a0f0de008554c0b3097270e7a220229 cgas=274423 bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 logs=[]} receipt{med=a62921fde935d287bfe58b44f346cddf6b9e0471de4c7cfeef90252bc2db7811 cgas=295423 bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 logs=[]}

Error: invalid receipt root hash (remote: 5fef6fae9510d239d366d15c2ccdd68e0fdd20dcdab530587ba5fbf1c7d97c6f local: 37e43c2d6a2d56f508033bc154229c66957a151d7b3348f8b94cec20476a9a45)

Can anyone help me out with that?

[karalabe]

Apparently there is data missing from your database. This can happen if your PC was shut down abruptly while Geth was in the middle of writing database entries. I don't think there's an easy way our of this, the db is corrupted. You'll have to remove your chaindata geth removedb and sync from the start.

[bloodyroar]

Thanks a lot for your reply!!
Well, then I have another serious problem: my password is lost, but I already got transferred some ETH on it, when it was at 74$. The money is los, right...? 🙁

If so: how do remove the database? Just uninstall everything and start again? What is this "geth removedb" Do I have to type this anywhere in the console?

[n10n]

karalabe, In my case, the behavior has been very inconsistent. I didn't shutdown or restart my machine at all. I have to reset/restore my chaindata in every other day with similar kinda of issues in past few weeks and today, I got different issue for that I have opened a different issue (ERROR[05-11|19:49:41] Failed to close database database=../Ethereum/geth/chaindata err="leveldb/table: corruption on table-footer (pos=2166570): bad magic number [file=044238.ldb]" #14465). This data corruption is happening way more frequently, I would like to see (or anybody would like to see). I'm not able to find a pattern yet but corruption is very common on my Macbook. Let me know if you want to see any log information etc.

[EvilGeniusFromHell]

Not a fix but an improvement if:

• You run another node in the local network and add it with admin.addPeer command
• You run geth separatedly before starting the wallet from a terminal

[mascondante]

The database is stored is separate files I believe. Would it be possible to turn this into a feature request to add a --repairdb option to roll back the database back to where it was working and pick up from there?

Edit: I'm trying to salvage the database by rolling back to when it seems something froze earlier today. If I have any luck snipping out individual corrupt files I'll let everyone know.

Edit 2: That was a no go. I had to delete the database and start over.

[ethought]

I found a simple restart of Geth fixed this for me.

[voldyy]

Similar to @ethought, I just exited my command prompt, and restarted the geth upgradedb process and it worked. Otherwise I was having the same error initially as well.

[kot-begemot]

It would be great to have something like geth --delete-blocks 1000 and that deletes last 1000 blocks from database.

Or hard set database of some block, say geth resetdb 1500000, where 1500000 is desired block number

[karalabe]

Like debug.setHead(1500000)? ;)

[kot-begemot]

Should have waited a bit with removing database ;)

I checked a bit on the internet and in manual and could not find this there. Would be convenient to have that as geth arg.

[ykeisuke]

debug.setHead(1500000)

This work for me:
debug.setHead('0x16E360')
(1500000 is 0x16E360 using hex.)

[liuhuanhui]

I have this problem, but I don't know why

[avatar-lavventura]

I am having the same error.

I have clique private proof-of-authority chain.

Geth version: Version: 1.8.15-unstable

The way I run my geth:

geth --syncmode fast --cache=1024 --shh --datadir $DATADIR/private --rpcaddr 127.0.0.1 --rpc --rpcport 8545 --rpccorsdomain="*" --networkid 12345 --rpcapi admin,eth,net,web3,debug,personal,shh

---

Error I am having on multiple nodes that are connected into the network.

########## BAD BLOCK #########
Chain config: {ChainID: 23422 Homestead: 1 DAO: <nil> DAOSupport: false EIP150: 2 EIP155: 3 EIP158: 3 Byzantium: 4 Constantinople: <nil> Engine: clique}

Number: 1260001
Hash: 0x659e96f35e1fa1c39fc3b8370a336f78787e482aef44e56bbe6dd9e10bb06bdc


Error: recently signed
##############################
 
WARN [10-05|10:36:26.616] Synchronisation failed, dropping peer    peer=fb6215e1f7ae62b3 err="retrieved hash chain is invalid"
INFO [10-05|10:36:26.617] message loop                             peer=fb6215e1f7ae62b3 err=EOF

WARN [10-05|10:11:12.990] Synchronisation failed, dropping peer    peer=e8b7096dcd99140d err="retrieved hash chain is invalid"

---

• I have reverted back the blockchain into some previous block number, debug.setHead("0x124F80") (1200000 th block) but it did not help.

• Please note that I have to remove my chaindata geth removedb and sync from the start, which didn't help.

[holiman]

I'm closing this, original report seems to be handled.
@avatar-lavventura you're hitting upon a Clique-issue, please open a separate ticket if you're still seeing this problem.

[galets]

@avatar-lavventura FWIW, I had a similar issue today. My geth is running over ZFS, so the files are checksummed. At first, I didn't understand what is going on, but then I realized that one file was corrupted. Disk showed no error, but ZFS did catch the corruption. Every time I rolled back using debug.setHead(), it played back up until it hit the corrupt file. I ended up restoring the data from backup

Granted that it's been 3 years, you likely moved on, but just trying to make sure there is a record of what really might be happening

[avatar-lavventura]

@galets as I understand it would be smart to backup the data regularly(daily) in case of corrupton on any file, since debug.setHead() won't help to resolve it. But wouldn't debug.setHead() overwrite the corrupted file?

[MrKoberman]

I tried debug.setHead and it did not resolve the issue
Beacon backfilling failed err="retrieved hash chain is invalid: missing parent"

Any suggestions?

Geth 1.14.7
Lighthouse 5.2.1