leveldb/table: corruption on table-footer (pos=1966032): bad magic number

[n10n]

Please answer these questions before submitting your issue. Thanks!

What did you do?

Running from MacOS terminal with the options "geth --cache 4096" and no intervention

What did you expect to see?

Blockchain should be synced but failed. When try to restart the "geth" process after restarting got the error:

Fatal: Error starting protocol stack: leveldb/table: corruption on table-footer (pos=1966032): bad magic number [file=044240.ldb]

What did you see instead?

Here is error:
INFO [05-11|13:42:42] Imported new chain segment blocks=12 txs=303 mgas=8.266 elapsed=778.637ms mgasps=10.616 number=3688802 hash=70721c…6cd55b
INFO [05-11|13:42:51] Imported new chain segment blocks=18 txs=311 mgas=10.156 elapsed=9.115s mgasps=1.114 number=3688820 hash=aa96c5…4ab0d5
INFO [05-11|13:42:52] Generating ethash verification cache epoch=124 percentage=25 elapsed=3.017s
INFO [05-11|13:42:55] Generating ethash verification cache epoch=124 percentage=66 elapsed=6.022s
INFO [05-11|13:42:56] Generated ethash verification cache epoch=124 elapsed=7.192s
INFO [05-11|13:42:59] Imported new chain segment blocks=88 txs=1587 mgas=66.977 elapsed=8.077s mgasps=8.292 number=3688908 hash=b17b19…3d63a5
INFO [05-11|13:43:07] Imported new chain segment blocks=139 txs=2599 mgas=107.943 elapsed=8.028s mgasps=13.445 number=3689047 hash=edbc67…fe68be
INFO [05-11|13:43:15] Imported new chain segment blocks=142 txs=2519 mgas=120.240 elapsed=8.066s mgasps=14.907 number=3689189 hash=3321cd…a527b9
WARN [05-11|13:43:17] Synchronisation failed, dropping peer peer=8cca25d44598cec3 err="retrieved hash chain is invalid"
WARN [05-11|13:43:20] Synchronisation failed, dropping peer peer=1a88b54028909af6 err="retrieved hash chain is invalid"
WARN [05-11|13:43:27] Synchronisation failed, dropping peer peer=2ba022b961f800e0 err="action from bad peer ignored"
WARN [05-11|13:43:55] Synchronisation failed, dropping peer peer=a1299ffc3c4e6a64 err="retrieved hash chain is invalid"
WARN [05-11|13:43:58] Synchronisation failed, dropping peer peer=378c0e8ae21fc735 err="retrieved hash chain is invalid"
WARN [05-11|13:44:02] Synchronisation failed, dropping peer peer=c1b63994a7a98bd1 err="retrieved hash chain is invalid"
WARN [05-11|13:47:20] Synchronisation failed, dropping peer peer=696632807228f69c err="retrieved hash chain is invalid"
WARN [05-11|13:47:25] Synchronisation failed, dropping peer peer=df1c8d9a6fa754ab err="retrieved hash chain is invalid"
WARN [05-11|13:49:37] Synchronisation failed, dropping peer peer=696632807228f69c err="retrieved hash chain is invalid"
WARN [05-11|13:49:52] Synchronisation failed, dropping peer peer=df5aa7f0d61184bc err="retrieved hash chain is invalid"
WARN [05-11|13:49:58] Synchronisation failed, dropping peer peer=117579643dcc13e2 err="retrieved hash chain is invalid"
WARN [05-11|13:51:03] Synchronisation failed, dropping peer peer=1a88b54028909af6 err="retrieved hash chain is invalid"
WARN [05-11|13:51:42] Synchronisation failed, dropping peer peer=2ecee00caef4a44e err="retrieved hash chain is invalid"
WARN [05-11|13:52:45] Synchronisation failed, dropping peer peer=95cb0e92896b8d39 err="retrieved hash chain is invalid"
WARN [05-11|13:52:49] Synchronisation failed, dropping peer peer=cd3c33a7c5ca4b92 err="retrieved hash chain is invalid"
WARN [05-11|13:52:57] Synchronisation failed, dropping peer peer=f01fe2f982e19709 err="retrieved hash chain is invalid"
WARN [05-11|13:53:04] Synchronisation failed, dropping peer peer=4deaab63c90013db err="retrieved hash chain is invalid"
WARN [05-11|13:53:12] Synchronisation failed, dropping peer peer=696632807228f69c err="action from bad peer ignored"
WARN [05-11|13:54:08] Synchronisation failed, dropping peer peer=696632807228f6

System details

Version: 1.6.0-stable
Go Version: go1.8.1
OS: darwin
uname -v: Darwin Kernel Version 16.4.0: Thu Dec 22 22:53:21 PST 2016; root:xnu-3789.41.3~3/RELEASE_X86_64
ProductName: Mac OS X
ProductVersion: 10.12.3
BuildVersion: 16D32

[jvilaikeo]

when i saw this issue, i went into the chaindata folder and deleted the offending ldb file and restart the sync. It seems only that block is corrupted.

[n10n]

I tried by deleting the offending ldb files as you did and in my case, I got error with next file and then file. After three tries I gave and I got the error:
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Engine: ethash}
Number: 3689237
Hash: 0xafbcb0575a442b7f68e3de544643d6515048aa52de7d04a2c22f5159c0716a0e
Error: insufficient balance to pay for gas
##############################
Synchronisation failed, dropping peer peer=4ae417bcced3f93c err="retrieved hash chain is invalid"

My challenge is that I could never run geth successfully for few days and I get these sync/backblock kind of problems after every other day. For syncing, I copy the "chaindata" folder from another ubuntu server then run geth on my macbook. This solution sucks and would not like to do.

[fjl]

What kind of disk are you using. We have seen reports of leveldb corruption when using USB flash drive, but it doesn't sound like you're using one.

[n10n]

Using SSD and "binaries" and "chaindata" are in different SSD (macbook osx 10.12.3 (16D32)). After restarting again, got the following error (if that helps):

Error: gas limit reached
##############################

WARN [05-12|17:40:51] Synchronisation failed, dropping peer peer=fd96f4baea51d8ff err="retrieved hash chain is invalid"
INFO [05-12|17:41:06] Imported new chain segment blocks=4 txs=237 mgas=15.443 elapsed=5.199s mgasps=2.970 number=3689447 hash=660b44…339760
ERROR[05-12|17:41:08] Unhandled trie error: Missing trie node 7a811db37f244f69d7c4e761ed21db524d45edf39c0b58f54645e1b3f4a68c7e
panic: runtime error: index out of range

goroutine 9764 [running]:
github.com/ethereum/go-ethereum/trie.compactHexEncode(0xc42e0b7e00, 0x5, 0x8, 0x20, 0x0, 0x0)
/Users/travis/gopath/src/github.com/ethereum/go-ethereum/trie/encoding.go:86 +0xfb
github.com/ethereum/go-ethereum/trie.(*Trie).resolveHash(0xc4d2d214a0, 0xc42741ea60, 0x20, 0x20, 0xc42e0b7e00, 0x4, 0x8, 0xc4c6b1dfef, 0x1, 0x1, ...)
/Users/travis/gopath/src/github.com/ethereum/go-ethereum/trie/trie.go:453 +0x236
github.com/ethereum/go-ethereum/trie.(*Trie).resolve(0xc4d2d214a0, 0x5072a20, 0xc42741ea80, 0xc42e0b7e00, 0x4, 0x8, 0xc4c6b1dfef, 0x1, 0x1, 0x402cc01, ...)
/Users/travis/gopath/src/github.com/ethereum/go-ethereum/trie/trie.go:440 +0xcd
github.com/ethereum/go-ethereum/trie.(*Trie).delete(0xc4d2d214a0, 0x50707e0, 0xc42d988280, 0xc42e0b7e00, 0x4, 0x8, 0xc42f311634, 0x3d, 0x3d, 0x0, ...)
/Users/travis/gopath/src/github.com/ethereum/go-ethereum/trie/trie.go:390 +0x36e
github.com/ethereum/go-ethereum/trie.(*Trie).delete(0xc4d2d214a0, 0x50707e0, 0xc42d9883c0, 0xc42e0b7e00, 0x3, 0x8, 0xc42f311633, 0x3e, 0x3e, 0xc428ba3180, ...)
/Users/travis/gopath/src/github.com/ethereum/go-ethereum/trie/trie.go:354 +0x156
github.com/ethereum/go-ethereum/trie.(*Trie).delete(0xc4d2d214a0, 0x50707e0, 0xc42d988500, 0xc42e0b7e00, 0x2, 0x8, 0xc42f311632, 0x3f, 0x3f, 0xc42e0b7de0, ...)
/Users/travis/gopath/src/github.com/

[n10n]

Another error after restoring the "chaindata":
INFO [05-19|14:57:39] Loaded most recent local full block number=3731518 hash=383838…9c9d01 td=274417596533562134447
INFO [05-19|14:57:39] Loaded most recent local fast block number=3731518 hash=383838…9c9d01 td=274417596533562134447
WARN [05-19|14:57:39] Blockchain not empty, fast sync disabled
INFO [05-19|14:57:39] Starting P2P networking
INFO [05-19|14:57:39] RLPx listener up self=enode://203d1a97dd093af2bffce4a0f2822540b910db19078ba1798414753f0042c27a297735c623ce8703481e69e69cd88fc7ffde7258a0efba4e896ea1328d73a60e@10.0.1.2:30303
INFO [05-19|14:57:39] IPC endpoint opened: /Users/n/Library/Ethereum/geth.ipc
INFO [05-19|14:57:39] Mapped network port proto=udp extport=30303 intport=30303 interface=NAT-PMP(192.168.2.1)
INFO [05-19|14:57:39] Mapped network port proto=tcp extport=30303 intport=30303 interface=NAT-PMP(192.168.2.1)
INFO [05-19|14:57:59] Block synchronisation started
panic: node 5c49fc4c53075bf00470eb692a44f94d664abd8afcfbf7f93a86f004695fd5fd: invalid number of list elements: 16

goroutine 846 [running]:
github.com/ethereum/go-ethereum/trie.mustDecodeNode(0xc4238090c0, 0x20, 0x20, 0xc423692fc0, 0x53, 0x60, 0x0, 0x0, 0x0)
/Users/travis/gopath/src/github.com/ethereum/go-ethereum/trie/node.go:110 +0x204
github.com/ethereum/go-ethereum/trie.(*Trie).resolveHash(0xc4235714a0, 0xc4238090c0, 0x20, 0x20, 0xc42379ea50, 0x3, 0x41, 0xc42379ea53, 0x3e, 0x3e, ...)
/Users/travis/gopath/src/github.com/ethereum/go-ethereum/trie/trie.go:458 +0x424
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc4235714a0, 0x5072a20, 0xc4238090e0, 0xc42379ea50, 0x41, 0x41, 0x3, 0x200, 0x43f0000, 0x50707e0, ...)
/Users/travis/gopath/src/github.com/ethereum/go-ethereum/trie/trie.go:182 +0x137
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc4235714a0, 0x50707e0, 0xc4238068c0, 0xc42379ea50, 0x41, 0x41, 0x2, 0xc42379ea52, 0x3f, 0x3f, ...)
/Users/travis/gopath/src/github.com/ethereum/go-ethereum/trie/trie.go:174 +0x87b
github.com/ethereum/go-ethereum/trie.(*Trie).tryGet(0xc4235714a0, 0x5072a20, 0xc423808e80, 0xc42379ea50, 0x41, 0x41, 0x2, 0x240, 0x43f0000, 0x50707e0, ...)
/Users/travis/gopath/src/github.com/ethereum/go-ethereum/trie/trie.go:186 +0x276

[jvilaikeo]

Quick question, are you mining on the same machine as you are loading the wallet?

[rpavlovs]

Same error.
Was doing a full sync and unplugged external HDD (which it was using to store the DB) while the laptop was in sleep.

Is there any way to recover?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.