BSC stuck at block 25 471 853

[dg123lzk]

hain config: (Chain: 56 Homestead: 0 D40: nil> DMSuport: false EIP150: 0 EIP15: 0 EP15: 0 Byzantim: 0 (ostantinople: 0 petersburg: 0 stanbul: 0, Muir 6lacir: 0, Ramamujan: 0, Miels: 0,Miroryc: 51840, Bruo: 13082000, Berlin: , y0l0 3: <nil), (atalystBlok: nil), lodon: nil), Arowdlacier: , mergfor', Euler: 18907621, Gibbs: 23846001, Nano: 21962149, Moran: 22107423, Engine: parlia)
umber: 25479269Hash: 0x0575bb7272d0fca7fb83246d678fa7fe73911bc978d09ac62d01d9a32dff8a57
ror: could not apply tx 0 0xbfofdblf8a3588281362248dcsfafh59hbeaeecd4251C23631975B19: once to high:

[kris-ss]

Hello, have you tried to restart your node?

Also, can you please share your node details. Are you running a full or a fast node? Please share your start-up command as well.

[dg123lzk]

Restart does not work
./geth_linuxv1.1.18 --datadir pwd/data --datadir.ancient pwd/ancient --networkid 56 --syncmode full --gcmode full --cache 16092 --txpool.globalslots 50000 --allow-insecure-unlock --http --http.addr "0.0.0.0" --http.port 7646 --maxpeers 100 --http.api "eth,net,web3,admin,debug,personal,txpool" --txpool.globalqueue 50000 --txpool.lifetime 5h0m0s --ws --ws.addr "0.0.0.0" --ws.api "txpool,personal,admin,eth,net,web3" --ws.origins "*" --snapshot=true

[kris-ss]

Hello,

Looks like your case is similar to #1299.

Please check if your snapshot layer is broken or not, with this command:
geth snapshot verify-state --datadir /data

Let us know the results.

[dg123lzk]

Fatal: Could not open database: open /mnt/sdf/bsc/data/geth/ancient/headers.cidx: no such file or directory

[kris-ss]

Thanks.

--datadir pwd/data --datadir.ancient pwd/ancient

Can you change this to --datadir pwd/data --datadir.ancient pwd/geth/ancient ? Or remove the datadir.ancient flag altogether?

[dg123lzk]

”./geth_linuxv1.1.18 snapshot verify-state --datadir pwd/data --datadir.ancient pwd/ancient“ or ” ./geth_linuxv1.1.18 snapshot verify-state --datadir pwd/data“ Turns out to be
“Fatal: Could not open database: ancient chain segments already extracted, please set --datadir.ancient to the correct path”

[kris-ss]

@dg123lzk

Please try the following steps described here - #628

1. Stop The node.
2. Upgrade the binary to the release (https://github.com/bnb-chain/bsc/releases/tag/v1.1.19) .
3. Start your node with --snapshot=false
4. Wait for few minutes(it depends on how fast your nodes are), until the block height is 128 higher than where it stopped.
5. Restart the node with --snapshot=true
6. The node will continue to sync and repair the corrupt data.

[dg123lzk]

After setting --snapshot=false, synchronization stops with no peers available

[blxdyx]

the same problem

t=2023-02-21T06:02:23+0000 lvl=warn msg="Snapshot extension registration failed" peer=a05e766c err="peer connected on snap without compatible eth support"
t=2023-02-21T06:02:26+0000 lvl=warn msg="Snapshot extension registration failed" peer=bd8186ab err="peer connected on snap without compatible eth support"
t=2023-02-21T06:02:35+0000 lvl=info msg="Importing heavy sidechain segment" blocks=1393 start=18,514,495 end=18,515,887
t=2023-02-21T06:02:35+0000 lvl=info msg="Skip duplicated bad block" number=18,514,495 hash=0xc4467f0eb177dff22fce6d6316872b55e02d1de15790c2ede15cddcf2cbf4be1
t=2023-02-21T06:02:35+0000 lvl=eror msg="\n########## BAD BLOCK #########\nChain config: {ChainID: 56 Homestead: 0 DAO: DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Ramanujan: 0, Niels: 0, MirrorSync: 5184000, Bruno: 13082000, Berlin: , YOLO v3: , CatalystBlock: , London: , ArrowGlacier: , MergeFork:, Euler: 18907621, Gibbs: 23846001, Nano: 21962149, Moran: 22107423, Engine: parlia}\n\nNumber: 18514495\nHash: 0xc4467f0eb177dff22fce6d6316872b55e02d1de15790c2ede15cddcf2cbf4be1\n\n\nError: could not apply tx 121 [0x8c2773e7da664de3ddfa22ba3e801b8d7205a87f488b146b47067009f9b2879d]: nonce too high: address 0x5774c26DD7266402AdF7ecBa7B041B7417720dFa, tx: 44 state: 0\n##############################\n"
t=2023-02-21T06:02:35+0000 lvl=warn msg="Synchronisation failed, dropping peer" peer=15a6f6a789a377152fc17349b0feec0cce246b9858931c90882c7793f5f82aba err="retrieved hash chain is invalid: could not apply tx 121 [0x8c2773e7da664de3ddfa22ba3e801b8d7205a87f488b146b47067009f9b2879d]: nonce too high: address 0x5774c26DD7266402AdF7ecBa7B041B7417720dFa, tx: 44 state: 0"
t=2023-02-21T06:02:35+0000 lvl=info msg="Downloader queue stats" receiptTasks=0 blockTasks=0 itemSize="53.04 KiB" throttle=4943
t=2023-02-21T06:03:03+0000 lvl=warn msg="Snapshot extension registration failed" peer=67338e12 err="peer connected on snap without compatible eth support"
t=2023-02-21T06:03:08+0000 lvl=warn msg="Snapshot extension registration failed" peer=cb5b9771 err="peer connected on snap without compatible eth support"
t=2023-02-21T06:03:27+0000 lvl=warn msg="Snapshot extension registration failed" peer=a4da81c9 err="peer connected on snap without compatible eth support"
t=2023-02-21T06:03:42+0000 lvl=warn msg="Snapshot extension registration failed" peer=67338e12 err="peer connected on snap without compatible eth support"
t=2023-02-21T06:04:01+0000 lvl=info msg="Importing heavy sidechain segment" blocks=1393 start=18,514,495 end=18,515,887
t=2023-02-21T06:04:01+0000 lvl=info msg="Skip duplicated bad block" number=18,514,495 hash=0xc4467f0eb177dff22fce6d6316872b55e02d1de15790c2ede15cddcf2cbf4be1
t=2023-02-21T06:04:01+0000 lvl=eror msg="\n########## BAD BLOCK #########\nChain config: {ChainID: 56 Homestead: 0 DAO: DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Ramanujan: 0, Niels: 0, MirrorSync: 5184000, Bruno: 13082000, Berlin: , YOLO v3: , CatalystBlock: , London: , ArrowGlacier: , MergeFork:, Euler: 18907621, Gibbs: 23846001, Nano: 21962149, Moran: 22107423, Engine: parlia}\n\nNumber: 18514495\nHash: 0xc4467f0eb177dff22fce6d6316872b55e02d1de15790c2ede15cddcf2cbf4be1\n\n\nError: could not apply tx 121 [0x8c2773e7da664de3ddfa22ba3e801b8d7205a87f488b146b47067009f9b2879d]: nonce too high: address 0x5774c26DD7266402AdF7ecBa7B041B7417720dFa, tx: 44 state: 0\n##############################\n"
t=2023-02-21T06:04:01+0000 lvl=warn msg="Synchronisation failed, dropping peer" peer=f3707dc660431fcdf7de3525f05f15186c4ccc2d8d6fd6e210f4ae94000fa0d4 err="retrieved hash chain is invalid: could not apply tx 121 [0x8c2773e7da664de3ddfa22ba3e801b8d7205a87f488b146b47067009f9b2879d]: nonce too high: address 0x5774c26DD7266402AdF7ecBa7B041B7417720dFa, tx: 44 state: 0"
t=2023-02-21T06:04:01+0000 lvl=info msg="Downloader queue stats" receiptTasks=0 blockTasks=0 itemSize="45.14 KiB" throttle=5808
t=2023-02-21T06:04:10+0000 lvl=warn msg="Snapshot extension registration failed" peer=cb5b9771 err="peer connected on snap without compatible eth support"
t=2023-02-21T06:04:19+0000 lvl=warn msg="Snapshot extension registration failed" peer=7bcee16b err="peer connected on snap without compatible eth support"
t=2023-02-21T06:04:31+0000 lvl=warn msg="Snapshot extension registration failed" peer=dd74b5b2 err="peer connected on snap without compatible eth support"
t=2023-02-21T06:04:32+0000 lvl=warn msg="Snapshot extension registration failed" peer=6b5f4ace err="peer connected on snap without compatible eth support"
t=2023-02-21T06:04:37+0000 lvl=warn msg="Snapshot extension registration failed" peer=67338e12 err="peer connected on snap without compatible eth support"
t=2023-02-21T06:04:49+0000 lvl=warn msg="Snapshot extension registration failed" peer=a4da81c9 err="peer connected on snap without compatible eth support"
t=2023-02-21T06:05:14+0000 lvl=warn msg="Snapshot extension registration failed" peer=6a0bcd3f err="peer connected on snap without compatible eth support"
t=2023-02-21T06:05:20+0000 lvl=warn msg="Snapshot extension registration failed" peer=f2b441d2 err="peer connected on snap without compatible eth support"
t=2023-02-21T06:05:27+0000 lvl=warn msg="Snapshot extension registration failed" peer=6b5f4ace err="peer connected on snap without compatible eth support"
t=2023-02-21T06:05:27+0000 lvl=info msg="Importing heavy sidechain segment" blocks=1393 start=18,514,495 end=18,515,887
t=2023-02-21T06:05:27+0000 lvl=info msg="Skip duplicated bad block" number=18,514,495 hash=0xc4467f0eb177dff22fce6d6316872b55e02d1de15790c2ede15cddcf2cbf4be1
t=2023-02-21T06:05:27+0000 lvl=eror msg="\n########## BAD BLOCK #########\nChain config: {ChainID: 56 Homestead: 0 DAO: DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Ramanujan: 0, Niels: 0, MirrorSync: 5184000, Bruno: 13082000, Berlin: , YOLO v3: , CatalystBlock: , London: , ArrowGlacier: , MergeFork:, Euler: 18907621, Gibbs: 23846001, Nano: 21962149, Moran: 22107423, Engine: parlia}\n\nNumber: 18514495\nHash: 0xc4467f0eb177dff22fce6d6316872b55e02d1de15790c2ede15cddcf2cbf4be1\n\n\nError: could not apply tx 121 [0x8c2773e7da664de3ddfa22ba3e801b8d7205a87f488b146b47067009f9b2879d]: nonce too high: address 0x5774c26DD7266402AdF7ecBa7B041B7417720dFa, tx: 44 state: 0\n##############################\n"
t=2023-02-21T06:05:27+0000 lvl=warn msg="Synchronisation failed, dropping peer" peer=21cf1e757c0e76ec44f779179d3ad354e26a28c10e5da03513bf1a30a2778146 err="retrieved hash chain is invalid: could not apply tx 121 [0x8c2773e7da664de3ddfa22ba3e801b8d7205a87f488b146b47067009f9b2879d]: nonce too high: address 0x5774c26DD7266402AdF7ecBa7B041B7417720dFa, tx: 44 state: 0"
t=2023-02-21T06:05:28+0000 lvl=info msg="Downloader queue stats" receiptTasks=0 blockTasks=0 itemSize="40.13 KiB" throttle=6533

[viraboy]

Same situation happened twice on last two weeks on v1.118 & v1.1.19 for two different servers:

1. First time was able to get out of problem with:
   geth snapshot prune-block --datadir /data/.ethereum --datadir.ancient /data/.ethereum/geth/chaindata/ancient --block-amount-reserved xxxxxx (be sure amount takes you after BADBlock).

after this restart with flag: ---tries-verify-mode none

2. Second time was not able to recover using either of methods described here and in other posts:
3. --snapshot=false/--snashopt-true method
4. -- get snapshot prune-block described above
5. debug.setHead("hex block before badblock")

Any idea what could be reason/cause for this, any other factors that may contribute to it (i.e slowness, hw i/o issues, etc) ?

[brilliant-lx]

We can observed similar issue in our device, which is caused database damage by force kill.
Since fast node does not keep the MPT, it relies on the integrity of snapshot.
Sometime, the shutdown of the node could take a lot time, if it is managed by a systemd service, it could be force killed due to shutdown time out.
It can not be recovered in this case, have to remove the local damaged data and resync from the new snapshot file again.

[brilliant-lx]

close it, since there is no way to recover a broken fast node right now. user have to re-deploy the node from scratch.