Besu shows INVALID block at `18947893` on mainnet #6357

yorickdowne · 2024-01-06T12:46:07Z

Description

Besu is showing an invalid block at 18947893 on mainnet. This has been reported across the ecosystem on 23.10.1, 23.10.2, 23.10.3 and 24.1-SNAPSHOT

Most Besu do not recover from this. I have one that did. Will add logs.

Logs (if a bug)

Logs of Besu that recovered

eth-main-execution-1  | 2024-01-06 11:29:25.882+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,892 / 187 tx / 16 ws / base fee 11.81 gwei / 17,562,696 (58.5%) gas / (0xc1141d53490d3b8ba41c88ef
6db4256fbddc77db8cbb03afaeb5b94e52ee42f3) in 0.528s. Peers: 25
eth-main-execution-1  | 2024-01-06 11:29:26.131+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xc1141d53490d3b8ba41c88ef6db4256fbddc77db8cbb03afaeb5b94e52ee42f3, fin
alized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 11:29:36.638+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,893 / 99 tx / 16 ws / base fee 12.07 gwei / 10,537,373 (35.1%) gas / (0xb41fc83658a61504771fa9904
d67f89b57c531cb902335358ff94c0680f05f07) in 0.336s. Peers: 25
eth-main-execution-1  | 2024-01-06 11:29:36.796+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0x
b41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:29:49.162+00:00 | vert.x-worker-thread-0 | INFO  | MainnetBlockValidator | Invalid block 18947894 (0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64): Optional[Unable to proce
ss block because parent world state 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 is not available]
eth-main-execution-1  | 2024-01-06 11:29:49.162+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, p
arentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, status: INVALID, validationError: Unable to process block because 
parent world state 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 is not available
eth-main-execution-1  | 2024-01-06 11:30:50.733+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, p
arentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad blo
ck manager.
eth-main-execution-1  | 2024-01-06 11:31:52.118+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, p
arentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad blo
ck manager.
eth-main-execution-1  | 2024-01-06 11:32:52.313+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, p
arentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad blo
ck manager.
eth-main-execution-1  | 2024-01-06 11:33:04.319+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,893 / 469 tx / 16 ws / base fee 12.07 gwei / 29,999,951 (100.0%) gas / (0xd521d6c7bab89f1b9995c63
397817f85b5c05f52e98a67d81aefa5bfad2dcd52) in 0.481s. Peers: 25
eth-main-execution-1  | 2024-01-06 11:34:02.847+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,894 / 663 tx / 16 ws / base fee 13.58 gwei / 29,988,026 (100.0%) gas / (0x896e0d1b5e0740622c7873c
6d381d07972f9f46291fc33c468805097fb1e29a4) in 1.852s. Peers: 25
eth-main-execution-1  | 2024-01-06 11:34:02.930+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, p
arentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad blo
ck manager.
eth-main-execution-1  | 2024-01-06 11:35:05.970+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, p
arentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad blo
ck manager.
eth-main-execution-1  | 2024-01-06 11:36:06.636+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, p
arentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad blo
ck manager.
eth-main-execution-1  | 2024-01-06 11:37:07.998+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, p
arentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad blo
ck manager.
eth-main-execution-1  | 2024-01-06 11:38:11.621+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, p
arentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad blo
ck manager.
eth-main-execution-1  | 2024-01-06 11:39:12.449+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 11:40:13.269+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 11:41:05.709+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,895 / 343 tx / 16 ws / base fee 15.27 gwei / 29,991,994 (100.0%) gas / (0x624de0f6b20b246f3c650946624e44538208ad0c6c8c50826c53c04446881530) in 0.796s. Peers: 25
eth-main-execution-1  | 2024-01-06 11:41:15.648+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 11:42:16.769+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 11:42:51.339+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,896 / 588 tx / 16 ws / base fee 17.18 gwei / 29,985,068 (100.0%) gas / (0x3c3687b0079d8e043a2791710e7c992a1b461ae70bdd5a8fbd281d37238e7b9d) in 1.073s. Peers: 25
eth-main-execution-1  | 2024-01-06 11:43:17.646+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 11:44:22.269+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 11:45:31.985+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 11:46:32.220+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 11:47:38.159+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 11:47:59.660+00:00 | vert.x-worker-thread-0 | INFO  | BackwardSyncContext | Starting a new backward sync session
eth-main-execution-1  | 2024-01-06 11:48:01.701+00:00 | nioEventLoopGroup-3-6 | INFO  | BackwardSyncStep | Backward sync phase 1 of 2 completed, downloaded a total of 200 headers. Peers: 25
eth-main-execution-1  | 2024-01-06 11:48:02.540+00:00 | ForkJoinPool.commonPool-worker-10 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 2.56% completed, imported 1 blocks of at least 39 (current head 18947893, target head 18947931). Peers: 25
eth-main-execution-1  | 2024-01-06 11:48:11.641+00:00 | nioEventLoopGroup-3-6 | INFO  | BackwardSyncStep | Backward sync phase 1 of 2 completed, downloaded a total of 200 headers. Peers: 25
eth-main-execution-1  | 2024-01-06 11:48:13.337+00:00 | nioEventLoopGroup-3-6 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 20.75% completed, imported 11 blocks of at least 53 (current head 18947903, target head 18947945). Peers: 25
eth-main-execution-1  | 2024-01-06 11:48:25.323+00:00 | nioEventLoopGroup-3-6 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 58.18% completed, imported 32 blocks of at least 55 (current head 18947924, target head 18947947). Peers: 25
eth-main-execution-1  | 2024-01-06 11:48:35.520+00:00 | nioEventLoopGroup-3-6 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 66.20% completed, imported 47 blocks of at least 71 (current head 18947939, target head 18947963). Peers: 25
eth-main-execution-1  | 2024-01-06 11:48:45.540+00:00 | ForkJoinPool.commonPool-worker-10 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 76.40% completed, imported 68 blocks of at least 89 (current head 18947960, target head 18947981). Peers: 25
eth-main-execution-1  | 2024-01-06 11:48:54.725+00:00 | ForkJoinPool.commonPool-worker-10 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2 completed, imported a total of 92 blocks. Peers: 25
eth-main-execution-1  | 2024-01-06 11:48:54.726+00:00 | ForkJoinPool.commonPool-worker-10 | INFO  | BackwardSyncAlgorithm | Current backward sync session is done
eth-main-execution-1  | 2024-01-06 11:49:01.157+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,985 / 112 tx / 16 ws / base fee 14.15 gwei / 10,104,305 (33.7%) gas / (0xb9e4a12fc34f2bc4ca5c4589af48b309f742316691050b8a6f7d9550a193d358) in 0.250s. Peers: 25
eth-main-execution-1  | 2024-01-06 11:49:01.412+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb9e4a12fc34f2bc4ca5c4589af48b309f742316691050b8a6f7d9550a193d358, finalized: 0x3e9ad67c153339f113d7f7ae736cd6ea33b133772a9169d6e5575ad79763a860, safeBlockHash: 0x9c27befd4f38b98cb13266cfb481d9250ad3f4da083a8eb38db66a0e864c32aa
eth-main-execution-1  | 2024-01-06 11:49:13.346+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,986 / 149 tx / 16 ws / base fee 13.57 gwei / 19,280,225 (64.3%) gas / (0x9af43c2a1b3f3ed5390d63f59de5c9db20fda8586e288701bfb43f3b4d24e143) in 0.352s. Peers: 25

Logs of Besu that did not recover

6db4256fbddc77db8cbb03afaeb5b94e52ee42f3) in 0.350s. Peers: 25
eth-main-execution-1  | 2024-01-06 11:29:36.191+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,893 / 99 tx / 16 ws / base fee 12.07 gwei / 10,537,373 (35.1%) gas / (0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07) in 0.203s. Peers: 25
eth-main-execution-1  | 2024-01-06 11:29:36.213+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:29:48.888+00:00 | vert.x-worker-thread-0 | INFO  | MainnetBlockValidator | Invalid block 18947894 (0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64): Optional[Unable to process block because parent world state 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 is not available]
eth-main-execution-1  | 2024-01-06 11:29:48.889+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, status: INVALID, validationError: Unable to process block because parent world state 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 is not available
eth-main-execution-1  | 2024-01-06 11:30:13.822+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:30:13.822+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 11:30:50.023+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:31:25.978+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:31:25.978+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 11:32:02.343+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:32:37.537+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:32:37.537+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 11:33:00.823+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,893 / 469 tx / 16 ws / base fee 12.07 gwei / 29,999,951 (100.0%) gas / (0xd521d6c7bab89f1b9995c63397817f85b5c05f52e98a67d81aefa5bfad2dcd52) in 0.435s. Peers: 25
eth-main-execution-1  | 2024-01-06 11:33:14.451+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:33:49.415+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:33:49.415+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 11:34:00.753+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,894 / 663 tx / 16 ws / base fee 13.58 gwei / 29,988,026 (100.0%) gas / (0x896e0d1b5e0740622c7873c6d381d07972f9f46291fc33c468805097fb1e29a4) in 0.816s. Peers: 25
eth-main-execution-1  | 2024-01-06 11:34:25.897+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0x
b41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:35:00.838+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:35:00.839+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 11:35:37.949+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:36:13.134+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:36:13.134+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 11:36:49.295+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:37:25.088+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:37:25.088+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 11:38:03.062+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:38:37.504+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:38:37.504+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 11:39:14.301+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:39:50.844+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:39:50.844+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 11:40:25.432+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:41:01.563+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:41:01.564+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 11:41:02.798+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,895 / 343 tx / 16 ws / base fee 15.27 gwei / 29,991,994 (100.0%) gas / (0x624de0f6b20b246f3c650946624e44538208ad0c6c8c50826c53c04446881530) in 0.541s. Peers: 25
eth-main-execution-1  | 2024-01-06 11:41:09.975+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 11:41:35.083+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:42:11.056+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:42:11.056+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 11:42:31.736+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 11:42:47.061+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:42:50.540+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,896 / 588 tx / 16 ws / base fee 17.18 gwei / 29,985,068 (100.0%) gas / (0x3c3687b0079d8e043a2791710e7c992a1b461ae70bdd5a8fbd281d37238e7b9d) in 0.791s. Peers: 25
eth-main-execution-1  | 2024-01-06 11:43:23.098+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:43:23.098+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 11:43:36.390+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 11:43:59.169+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:44:35.073+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:44:35.073+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 11:44:39.380+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 11:45:11.092+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:45:45.458+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 11:45:47.058+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:45:47.058+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 11:46:23.060+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 11:46:51.464+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.

In failure case, Besu does not backwards sync and stays failed, here's current logs:

eth-main-execution-1  | 2024-01-06 12:39:35.092+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 12:40:06.749+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 12:40:11.094+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 12:40:47.109+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 12:40:47.109+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 12:41:23.059+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 12:41:41.945+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 12:41:59.059+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 12:41:59.059+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 12:42:35.074+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 12:42:52.286+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,905 / 666 tx / 16 ws / base fee 29.98 gwei / 29,992,926 (100.0%) gas / (0xe9494037ab27eb6ebf2a9fafdd3cfbdd3f553c002a50c549bc5f1757669bde41) in 0.750s. Peers: 25
eth-main-execution-1  | 2024-01-06 12:43:02.654+00:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947894, hash: 0x723b828119894c0b0b21775124954e0d94bca34329a47c7e200c44fc0ffaaa64, parentHash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.
eth-main-execution-1  | 2024-01-06 12:43:11.145+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07
eth-main-execution-1  | 2024-01-06 12:43:11.145+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, finalized: 0x01a5198958cf9ffc7768603666b29401a5883699fa6c63a18fa2c8e99b4ac6c8, safeBlockHash: 0x972f93a44e4b126fc69d2fd834386ac197d4145e7e03aefb4e0a2463edef7df9
eth-main-execution-1  | 2024-01-06 12:43:41.242+00:00 | vert.x-worker-thread-0 | ERROR | MergeCoordinator | Could not persist world for root hash 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 and block hash 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07

The text was updated successfully, but these errors were encountered:

e1ghtball · 2024-01-06T12:51:55Z

Same issue

aaronmboyd · 2024-01-06T13:15:58Z

Besu/Lodestar here same issue.

Tried:

Bounce Besu
Bounce Lodestar Beacon Node

Tries to resync and hits the same issue. JVM error trace is:

execution-client  | java.lang.RuntimeException: World State Root does not match expected value, header 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 calculated 0xf26bfa5c260e327582633c0c77d8dbe900a4877ab57e067ec814acd81d4b98ba
execution-client  | 	at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldState.verifyWorldStateRoot(BonsaiWorldState.java:448)
execution-client  | 	at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldState.persist(BonsaiWorldState.java:406)
execution-client  | 	at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:194)
execution-client  | 	at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:79)
execution-client  | 	at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:207)
execution-client  | 	at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:135)
execution-client  | 	at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:83)
execution-client  | 	at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:546)
execution-client  | 	at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:576)
execution-client  | 	at org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:158)
execution-client  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:302)
execution-client  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:90)
execution-client  | 	at io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137)
execution-client  | 	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
execution-client  | 	at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135)
execution-client  | 	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
execution-client  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
execution-client  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
execution-client  | 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
execution-client  | 	at java.base/java.lang.Thread.run(Thread.java:833)

auto-call · 2024-01-06T13:25:14Z

Same here (besu+teku)

Upgraded from 23.10.2 to 23.10.3, with no luck.

Spacesider · 2024-01-06T13:26:03Z

Same thing here. Have restarted Besu multiple times, as well as Teku, but it is still stuck on:

vert.x-worker-thread-0 | WARN | AbstractEngineNewPayload | Invalid new payload: number: 18947893, hash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, parentHash: 0xc1141d53490d3b8ba41c88ef6db4256fbddc77db8cbb03afaeb5b94e52ee42f3, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.

Besu version v23.10.2

anthonygraignic · 2024-01-06T13:30:29Z

Same issue here, but luckily recovered after upgrading from 23.10.2 to 23.10.3 thanks to BackwardSync.

aaronmboyd · 2024-01-06T13:36:10Z

Same issue here, but luckily recovered after upgrading from 23.10.2 to 23.10.3 thanks to BackwardSync.

Can you describe further how you resolved please?

bulikos · 2024-01-06T13:36:33Z

Same issue here, but luckily recovered after upgrading from 23.10.2 to 23.10.3 thanks to BackwardSync.

What exactly you did after upgrading? I'm having same issue.

anthonygraignic · 2024-01-06T13:45:35Z

Same issue here, but luckily recovered after upgrading from 23.10.2 to 23.10.3 thanks to BackwardSync.

Can you describe further how you resolved please?

As @yorickdowne mentioned, some Besu recover (see attached logs in his initial post), that's what happened to me.
For the precise steps, I tried to restart the service a few times before upgrading without success and then just upgraded the docker image with the Rockerpool cli rocketpool service config and started Besu.

Logs:

2024-01-06 13:02:29.476+00:00 | main | INFO  | DefaultSynchronizer | Starting synchronizer.
2024-01-06 13:02:29.477+00:00 | main | INFO  | TransactionPoolFactory | Enabling transaction handling following initial sync
2024-01-06 13:02:29.485+00:00 | main | INFO  | DefaultSynchronizer | Stopping block propagation.
2024-01-06 13:02:29.485+00:00 | main | INFO  | DefaultSynchronizer | Stopping the pruner.
2024-01-06 13:02:29.486+00:00 | main | INFO  | Runner | Ethereum main loop is up.
2024-01-06 13:02:30.639+00:00 | vert.x-worker-thread-0 | INFO  | BackwardSyncContext | Starting a new backward sync session
2024-01-06 13:02:38.969+00:00 | nioEventLoopGroup-3-2 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, -0.68% completed, imported -3 blocks of at least 440 (current head 18947893, target head 18948336). Peers: 16
2024-01-06 13:02:50.840+00:00 | nioEventLoopGroup-3-7 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 3.40% completed, imported 15 blocks of at least 441 (current head 18947911, target head 18948337). Peers: 17
2024-01-06 13:03:01.456+00:00 | nioEventLoopGroup-3-4 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 8.84% completed, imported 39 blocks of at least 441 (current head 18947935, target head 18948337). Peers: 18
2024-01-06 13:03:11.482+00:00 | ForkJoinPool.commonPool-worker-13 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 16.97% completed, imported 75 blocks of at least 442 (current head 18947971, target head 18948338). Peers: 17
2024-01-06 13:03:21.569+00:00 | nioEventLoopGroup-3-8 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 20.09% completed, imported 89 blocks of at least 443 (current head 18947985, target head 18948339). Peers: 17
2024-01-06 13:03:31.825+00:00 | nioEventLoopGroup-3-4 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 25.00% completed, imported 111 blocks of at least 444 (current head 18948007, target head 18948340). Peers: 18
2024-01-06 13:03:46.732+00:00 | nioEventLoopGroup-3-4 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 28.31% completed, imported 126 blocks of at least 445 (current head 18948022, target head 18948341). Peers: 18
2024-01-06 13:03:56.832+00:00 | nioEventLoopGroup-3-4 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 29.82% completed, imported 133 blocks of at least 446 (current head 18948029, target head 18948342). Peers: 18
2024-01-06 13:04:07.119+00:00 | nioEventLoopGroup-3-4 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 32.44% completed, imported 145 blocks of at least 447 (current head 18948041, target head 18948343). Peers: 18
2024-01-06 13:04:17.327+00:00 | nioEventLoopGroup-3-4 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 37.50% completed, imported 168 blocks of at least 448 (current head 18948064, target head 18948344). Peers: 18
2024-01-06 13:04:28.762+00:00 | nioEventLoopGroup-3-4 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 43.65% completed, imported 196 blocks of at least 449 (current head 18948092, target head 18948345). Peers: 18
2024-01-06 13:04:38.919+00:00 | nioEventLoopGroup-3-2 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 49.67% completed, imported 223 blocks of at least 449 (current head 18948119, target head 18948345). Peers: 19
2024-01-06 13:04:49.058+00:00 | nioEventLoopGroup-3-2 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 55.23% completed, imported 248 blocks of at least 449 (current head 18948144, target head 18948345). Peers: 20
2024-01-06 13:04:59.220+00:00 | nioEventLoopGroup-3-10 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 61.56% completed, imported 277 blocks of at least 450 (current head 18948173, target head 18948346). Peers: 21
2024-01-06 13:05:09.595+00:00 | nioEventLoopGroup-3-10 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 68.07% completed, imported 307 blocks of at least 451 (current head 18948203, target head 18948347). Peers: 21
2024-01-06 13:05:19.687+00:00 | ForkJoinPool.commonPool-worker-3 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 76.55% completed, imported 346 blocks of at least 452 (current head 18948242, target head 18948348). Peers: 20
2024-01-06 13:05:30.992+00:00 | ForkJoinPool.commonPool-worker-3 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 83.00% completed, imported 376 blocks of at least 453 (current head 18948272, target head 18948349). Peers: 21
2024-01-06 13:05:41.116+00:00 | ForkJoinPool.commonPool-worker-3 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 90.97% completed, imported 413 blocks of at least 454 (current head 18948309, target head 18948350). Peers: 21
2024-01-06 13:05:51.144+00:00 | ForkJoinPool.commonPool-worker-3 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, 99.12% completed, imported 451 blocks of at least 455 (current head 18948347, target head 18948351). Peers: 21
2024-01-06 13:05:52.146+00:00 | ForkJoinPool.commonPool-worker-3 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2 completed, imported a total of 455 blocks. Peers: 21
2024-01-06 13:05:52.147+00:00 | ForkJoinPool.commonPool-worker-3 | INFO  | BackwardSyncAlgorithm | Current backward sync session is done
2024-01-06 13:05:59.842+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,948,352 / 133 tx / 16 ws / base fee 17.53 gwei / 11,106,870 (37.0%) gas / (0xbdd551554dfdc16277427f008ec098327ce1cdd24a39e4f8b0f0fd8590b4bf91) in 0.197s. Peers: 21
2024-01-06 13:06:12.592+00:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,948,353 / 181 tx / 16 ws / base fee 16.96 gwei / 17,643,128 (58.8%) gas / (0x4fcff447f7b3096f5073a5138a863dd7cd3c58b6e58e977e0b84ac95accd6b6b) in 0.244s. Peers: 21

yorickdowne · 2024-01-06T13:49:40Z

Attempts with debug_setHead (back 1, back 100 and back 1,000 before failed block were tried) and debug_replayBlock (on the failed block) did not recover from this

N0lanV · 2024-01-06T14:05:16Z

besu/v23.10.0/linux-x86_64/openjdk-java-17
Same issue:

Jan 06 12:29:13 validator-eth besu[295682]: 2024-01-06 12:29:13.472+01:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,891 / 125 tx / 16 ws / base fee 12.21 gwei / 11,071,834 (36.9%) gas / (0xa9ddaf079c9b5ed5baece459d7b7e2866983566bbb08f92c3134798eb0e67597) in 0.962s. Peers: 50
Jan 06 12:29:26 validator-eth besu[295682]: 2024-01-06 12:29:26.260+01:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #18,947,892 / 187 tx / 16 ws / base fee 11.81 gwei / 17,562,696 (58.5%) gas / (0xc1141d53490d3b8ba41c88ef6db4256fbddc77db8cbb03afaeb5b94e52ee42f3) in 1.362s. Peers: 50
Jan 06 12:29:36 validator-eth besu[295682]: 2024-01-06 12:29:36.580+01:00 | vert.x-worker-thread-0 | ERROR | AbstractBlockProcessor | failed persisting block
Jan 06 12:29:36 validator-eth besu[295682]: java.lang.RuntimeException: World State Root does not match expected value, header 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 calculated 0xf26bfa5c260e327582633c0c77d8dbe900a4877ab57e067ec814acd81d4b98ba
Jan 06 12:29:36 validator-eth besu[295682]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldState.verifyWorldStateRoot(BonsaiWorldState.java:435)
Jan 06 12:29:36 validator-eth besu[295682]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldState.persist(BonsaiWorldState.java:393)
Jan 06 12:29:36 validator-eth besu[295682]:         at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:194)
Jan 06 12:29:36 validator-eth besu[295682]:         at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:79)
Jan 06 12:29:36 validator-eth besu[295682]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:207)
Jan 06 12:29:36 validator-eth besu[295682]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:135)
Jan 06 12:29:36 validator-eth besu[295682]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:83)
Jan 06 12:29:36 validator-eth besu[295682]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:539)
Jan 06 12:29:36 validator-eth besu[295682]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:569)
Jan 06 12:29:36 validator-eth besu[295682]:         at org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:158)
Jan 06 12:29:36 validator-eth besu[295682]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:301)
Jan 06 12:29:36 validator-eth besu[295682]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:90)
Jan 06 12:29:36 validator-eth besu[295682]:         at io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137)
Jan 06 12:29:36 validator-eth besu[295682]:         at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
Jan 06 12:29:36 validator-eth besu[295682]:         at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135)
Jan 06 12:29:36 validator-eth besu[295682]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Jan 06 12:29:36 validator-eth besu[295682]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Jan 06 12:29:36 validator-eth besu[295682]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Jan 06 12:29:36 validator-eth besu[295682]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Jan 06 12:29:36 validator-eth besu[295682]:         at java.base/java.lang.Thread.run(Thread.java:840)
Jan 06 12:29:36 validator-eth besu[295682]: 2024-01-06 12:29:36.585+01:00 | vert.x-worker-thread-0 | INFO  | MainnetBlockValidator | Invalid block 18947893 (0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07): Optional[World State Root does not match expected value, header 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 calculated 0xf26bfa5c260e327582633c0c77d8dbe900a4877ab57e067ec814acd81d4b98ba], caused by java.lang.RuntimeException: World State Root does not match expected value, header 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 calculated 0xf26bfa5c260e327582633c0c77d8dbe900a4877ab57e067ec814acd81d4b98ba
Jan 06 12:29:36 validator-eth besu[295682]: 2024-01-06 12:29:36.586+01:00 | vert.x-worker-thread-0 | WARN  | AbstractEngineNewPayload | Invalid new payload: number: 18947893, hash: 0xb41fc83658a61504771fa9904d67f89b57c531cb902335358ff94c0680f05f07, parentHash: 0xc1141d53490d3b8ba41c88ef6db4256fbddc77db8cbb03afaeb5b94e52ee42f3, latestValidHash: 0xc1141d53490d3b8ba41c88ef6db4256fbddc77db8cbb03afaeb5b94e52ee42f3, status: INVALID, validationError: World State Root does not match expected value, header 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 calculated 0xf26bfa5c260e327582633c0c77d8dbe900a4877ab57e067ec814acd81d4b98ba

inDane · 2024-01-06T14:18:33Z

i experienced the same. my besu stopped at 12:47 CET (UTC+1)

Jan 06 12:47:51 validator besu[1181762]: java.lang.RuntimeException: World State Root does not match expected value, header 0xf9029a6c...

Besu 23.10.2
Ubuntu 20.04.6 LTS
openjdk 17.0.9 2023-10-17

EDIT: both of my besu machines actually died. :-(

yorickdowne · 2024-01-06T14:18:38Z

If feeling adventurous, this takes 3+ hours and may or may not help: Enable debug with --rpc-http-api WEB3,ETH,NET,DEBUG, in service or Eth Docker EL_EXTRAS, then on host for systemd or inside container for Eth Docker run curl -X POST --data '{"jsonrpc":"2.0","method":"debug_resyncWorldState","params":[],"id":1}' http://127.0.0.1:8545

It'll force a backwards sync and world state sync. We do not know this resolves it, and it does take 3+ hours depending on hardware.

inDane · 2024-01-06T14:30:48Z

If feeling adventurous, this takes 3+ hours and may or may not help: Enable debug with --rpc-http-api WEB3,ETH,NET,DEBUG, in service or Eth Docker EL_EXTRAS, then on host for systemd or inside container for Eth Docker run curl -X POST --data '{"jsonrpc":"2.0","method":"debug_resyncWorldState","params":[],"id":1}' http://127.0.0.1:8545

It'll force a backwards sync and world state sync. We do not know this resolves it, and it does take 3+ hours depending on hardware.

i am not sure if this is needed to start the backward sync.
On my bare metal installation it does show:

Jan 06 15:22:12 validator besu[854]: 2024-01-06 15:22:12.335+01:00 | nioEventLoopGroup-3-9 | INFO  | BackwardSyncStep | Backward sync phase 1 of 2, 0.27% completed, downloaded 2 headers of at least 729. Peers: 23
Jan 06 15:22:12 validator besu[854]: 2024-01-06 15:22:12.883+01:00 | ForkJoinPool.commonPool-worker-3 | INFO  | BackwardSyncStep | Backward sync phase 1 of 2 completed, downloaded a total of 733 headers. Peers: 23
Jan 06 15:22:14 validator besu[854]: 2024-01-06 15:22:14.736+01:00 | ForkJoinPool.commonPool-worker-7 | INFO  | BackwardSyncStep | Backward sync phase 1 of 2 completed, downloaded a total of 737 headers. Peers: 23
Jan 06 15:22:20 validator besu[854]: 2024-01-06 15:22:20.666+01:00 | ForkJoinPool.commonPool-worker-4 | INFO  | BackwardSyncStep | Backward sync phase 1 of 2 completed, downloaded a total of 764 headers. Peers: 23

After restarting the service. This is besu 23.10.2.

EDIT: or is this different to what you suggested?

yorickdowne · 2024-01-06T14:35:49Z

Different. Some people have a backward sync just starting on its own, we don't know yet what anyone can do to trigger it. The debug_resyncWorldState forces a more complete resync, and we don't know yet whether it'll help: For people where backward sync doesn't "just" kick in

If you are seeing a backward sync already let it complete, it should fix it

TheAviationDoctor · 2024-01-06T14:41:33Z

Working fine for me with no intervention. I can see from the logs that Besu v23.10.3 did fail to import 18,947,893 and subsequent blocks, but then did a backward sync and recovered all by itself. I'm running Nimbus v23.11.0 as my CL client if that matters.

yorickdowne · 2024-01-06T14:52:14Z

From a Besu dev: "A possible way to force a backward sync, that at least works with Teku, is to stop the CL client, delete its beacon db, be sure it is configured to checkpoint sync, and restart"

For Eth Docker users this is ./ethd resync-consensus and make sure RAPID_SYNC_URL is set in .env, e.g. RAPID_SYNC_URL=https://beaconstate.info

inDane · 2024-01-06T14:58:29Z

This cant be real, i actually missed a proposal. Im pissed now. After 5 months of nothing.

EDIT: What did i learn from this? Dont have the same software on your backup node... I swapped the EL to Erigon on my Backup node.

MarekM25 · 2024-01-06T15:41:02Z

Hey Besu team,

Traces from Nethermind node for a block 18947893. Let me know if we can help you in any other way.
traces block 18947893.zip

kooshan75 · 2024-01-06T15:50:06Z

Had the same issue on 23.10.2 and 23.10.1, updating to 23.10.3 fixed it.

matkt · 2024-01-06T16:15:14Z

I recommend to follow this https://discord.com/channels/905194001349627914/938504958909747250/1193223408058630295 for people having this issue

RogerHKW · 2024-01-06T16:19:17Z

@matkt

I recommend to follow this https://discord.com/channels/905194001349627914/938504958909747250/1193223408058630295 for people having this issue

Discord is an incredible mess (or the app is totally messed up for non subscribers).

Could you please copy paste the relevant text? Thanks

RogerHKW · 2024-01-06T16:22:32Z

@yorickdowne

From a Besu dev: "A possible way to force a backward sync, that at least works with Teku, is to stop the CL client, delete its beacon db, be sure it is configured to checkpoint sync, and restart"

For Eth Docker users this is ./ethd resync-consensus and make sure RAPID_SYNC_URL is set in .env, e.g. RAPID_SYNC_URL=https://beaconstate.info

I did it, it doesn't solve the problem on my computer. The backward synch happened but Besu is still locked in the bad block loop.

matkt · 2024-01-06T16:29:08Z

Besu world state issue recap.

Around 2024-01-06T11:29:36 UTC, Besu started reporting errors like this one:
World State Root does not match expected value, header 0xf9029a6ce0a53e912643642e3458967dd2e38edd60d77e312156d8b1c432a433 calculated 0xf26bfa5c260e327582633c0c77d8dbe900a4877ab57e067ec814acd81d4b98ba followed by many Invalid new payload messages, with the effect that Besu is not in sync and the CL client is stuck too and not able to publish attestations or blocks.

The cause of this issue are still under investigation, but after collecting feedback from user and testing some options, there are some workarounds to recover you node, until a proper fix is released.

Recovering options:

If you are still running Besu with version 23.10.2 or lower, then upgrade to 23.10.3

https://hyperledger.jfrog.io/artifactory/besu-binaries/besu/23.10.3/besu-23.10.3.tar.gz / sha256 73c834cf32c7bbe255d7d8cc7ca5d1eb0df8430b9114935c8dcf3a675b2acbc2

If you are already on version 23.10.3, then keep Besu running, and operate on your CL client, removing its beacon db and restarting, this will trigger a backward sync in Besu that could help healing the worldstate. Instructions on how to delete the beacon db, depends on your client, for example for Teku you need to remove the beacon folder in the Teku data path, for other client refer to their documentation.
If after the backward sync session the issue is still there then try point 3.
- If previous options have not worked then, you can try to resync only the world state, it can takes some hours, but it is faster from a resync from scratch, for this to work you have to enabled the DEBUG API, --rpc-http-api=ETH,NET,WEB3,DEBUG see https://besu.hyperledger.org/public-networks/reference/cli/options#rpc-http-api, and then run curl -X POST --data '{"jsonrpc":"2.0","method":"debug_resyncWorldState","params":[],"id":1}' http://localhost:8545

auto-call · 2024-01-06T16:41:31Z

@yorickdowne

From a Besu dev: "A possible way to force a backward sync, that at least works with Teku, is to stop the CL client, delete its beacon db, be sure it is configured to checkpoint sync, and restart"
For Eth Docker users this is ./ethd resync-consensus and make sure RAPID_SYNC_URL is set in .env, e.g. RAPID_SYNC_URL=https://beaconstate.info

I did it, it doesn't solve the problem on my computer. The backward synch happened but Besu is still locked in the bad block loop.

Same here. Now I initiated a world state resync. I think it takes a few hours, I will report the results.

Xexr · 2024-01-06T17:19:17Z

Updating to 23.10.3 and restarting did the trick for me with a backward sync which was failing on .2

auto-call · 2024-01-06T20:29:43Z

My validator is back up after a world state sync.

matkt · 2024-01-07T08:54:06Z

We have just created a hotfix. Even if your node works well it is better to update Besu to avoid future issues. If your node already works well, no need to do anything else, just an update and use hotfix

TheAviationDoctor · 2024-01-07T09:04:06Z

We have just created a hotfix. Even if your node works well it is better to update Besu to avoid future issues.

Fantastic reactivity, especially over a weekend. Thanks and congratulations to the team!

yorickdowne · 2024-01-07T21:39:36Z

Resolved with 23.10.3-hotfix. Thanks!

sfbahr mentioned this issue Jan 6, 2024

World State Root does not match expected value -> Error processing block/Failed to import block: Invalid block #4151

Closed

bharath-123 mentioned this issue Jan 6, 2024

update besu version stader-labs/stader-node#174

Merged

This was referenced Jan 6, 2024

Hotfix 23.10.3-hotfix Log/Address Trielog rolling failure #6315

Merged

fixing trielog generation when selfdestruct/create2 on same block #6359

Merged

yorickdowne closed this as completed Jan 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Besu shows INVALID block at `18947893` on mainnet #6357

Besu shows INVALID block at `18947893` on mainnet #6357

yorickdowne commented Jan 6, 2024

e1ghtball commented Jan 6, 2024

aaronmboyd commented Jan 6, 2024

auto-call commented Jan 6, 2024

Spacesider commented Jan 6, 2024

anthonygraignic commented Jan 6, 2024

aaronmboyd commented Jan 6, 2024

bulikos commented Jan 6, 2024

anthonygraignic commented Jan 6, 2024 •

edited

Loading

yorickdowne commented Jan 6, 2024 •

edited

Loading

N0lanV commented Jan 6, 2024

inDane commented Jan 6, 2024 •

edited

Loading

yorickdowne commented Jan 6, 2024 •

edited

Loading

inDane commented Jan 6, 2024 •

edited

Loading

yorickdowne commented Jan 6, 2024

TheAviationDoctor commented Jan 6, 2024 •

edited

Loading

yorickdowne commented Jan 6, 2024 •

edited

Loading

inDane commented Jan 6, 2024 •

edited

Loading

MarekM25 commented Jan 6, 2024

kooshan75 commented Jan 6, 2024

matkt commented Jan 6, 2024

RogerHKW commented Jan 6, 2024 •

edited

Loading

RogerHKW commented Jan 6, 2024

matkt commented Jan 6, 2024

auto-call commented Jan 6, 2024

Xexr commented Jan 6, 2024

auto-call commented Jan 6, 2024

matkt commented Jan 7, 2024 •

edited

Loading

TheAviationDoctor commented Jan 7, 2024

yorickdowne commented Jan 7, 2024

Besu shows INVALID block at 18947893 on mainnet #6357

Besu shows INVALID block at 18947893 on mainnet #6357

Comments

yorickdowne commented Jan 6, 2024

Description

Logs (if a bug)

e1ghtball commented Jan 6, 2024

aaronmboyd commented Jan 6, 2024

auto-call commented Jan 6, 2024

Spacesider commented Jan 6, 2024

anthonygraignic commented Jan 6, 2024

aaronmboyd commented Jan 6, 2024

bulikos commented Jan 6, 2024

anthonygraignic commented Jan 6, 2024 • edited Loading

yorickdowne commented Jan 6, 2024 • edited Loading

N0lanV commented Jan 6, 2024

inDane commented Jan 6, 2024 • edited Loading

yorickdowne commented Jan 6, 2024 • edited Loading

inDane commented Jan 6, 2024 • edited Loading

yorickdowne commented Jan 6, 2024

TheAviationDoctor commented Jan 6, 2024 • edited Loading

yorickdowne commented Jan 6, 2024 • edited Loading

inDane commented Jan 6, 2024 • edited Loading

MarekM25 commented Jan 6, 2024

kooshan75 commented Jan 6, 2024

matkt commented Jan 6, 2024

RogerHKW commented Jan 6, 2024 • edited Loading

RogerHKW commented Jan 6, 2024

matkt commented Jan 6, 2024

auto-call commented Jan 6, 2024

Xexr commented Jan 6, 2024

auto-call commented Jan 6, 2024

matkt commented Jan 7, 2024 • edited Loading

TheAviationDoctor commented Jan 7, 2024

yorickdowne commented Jan 7, 2024

Besu shows INVALID block at `18947893` on mainnet #6357

Besu shows INVALID block at `18947893` on mainnet #6357

anthonygraignic commented Jan 6, 2024 •

edited

Loading

yorickdowne commented Jan 6, 2024 •

edited

Loading

inDane commented Jan 6, 2024 •

edited

Loading

yorickdowne commented Jan 6, 2024 •

edited

Loading

inDane commented Jan 6, 2024 •

edited

Loading

TheAviationDoctor commented Jan 6, 2024 •

edited

Loading

yorickdowne commented Jan 6, 2024 •

edited

Loading

inDane commented Jan 6, 2024 •

edited

Loading

RogerHKW commented Jan 6, 2024 •

edited

Loading

matkt commented Jan 7, 2024 •

edited

Loading