IBFT2 fails block validation with BONSAI DB

[matthew1001]

This log was captured from the acceptanceTestBftSoak acceptance test. The test was recently updated to use a combination of FOREST and BONSAI DB nodes - see PR #7496. I believe it passed successfully before the PR was merged, but it seems to be semi-reliably failing now.

 shouldBeStableDuringLongTest | miner3 | 2024-09-17 08:54:27.703+00:00 | EthScheduler-Services-5 (importBlock) | INFO  | MainnetBlockValidator | Invalid block 305 (0x6a23d1450d82a1cf5431db18ddc02f80384c80c7d2a3356eb23726929c4ba9ad): Optional[Unable to process block because parent world state 0x305cc70172d6f210434ae4c650d3ac02629eb2f126ea094f921aa9692fc972f1 is not available]
    shouldBeStableDuringLongTest | miner3 | 2024-09-17 08:54:27.703+00:00 | EthScheduler-Services-5 (importBlock) | INFO  | Pipeline | Unexpected exception in pipeline. Aborting.
    shouldBeStableDuringLongTest | miner3 | Throwable summary: org.hyperledger.besu.ethereum.eth.sync.tasks.exceptions.InvalidBlockException: Failed to import block: Invalid block 305 (0x6a23d1450d82a1cf5431db18ddc02f80384c80c7d2a3356eb23726929c4ba9ad)
    shouldBeStableDuringLongTest | miner3 |     at: org.hyperledger.besu.ethereum.eth.sync.tasks.exceptions.InvalidBlockException.fromInvalidBlock(InvalidBlockException.java:36)
    shouldBeStableDuringLongTest | miner3 | 2024-09-17 08:54:27.703+00:00 | EthScheduler-Services-5 (importBlock) | WARN  | PipelineChainDownloader | Invalid block detected (BREACH_OF_PROTOCOL). Disconnecting from sync target. Failed to import block: Invalid block 305 (0x6a23d1450d82a1cf5431db18ddc02f80384c80c7d2a3356eb23726929c4ba9ad)
    shouldBeStableDuringLongTest | miner3 | 2024-09-17 08:54:27.704+00:00 | EthScheduler-Workers-0 | INFO  | MainnetBlockValidator | Invalid block 305 (0x6a23d1450d82a1cf5431db18ddc02f80384c80c7d2a3356eb23726929c4ba9ad): Optional[Unable to process block because parent world state 0x305cc70172d6f210434ae4c650d3ac02629eb2f126ea094f921aa9692fc972f1 is not available]
    shouldBeStableDuringLongTest | miner3 | 2024-09-17 08:54:27.704+00:00 | EthScheduler-Workers-0 | WARN  | BlockPropagationManager | Failed to import announced block 305 (0x6a23d1450d82a1cf5431db18ddc02f80384c80c7d2a3356eb23726929c4ba9ad)
    shouldBeStableDuringLongTest | miner3 | 2024-09-17 08:54:27.704+00:00 | EthScheduler-Services-5 (importBlock) | WARN  | PipelineChainDownloader | Chain download failed. Restarting after short delay.
    shouldBeStableDuringLongTest | miner3 | java.util.concurrent.CompletionException: org.hyperledger.besu.ethereum.eth.sync.tasks.exceptions.InvalidBlockException: Failed to import block: Invalid block 305 (0x6a23d1450d82a1cf5431db18ddc02f80384c80c7d2a3356eb23726929c4ba9ad)
    shouldBeStableDuringLongTest | miner3 |     at java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368)
    shouldBeStableDuringLongTest | miner3 |     at java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:377)
    shouldBeStableDuringLongTest | miner3 |     at java.base/java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1097)
    shouldBeStableDuringLongTest | miner3 |     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    shouldBeStableDuringLongTest | miner3 |     at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
    shouldBeStableDuringLongTest | miner3 |     at org.hyperledger.besu.services.pipeline.Pipeline.abort(Pipeline.java:220)
    shouldBeStableDuringLongTest | miner3 |     at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:199)
    shouldBeStableDuringLongTest | miner3 |     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
    shouldBeStableDuringLongTest | miner3 |     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    shouldBeStableDuringLongTest | miner3 |     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    shouldBeStableDuringLongTest | miner3 |     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    shouldBeStableDuringLongTest | miner3 |     at java.base/java.lang.Thread.run(Thread.java:1595)
    shouldBeStableDuringLongTest | miner3 | Caused by: org.hyperledger.besu.ethereum.eth.sync.tasks.exceptions.InvalidBlockException: Failed to import block: Invalid block 305 (0x6a23d1450d82a1cf5431db18ddc02f80384c80c7d2a3356eb23726929c4ba9ad)
    shouldBeStableDuringLongTest | miner3 |     at org.hyperledger.besu.ethereum.eth.sync.tasks.exceptions.InvalidBlockException.fromInvalidBlock(InvalidBlockException.java:36)
    shouldBeStableDuringLongTest | miner3 |     at org.hyperledger.besu.ethereum.eth.sync.tasks.exceptions.InvalidBlockException.fromInvalidBlock(InvalidBlockException.java:30)
    shouldBeStableDuringLongTest | miner3 |     at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:65)
    shouldBeStableDuringLongTest | miner3 |     at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:32)
    shouldBeStableDuringLongTest | miner3 |     at org.hyperledger.besu.services.pipeline.CompleterStage.run(CompleterStage.java:37)
    shouldBeStableDuringLongTest | miner3 |     at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:183)
    shouldBeStableDuringLongTest | miner3 |     ... 5 more
    shouldBeStableDuringLongTest | miner4 | 2024-09-17 08:54:28.008+00:00 | BftProcessorExecutor-IBFT-0 | INFO  | IbftBesuControllerBuilder | Imported #309 / 0 tx / 0 pending / 0 (0.0%) gas / (0xbe5a2e37c51131f371ca7d9fde5e6342075183fb6b0c69d666cdd54a2a0ffc6f)

Steps to Reproduce (Bug)

Run gradlew acceptanceTestBftSoak

Expected behavior: The test passes

Actual behavior: The test fails on the IBFT2 step