BAD Block in private QBFT chain

[cryptohazard]

Description

We ran a private chain of 4 nodes, with QBFT and all the milestones enabled. We added Shanghai once we migrated to 24.3.0.
One node, which is our JSON-RPC node fell out of sync. It showed a BAD BLOCK error and from there we couldn't make it join the network any more (we ran into #6760 then).

My question is: do you see a way a heavy (relatively) load on the JSON-RPC could end up with bad blocks?
Looking through other issues with bad blocks here, I'm pretty unsure how we caused it.

[MASDXI]

can you provide your configuration, did you use bonsai data structure?

[macfarla]

@cryptohazard is this the same as your issue #6680

[cryptohazard]

@cryptohazard is this the same as your issue #6680

No it doesn't seem so.

I will run some tests and share the errors I'm getting (annoyingly, I can't copy/paste everything due to work policy so I have to adapt/clean the logs, config...)

[cryptohazard]

can you provide your configuration, did you use bonsai data structure?

Forest, since it was the old default and our chain was already up before you changed the default to Bonsai.

[cryptohazard]

the error:

WARN withdrawlsRoot must not be null when Withdrawals are activate
INFO Invalid block

I can also say that updating to the current version 24.5.1after activating Shanghai on a previous version (24.3.0) doesn't fix the issue.

[macfarla]

@matthew1001
WARN withdrawlsRoot must not be null when Withdrawals are activate
INFO Invalid block
on QBFT, shanghai, 24.5.1 - any insights on this one?

[matthew1001]

Yeah I'm just having a look now. There was a fix in 24.5.1 that is related to this area (#6765) but I think it may not be complete enough. Going to look at this issue today.

[matthew1001]

I've raised a PR which resolves the issue (#7115). Tested by syncing 24.5.1 + fix against an existing 24.3.0 shanghai chain. Without the fix I hit the issue described. With the fix the node syncs and then continues mining blocks.

@cryptohazard do you have a Besu build environment? If you do and are able to test out https://github.com/matthew1001/besu/tree/qbft-ignore-withdrawals please do.

@jflo @non-fungible-nelson it could be difficult for users to have to wait long for this fix. Not sure what the options are for getting it into a release in the near future.

[non-fungible-nelson]

we were not planning on having as long of a release cycle for the next release. We would like to restart the two week cadence.

[cryptohazard]

@cryptohazard do you have a Besu build environment? If you do and are able to test out https://github.com/matthew1001/besu/tree/qbft-ignore-withdrawals please do.

we don't yet but we are looking into setting one up. So that we can test the updates faster from now on.

Thanks for the work on that issue. This solves a lot of headaches we were having.