Improve processRewardsAndPenalties

[twoeths]

Problem description

The processRewardsAndPenalties time is inconsistent, on most of test mainnet nodes it's all good

However on a lot of other mainnet nodes it's so different

Note that all of these nodes are with v1.13.0 and are all deployed at the same. Having a look at profile it seems the differences come from how much gc run mainly due to packedRootsBytesToLeafNodes function

Solution description

1. Reuse the whole balances nodes branch to avoid gc, this helps avoid recreating the whole balances branch per epoch transition

• Set new h${i} values to existing leaf nodes
• For all branch nodes, set h0 to null to trigger a rehash later https://github.com/ChainSafe/ssz/blob/93ab30e4fc2928fb808b4c76ee015631b8770126/packages/persistent-merkle-tree/src/node.ts#L74

Update: not possible to do this as branch nodes are shared with other states, should not mutate them.

2. Right now we serialize the whole balances before recreating the branch. We can bypass it by setting each balances[i] to LeafNodes[Math.floor(i/4)], h[(i % 4) * 2] and h[(i % 4) * 2 + 1] directly

Additional context

No response

[twoeths]

idea 1 is not possible for now because LeftNodes are shared across states, can revisit once we're done with n-historical state #5968 work
given a deleted check point state, can balances tree be reused for the next epoch transition?

[twoeths]

epoch transition time is getting worse and may cause missed attestations at slot 0 as it could be up to 6s
Just take a look at gossipsub metric, bandwidth is in subscribe-all-subnets test mainnet node is even bigger than that in production mainnet node.

vs our test node subscribing to all subnets

The difference I see is in publish flow where production mainnet node has to serialize/prefix data while test mainnet node does not have to do that, this may cause gc to run more in processRewardsAndPenalties function (see the profile above)

we should prioritize gossipsub publish in batch issue which only require to serialize once per message (instead of per peer) ChainSafe/js-libp2p-gossipsub#344 cc @wemeetagain @philknows

[twoeths]

in progress PR ChainSafe/js-libp2p-gossipsub#480

[twoeths]

PR was merged, need to update gossipsub option when new gossipsub is released

[twoeths]

processRewardsAndPenalties is great since we merge ssz v0.15.1 ChainSafe/ssz#352

this is what I see on a mainnet node as of Jul 2024