Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: fix archiver's cleanup of non checkpoint roots #5439

Merged
merged 2 commits into from
May 1, 2023
Merged

fix: fix archiver's cleanup of non checkpoint roots #5439

merged 2 commits into from
May 1, 2023

Conversation

g11tech
Copy link
Contributor

@g11tech g11tech commented Apr 30, 2023 •
edited
Loading

Archiver's getNonCheckpointBlocks assumed ancestor to child order of blocks input (increasing slot) and then iterated in reverse to find the non checkpoint slots from new finalized checkpoint's ancestors obtained via

  const finalizedCanonicalBlocks = forkChoice.getAllAncestorBlocks(finalizedCheckpoint.rootHex);

However getAllAncestorBlocks returns data in child to ancestor i.e. decreasing slot. This leads to incorrect calculation of the non checkpoint roots (which included actual checkpoint roots which are not perfectly aligned with epoch boundary) which lead to errors like this

As checkpointed data gets incorrectly pruned out for these non perfectly epoch boundary aligned checkpoints.

Closes #4183, Closes #5129

@g11tech g11tech requested a review from a team as a code owner April 30, 2023 15:10
@g11tech g11tech enabled auto-merge (squash) April 30, 2023 15:10
@g11tech g11tech changed the title fix: Fix archiver's cleanup of non checkpoint roots fix: fix archiver's cleanup of non checkpoint roots Apr 30, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Apr 30, 2023 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 801763a Previous: 2fbf178 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 801.33 us/op 505.54 us/op 1.59
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 43.627 us/op 46.593 us/op 0.94
BLS verify - blst-native 1.1670 ms/op 1.2273 ms/op 0.95
BLS verifyMultipleSignatures 3 - blst-native 2.3722 ms/op 2.4959 ms/op 0.95
BLS verifyMultipleSignatures 8 - blst-native 5.0874 ms/op 5.3762 ms/op 0.95
BLS verifyMultipleSignatures 32 - blst-native 18.442 ms/op 19.383 ms/op 0.95
BLS aggregatePubkeys 32 - blst-native 24.571 us/op 25.814 us/op 0.95
BLS aggregatePubkeys 128 - blst-native 96.248 us/op 99.982 us/op 0.96
getAttestationsForBlock 51.571 ms/op 53.235 ms/op 0.97
isKnown best case - 1 super set check 249.00 ns/op 251.00 ns/op 0.99
isKnown normal case - 2 super set checks 242.00 ns/op 243.00 ns/op 1.00
isKnown worse case - 16 super set checks 241.00 ns/op 243.00 ns/op 0.99
CheckpointStateCache - add get delete 4.7650 us/op 4.9180 us/op 0.97
validate gossip signedAggregateAndProof - struct 2.6366 ms/op 2.7750 ms/op 0.95
validate gossip attestation - struct 1.2589 ms/op 1.3314 ms/op 0.95
pickEth1Vote - no votes 1.1591 ms/op 1.2078 ms/op 0.96
pickEth1Vote - max votes 10.450 ms/op 10.758 ms/op 0.97
pickEth1Vote - Eth1Data hashTreeRoot value x2048 8.2953 ms/op 8.8227 ms/op 0.94
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 14.045 ms/op 15.265 ms/op 0.92
pickEth1Vote - Eth1Data fastSerialize value x2048 599.99 us/op 673.27 us/op 0.89
pickEth1Vote - Eth1Data fastSerialize tree x2048 6.2234 ms/op 7.9822 ms/op 0.78
bytes32 toHexString 521.00 ns/op 489.00 ns/op 1.07
bytes32 Buffer.toString(hex) 408.00 ns/op 355.00 ns/op 1.15
bytes32 Buffer.toString(hex) from Uint8Array 547.00 ns/op 556.00 ns/op 0.98
bytes32 Buffer.toString(hex) + 0x 340.00 ns/op 359.00 ns/op 0.95
Object access 1 prop 0.17600 ns/op 0.15900 ns/op 1.11
Map access 1 prop 0.16600 ns/op 0.16900 ns/op 0.98
Object get x1000 6.5200 ns/op 6.5060 ns/op 1.00
Map get x1000 0.62700 ns/op 0.62300 ns/op 1.01
Object set x1000 52.009 ns/op 54.171 ns/op 0.96
Map set x1000 43.944 ns/op 44.249 ns/op 0.99
Return object 10000 times 0.22970 ns/op 0.23720 ns/op 0.97
Throw Error 10000 times 4.0433 us/op 4.1253 us/op 0.98
fastMsgIdFn sha256 / 200 bytes 3.3870 us/op 3.3610 us/op 1.01
fastMsgIdFn h32 xxhash / 200 bytes 263.00 ns/op 277.00 ns/op 0.95
fastMsgIdFn h64 xxhash / 200 bytes 377.00 ns/op 396.00 ns/op 0.95
fastMsgIdFn sha256 / 1000 bytes 11.193 us/op 11.548 us/op 0.97
fastMsgIdFn h32 xxhash / 1000 bytes 383.00 ns/op 403.00 ns/op 0.95
fastMsgIdFn h64 xxhash / 1000 bytes 441.00 ns/op 480.00 ns/op 0.92
fastMsgIdFn sha256 / 10000 bytes 99.679 us/op 101.90 us/op 0.98
fastMsgIdFn h32 xxhash / 10000 bytes 1.8590 us/op 1.9610 us/op 0.95
fastMsgIdFn h64 xxhash / 10000 bytes 1.3140 us/op 1.4230 us/op 0.92
enrSubnets - fastDeserialize 64 bits 1.2380 us/op 1.3060 us/op 0.95
enrSubnets - ssz BitVector 64 bits 468.00 ns/op 479.00 ns/op 0.98
enrSubnets - fastDeserialize 4 bits 163.00 ns/op 170.00 ns/op 0.96
enrSubnets - ssz BitVector 4 bits 471.00 ns/op 497.00 ns/op 0.95
prioritizePeers score -10:0 att 32-0.1 sync 2-0 99.312 us/op 106.13 us/op 0.94
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 130.92 us/op 142.60 us/op 0.92
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 165.41 us/op 168.70 us/op 0.98
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 298.32 us/op 303.86 us/op 0.98
prioritizePeers score 0:0 att 64-1 sync 4-1 358.96 us/op 374.06 us/op 0.96
array of 16000 items push then shift 1.6027 us/op 1.5866 us/op 1.01
LinkedList of 16000 items push then shift 8.6870 ns/op 8.8680 ns/op 0.98
array of 16000 items push then pop 78.713 ns/op 86.268 ns/op 0.91
LinkedList of 16000 items push then pop 8.4370 ns/op 8.3590 ns/op 1.01
array of 24000 items push then shift 2.3363 us/op 2.3805 us/op 0.98
LinkedList of 24000 items push then shift 8.4370 ns/op 8.8170 ns/op 0.96
array of 24000 items push then pop 72.366 ns/op 74.273 ns/op 0.97
LinkedList of 24000 items push then pop 8.3350 ns/op 8.3900 ns/op 0.99
intersect bitArray bitLen 8 13.052 ns/op 13.187 ns/op 0.99
intersect array and set length 8 76.018 ns/op 74.780 ns/op 1.02
intersect bitArray bitLen 128 43.314 ns/op 43.549 ns/op 0.99
intersect array and set length 128 1.0323 us/op 998.48 ns/op 1.03
Buffer.concat 32 items 2.8480 us/op 2.4840 us/op 1.15
Uint8Array.set 32 items 2.3960 us/op 2.0990 us/op 1.14
pass gossip attestations to forkchoice per slot 2.6732 ms/op 2.7416 ms/op 0.98
computeDeltas 2.7795 ms/op 3.3479 ms/op 0.83
computeProposerBoostScoreFromBalances 1.7452 ms/op 1.8413 ms/op 0.95
altair processAttestation - 250000 vs - 7PWei normalcase 2.0299 ms/op 2.1482 ms/op 0.94
altair processAttestation - 250000 vs - 7PWei worstcase 3.2674 ms/op 3.3068 ms/op 0.99
altair processAttestation - setStatus - 1/6 committees join 134.20 us/op 139.47 us/op 0.96
altair processAttestation - setStatus - 1/3 committees join 257.06 us/op 270.49 us/op 0.95
altair processAttestation - setStatus - 1/2 committees join 356.41 us/op 364.75 us/op 0.98
altair processAttestation - setStatus - 2/3 committees join 437.48 us/op 462.56 us/op 0.95
altair processAttestation - setStatus - 4/5 committees join 625.95 us/op 649.19 us/op 0.96
altair processAttestation - setStatus - 100% committees join 733.47 us/op 729.59 us/op 1.01
altair processBlock - 250000 vs - 7PWei normalcase 17.904 ms/op 15.984 ms/op 1.12
altair processBlock - 250000 vs - 7PWei normalcase hashState 27.341 ms/op 26.783 ms/op 1.02
altair processBlock - 250000 vs - 7PWei worstcase 45.717 ms/op 46.406 ms/op 0.99
altair processBlock - 250000 vs - 7PWei worstcase hashState 66.661 ms/op 68.634 ms/op 0.97
phase0 processBlock - 250000 vs - 7PWei normalcase 1.9415 ms/op 2.0562 ms/op 0.94
phase0 processBlock - 250000 vs - 7PWei worstcase 27.803 ms/op 27.847 ms/op 1.00
altair processEth1Data - 250000 vs - 7PWei normalcase 444.71 us/op 442.02 us/op 1.01
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 6.4650 us/op 6.7640 us/op 0.96
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 19.523 us/op 19.220 us/op 1.02
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 8.0970 us/op 8.1070 us/op 1.00
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 6.0990 us/op 6.2160 us/op 0.98
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 74.069 us/op 73.879 us/op 1.00
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 642.37 us/op 610.25 us/op 1.05
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 908.62 us/op 903.01 us/op 1.01
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 863.86 us/op 894.76 us/op 0.97
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.1649 ms/op 2.2001 ms/op 0.98
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.4440 ms/op 1.4588 ms/op 0.99
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 3.8626 ms/op 3.8785 ms/op 1.00
Tree 40 250000 create 286.14 ms/op 288.25 ms/op 0.99
Tree 40 250000 get(125000) 173.68 ns/op 174.65 ns/op 0.99
Tree 40 250000 set(125000) 856.40 ns/op 862.14 ns/op 0.99
Tree 40 250000 toArray() 17.662 ms/op 17.835 ms/op 0.99
Tree 40 250000 iterate all - toArray() + loop 17.667 ms/op 17.679 ms/op 1.00
Tree 40 250000 iterate all - get(i) 66.809 ms/op 65.946 ms/op 1.01
MutableVector 250000 create 9.9924 ms/op 9.9927 ms/op 1.00
MutableVector 250000 get(125000) 6.1210 ns/op 6.2920 ns/op 0.97
MutableVector 250000 set(125000) 255.97 ns/op 257.96 ns/op 0.99
MutableVector 250000 toArray() 2.7922 ms/op 2.7631 ms/op 1.01
MutableVector 250000 iterate all - toArray() + loop 2.8509 ms/op 3.0520 ms/op 0.93
MutableVector 250000 iterate all - get(i) 1.4721 ms/op 1.4954 ms/op 0.98
Array 250000 create 2.6366 ms/op 2.4647 ms/op 1.07
Array 250000 clone - spread 1.0420 ms/op 1.0899 ms/op 0.96
Array 250000 get(125000) 0.49100 ns/op 0.53200 ns/op 0.92
Array 250000 set(125000) 0.57300 ns/op 0.61000 ns/op 0.94
Array 250000 iterate all - loop 102.03 us/op 81.164 us/op 1.26
effectiveBalanceIncrements clone Uint8Array 300000 22.275 us/op 25.428 us/op 0.88
effectiveBalanceIncrements clone MutableVector 300000 311.00 ns/op 332.00 ns/op 0.94
effectiveBalanceIncrements rw all Uint8Array 300000 159.72 us/op 168.45 us/op 0.95
effectiveBalanceIncrements rw all MutableVector 300000 73.664 ms/op 80.141 ms/op 0.92
phase0 afterProcessEpoch - 250000 vs - 7PWei 107.25 ms/op 115.75 ms/op 0.93
phase0 beforeProcessEpoch - 250000 vs - 7PWei 39.429 ms/op 41.781 ms/op 0.94
altair processEpoch - mainnet_e81889 298.38 ms/op 327.14 ms/op 0.91
mainnet_e81889 - altair beforeProcessEpoch 63.911 ms/op 66.222 ms/op 0.97
mainnet_e81889 - altair processJustificationAndFinalization 17.053 us/op 17.694 us/op 0.96
mainnet_e81889 - altair processInactivityUpdates 5.4247 ms/op 5.3510 ms/op 1.01
mainnet_e81889 - altair processRewardsAndPenalties 51.342 ms/op 64.512 ms/op 0.80
mainnet_e81889 - altair processRegistryUpdates 2.7670 us/op 2.6900 us/op 1.03
mainnet_e81889 - altair processSlashings 515.00 ns/op 435.00 ns/op 1.18
mainnet_e81889 - altair processEth1DataReset 548.00 ns/op 467.00 ns/op 1.17
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.1931 ms/op 1.2497 ms/op 0.95
mainnet_e81889 - altair processSlashingsReset 5.0970 us/op 5.1750 us/op 0.98
mainnet_e81889 - altair processRandaoMixesReset 4.8450 us/op 4.3590 us/op 1.11
mainnet_e81889 - altair processHistoricalRootsUpdate 604.00 ns/op 551.00 ns/op 1.10
mainnet_e81889 - altair processParticipationFlagUpdates 2.5260 us/op 2.4600 us/op 1.03
mainnet_e81889 - altair processSyncCommitteeUpdates 564.00 ns/op 459.00 ns/op 1.23
mainnet_e81889 - altair afterProcessEpoch 115.66 ms/op 118.05 ms/op 0.98
phase0 processEpoch - mainnet_e58758 305.54 ms/op 324.45 ms/op 0.94
mainnet_e58758 - phase0 beforeProcessEpoch 119.70 ms/op 121.78 ms/op 0.98
mainnet_e58758 - phase0 processJustificationAndFinalization 16.028 us/op 17.256 us/op 0.93
mainnet_e58758 - phase0 processRewardsAndPenalties 55.919 ms/op 57.192 ms/op 0.98
mainnet_e58758 - phase0 processRegistryUpdates 9.0940 us/op 8.9660 us/op 1.01
mainnet_e58758 - phase0 processSlashings 456.00 ns/op 517.00 ns/op 0.88
mainnet_e58758 - phase0 processEth1DataReset 465.00 ns/op 519.00 ns/op 0.90
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 939.68 us/op 1.0598 ms/op 0.89
mainnet_e58758 - phase0 processSlashingsReset 2.7700 us/op 3.2620 us/op 0.85
mainnet_e58758 - phase0 processRandaoMixesReset 4.2950 us/op 6.3000 us/op 0.68
mainnet_e58758 - phase0 processHistoricalRootsUpdate 596.00 ns/op 853.00 ns/op 0.70
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.9720 us/op 5.0590 us/op 0.79
mainnet_e58758 - phase0 afterProcessEpoch 94.801 ms/op 96.933 ms/op 0.98
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2309 ms/op 1.2387 ms/op 0.99
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.5512 ms/op 1.6224 ms/op 0.96
altair processInactivityUpdates - 250000 normalcase 19.659 ms/op 22.771 ms/op 0.86
altair processInactivityUpdates - 250000 worstcase 22.601 ms/op 27.318 ms/op 0.83
phase0 processRegistryUpdates - 250000 normalcase 5.9900 us/op 6.9710 us/op 0.86
phase0 processRegistryUpdates - 250000 badcase_full_deposits 269.02 us/op 270.46 us/op 0.99
phase0 processRegistryUpdates - 250000 worstcase 0.5 116.20 ms/op 110.48 ms/op 1.05
altair processRewardsAndPenalties - 250000 normalcase 52.598 ms/op 64.784 ms/op 0.81
altair processRewardsAndPenalties - 250000 worstcase 52.348 ms/op 70.075 ms/op 0.75
phase0 getAttestationDeltas - 250000 normalcase 6.4179 ms/op 6.6577 ms/op 0.96
phase0 getAttestationDeltas - 250000 worstcase 6.5352 ms/op 6.5876 ms/op 0.99
phase0 processSlashings - 250000 worstcase 3.4674 ms/op 3.3859 ms/op 1.02
altair processSyncCommitteeUpdates - 250000 173.24 ms/op 177.23 ms/op 0.98
BeaconState.hashTreeRoot - No change 255.00 ns/op 369.00 ns/op 0.69
BeaconState.hashTreeRoot - 1 full validator 49.146 us/op 50.937 us/op 0.96
BeaconState.hashTreeRoot - 32 full validator 483.90 us/op 549.17 us/op 0.88
BeaconState.hashTreeRoot - 512 full validator 4.8418 ms/op 5.3780 ms/op 0.90
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 63.709 us/op 60.712 us/op 1.05
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 908.82 us/op 881.17 us/op 1.03
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 11.368 ms/op 11.324 ms/op 1.00
BeaconState.hashTreeRoot - 1 balances 49.756 us/op 52.552 us/op 0.95
BeaconState.hashTreeRoot - 32 balances 428.62 us/op 450.77 us/op 0.95
BeaconState.hashTreeRoot - 512 balances 4.1570 ms/op 4.4361 ms/op 0.94
BeaconState.hashTreeRoot - 250000 balances 72.644 ms/op 70.658 ms/op 1.03
aggregationBits - 2048 els - zipIndexesInBitList 16.445 us/op 15.547 us/op 1.06
regular array get 100000 times 32.952 us/op 34.737 us/op 0.95
wrappedArray get 100000 times 32.904 us/op 32.758 us/op 1.00
arrayWithProxy get 100000 times 16.111 ms/op 15.679 ms/op 1.03
ssz.Root.equals 543.00 ns/op 545.00 ns/op 1.00
byteArrayEquals 550.00 ns/op 545.00 ns/op 1.01
shuffle list - 16384 els 6.8076 ms/op 6.7732 ms/op 1.01
shuffle list - 250000 els 99.854 ms/op 99.524 ms/op 1.00
processSlot - 1 slots 8.8230 us/op 8.8300 us/op 1.00
processSlot - 32 slots 1.4138 ms/op 1.3800 ms/op 1.02
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 33.568 ms/op 37.139 ms/op 0.90
getCommitteeAssignments - req 1 vs - 250000 vc 2.8847 ms/op 3.0980 ms/op 0.93
getCommitteeAssignments - req 100 vs - 250000 vc 4.0841 ms/op 4.3272 ms/op 0.94
getCommitteeAssignments - req 1000 vs - 250000 vc 4.4023 ms/op 4.5509 ms/op 0.97
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.4300 ns/op 4.7500 ns/op 0.93
state getBlockRootAtSlot - 250000 vs - 7PWei 827.77 ns/op 583.06 ns/op 1.42
computeProposers - vc 250000 10.341 ms/op 11.096 ms/op 0.93
computeEpochShuffling - vc 250000 105.90 ms/op 102.74 ms/op 1.03
getNextSyncCommittee - vc 250000 171.72 ms/op 179.00 ms/op 0.96
computeSigningRoot for AttestationData 12.935 us/op 13.860 us/op 0.93
hash AttestationData serialized data then Buffer.toString(base64) 2.3013 us/op 2.4531 us/op 0.94
toHexString serialized data 1.0677 us/op 1.0729 us/op 1.00
Buffer.toString(base64) 304.48 ns/op 326.00 ns/op 0.93

by benchmarkbot/action

@g11tech g11tech merged commit 229b77b into unstable May 1, 2023
@g11tech g11tech deleted the g11tech/fix-non-checkpoint branch May 1, 2023 09:24
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.8.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@twoeths twoeths twoeths approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

light client get_bootstrap endpoint is failing Sync committee witness becomes unavailable after a while
3 participants
@g11tech @wemeetagain @twoeths