Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: prune checkpoint states at syncing time #7241

Merged
merged 4 commits into from
Nov 27, 2024
Merged

fix: prune checkpoint states at syncing time #7241

merged 4 commits into from
Nov 27, 2024

Conversation

twoeths
Copy link
Contributor

@twoeths twoeths commented Nov 26, 2024 •
edited
Loading

Motivation

Right now we only prune states at synced time. When not is restarted and network faces big unfinality period, it went OOM. See #7240

Description

  • always prune checkpoint state caches either when node is syncing or synced
  • remove unused variables

Closes #7240

Test

  • was able to sync nft-devnet-0 from genesis (see nogroup 1). It took 1h15m to sync from slot 0 to ~25k. Note that my node does not include an EL

  • state is consistent during syncing time (4 to 7 states)

  • heap is consistent during syncing time (on main thread, it's from 1.5GB to 2.5GB)

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 26, 2024 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 5080dbf Previous: 18f4218 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.7031 ms/op 1.9524 ms/op 1.38
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 72.211 us/op 42.152 us/op 1.71
BLS verify - blst 980.06 us/op 881.18 us/op 1.11
BLS verifyMultipleSignatures 3 - blst 1.3831 ms/op 1.7449 ms/op 0.79
BLS verifyMultipleSignatures 8 - blst 2.2321 ms/op 2.0996 ms/op 1.06
BLS verifyMultipleSignatures 32 - blst 6.3163 ms/op 4.4837 ms/op 1.41
BLS verifyMultipleSignatures 64 - blst 11.708 ms/op 9.0045 ms/op 1.30
BLS verifyMultipleSignatures 128 - blst 18.924 ms/op 16.094 ms/op 1.18
BLS deserializing 10000 signatures 734.85 ms/op 631.17 ms/op 1.16
BLS deserializing 100000 signatures 7.4864 s/op 6.3028 s/op 1.19
BLS verifyMultipleSignatures - same message - 3 - blst 1.0614 ms/op 1.0428 ms/op 1.02
BLS verifyMultipleSignatures - same message - 8 - blst 1.2904 ms/op 1.1274 ms/op 1.14
BLS verifyMultipleSignatures - same message - 32 - blst 2.0298 ms/op 1.6910 ms/op 1.20
BLS verifyMultipleSignatures - same message - 64 - blst 3.0601 ms/op 2.5148 ms/op 1.22
BLS verifyMultipleSignatures - same message - 128 - blst 4.7990 ms/op 4.1142 ms/op 1.17
BLS aggregatePubkeys 32 - blst 21.288 us/op 18.158 us/op 1.17
BLS aggregatePubkeys 128 - blst 74.795 us/op 64.005 us/op 1.17
notSeenSlots=1 numMissedVotes=1 numBadVotes=10 103.40 ms/op 83.374 ms/op 1.24
notSeenSlots=1 numMissedVotes=0 numBadVotes=4 64.137 ms/op 65.261 ms/op 0.98
notSeenSlots=2 numMissedVotes=1 numBadVotes=10 56.019 ms/op 41.730 ms/op 1.34
getSlashingsAndExits - default max 99.099 us/op 74.184 us/op 1.34
getSlashingsAndExits - 2k 332.78 us/op 336.02 us/op 0.99
proposeBlockBody type=full, size=empty 6.8021 ms/op 6.1156 ms/op 1.11
isKnown best case - 1 super set check 391.00 ns/op 599.00 ns/op 0.65
isKnown normal case - 2 super set checks 444.00 ns/op 561.00 ns/op 0.79
isKnown worse case - 16 super set checks 461.00 ns/op 637.00 ns/op 0.72
InMemoryCheckpointStateCache - add get delete 3.1380 us/op 2.7810 us/op 1.13
validate api signedAggregateAndProof - struct 1.5125 ms/op 1.8835 ms/op 0.80
validate gossip signedAggregateAndProof - struct 1.6913 ms/op 1.8017 ms/op 0.94
batch validate gossip attestation - vc 640000 - chunk 32 141.98 us/op 130.01 us/op 1.09
batch validate gossip attestation - vc 640000 - chunk 64 115.08 us/op 113.79 us/op 1.01
batch validate gossip attestation - vc 640000 - chunk 128 106.74 us/op 115.57 us/op 0.92
batch validate gossip attestation - vc 640000 - chunk 256 106.69 us/op 106.34 us/op 1.00
pickEth1Vote - no votes 1.1282 ms/op 864.90 us/op 1.30
pickEth1Vote - max votes 6.7013 ms/op 5.7705 ms/op 1.16
pickEth1Vote - Eth1Data hashTreeRoot value x2048 13.987 ms/op 11.654 ms/op 1.20
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 22.191 ms/op 18.539 ms/op 1.20
pickEth1Vote - Eth1Data fastSerialize value x2048 513.63 us/op 370.63 us/op 1.39
pickEth1Vote - Eth1Data fastSerialize tree x2048 3.9084 ms/op 3.5914 ms/op 1.09
bytes32 toHexString 471.00 ns/op 632.00 ns/op 0.75
bytes32 Buffer.toString(hex) 255.00 ns/op 468.00 ns/op 0.54
bytes32 Buffer.toString(hex) from Uint8Array 409.00 ns/op 558.00 ns/op 0.73
bytes32 Buffer.toString(hex) + 0x 261.00 ns/op 480.00 ns/op 0.54
Object access 1 prop 0.15300 ns/op 0.33600 ns/op 0.46
Map access 1 prop 0.14500 ns/op 0.32300 ns/op 0.45
Object get x1000 6.0660 ns/op 5.6860 ns/op 1.07
Map get x1000 6.6030 ns/op 6.6140 ns/op 1.00
Object set x1000 41.606 ns/op 29.109 ns/op 1.43
Map set x1000 25.715 ns/op 20.276 ns/op 1.27
Return object 10000 times 0.30020 ns/op 0.31150 ns/op 0.96
Throw Error 10000 times 3.4284 us/op 2.9703 us/op 1.15
toHex 165.40 ns/op 141.10 ns/op 1.17
Buffer.from 157.92 ns/op 120.56 ns/op 1.31
shared Buffer 106.83 ns/op 79.098 ns/op 1.35
fastMsgIdFn sha256 / 200 bytes 2.3240 us/op 2.1850 us/op 1.06
fastMsgIdFn h32 xxhash / 200 bytes 260.00 ns/op 521.00 ns/op 0.50
fastMsgIdFn h64 xxhash / 200 bytes 268.00 ns/op 484.00 ns/op 0.55
fastMsgIdFn sha256 / 1000 bytes 7.3710 us/op 5.9110 us/op 1.25
fastMsgIdFn h32 xxhash / 1000 bytes 384.00 ns/op 651.00 ns/op 0.59
fastMsgIdFn h64 xxhash / 1000 bytes 350.00 ns/op 581.00 ns/op 0.60
fastMsgIdFn sha256 / 10000 bytes 65.467 us/op 50.992 us/op 1.28
fastMsgIdFn h32 xxhash / 10000 bytes 1.9090 us/op 2.1020 us/op 0.91
fastMsgIdFn h64 xxhash / 10000 bytes 1.2240 us/op 1.4220 us/op 0.86
send data - 1000 256B messages 12.714 ms/op 13.138 ms/op 0.97
send data - 1000 512B messages 17.870 ms/op 20.279 ms/op 0.88
send data - 1000 1024B messages 28.903 ms/op 30.035 ms/op 0.96
send data - 1000 1200B messages 32.449 ms/op 30.757 ms/op 1.06
send data - 1000 2048B messages 36.416 ms/op 31.605 ms/op 1.15
send data - 1000 4096B messages 35.683 ms/op 32.852 ms/op 1.09
send data - 1000 16384B messages 79.033 ms/op 71.452 ms/op 1.11
send data - 1000 65536B messages 221.73 ms/op 341.02 ms/op 0.65
enrSubnets - fastDeserialize 64 bits 1.3400 us/op 1.2060 us/op 1.11
enrSubnets - ssz BitVector 64 bits 453.00 ns/op 654.00 ns/op 0.69
enrSubnets - fastDeserialize 4 bits 190.00 ns/op 426.00 ns/op 0.45
enrSubnets - ssz BitVector 4 bits 459.00 ns/op 650.00 ns/op 0.71
prioritizePeers score -10:0 att 32-0.1 sync 2-0 160.77 us/op 183.11 us/op 0.88
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 183.28 us/op 148.52 us/op 1.23
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 327.63 us/op 270.86 us/op 1.21
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 523.42 us/op 424.28 us/op 1.23
prioritizePeers score 0:0 att 64-1 sync 4-1 696.40 us/op 721.93 us/op 0.96
array of 16000 items push then shift 1.7207 us/op 1.3118 us/op 1.31
LinkedList of 16000 items push then shift 8.5140 ns/op 8.3460 ns/op 1.02
array of 16000 items push then pop 133.45 ns/op 123.44 ns/op 1.08
LinkedList of 16000 items push then pop 7.5790 ns/op 6.9130 ns/op 1.10
array of 24000 items push then shift 2.5981 us/op 1.9320 us/op 1.34
LinkedList of 24000 items push then shift 8.2810 ns/op 6.6210 ns/op 1.25
array of 24000 items push then pop 175.83 ns/op 118.67 ns/op 1.48
LinkedList of 24000 items push then pop 8.4210 ns/op 6.3020 ns/op 1.34
intersect bitArray bitLen 8 6.5460 ns/op 5.7850 ns/op 1.13
intersect array and set length 8 60.877 ns/op 38.262 ns/op 1.59
intersect bitArray bitLen 128 32.380 ns/op 26.919 ns/op 1.20
intersect array and set length 128 897.61 ns/op 675.21 ns/op 1.33
bitArray.getTrueBitIndexes() bitLen 128 1.8710 us/op 2.3200 us/op 0.81
bitArray.getTrueBitIndexes() bitLen 248 2.9170 us/op 3.5150 us/op 0.83
bitArray.getTrueBitIndexes() bitLen 512 5.7330 us/op 7.1310 us/op 0.80
Buffer.concat 32 items 1.0380 us/op 1.0560 us/op 0.98
Uint8Array.set 32 items 1.7450 us/op 1.4470 us/op 1.21
Buffer.copy 1.8890 us/op 2.3660 us/op 0.80
Uint8Array.set - with subarray 2.6520 us/op 2.8830 us/op 0.92
Uint8Array.set - without subarray 1.5190 us/op 2.2970 us/op 0.66
getUint32 - dataview 268.00 ns/op 460.00 ns/op 0.58
getUint32 - manual 195.00 ns/op 354.00 ns/op 0.55
Set add up to 64 items then delete first 2.6738 us/op 1.7920 us/op 1.49
OrderedSet add up to 64 items then delete first 3.7194 us/op 2.8668 us/op 1.30
Set add up to 64 items then delete last 2.9522 us/op 2.1009 us/op 1.41
OrderedSet add up to 64 items then delete last 4.4117 us/op 3.0898 us/op 1.43
Set add up to 64 items then delete middle 2.7215 us/op 2.0410 us/op 1.33
OrderedSet add up to 64 items then delete middle 6.2493 us/op 4.5580 us/op 1.37
Set add up to 128 items then delete first 5.8994 us/op 4.0353 us/op 1.46
OrderedSet add up to 128 items then delete first 9.3270 us/op 6.3447 us/op 1.47
Set add up to 128 items then delete last 5.7332 us/op 3.9750 us/op 1.44
OrderedSet add up to 128 items then delete last 8.6008 us/op 5.9651 us/op 1.44
Set add up to 128 items then delete middle 5.4387 us/op 3.8608 us/op 1.41
OrderedSet add up to 128 items then delete middle 15.038 us/op 12.052 us/op 1.25
Set add up to 256 items then delete first 11.709 us/op 7.9101 us/op 1.48
OrderedSet add up to 256 items then delete first 17.945 us/op 12.341 us/op 1.45
Set add up to 256 items then delete last 11.591 us/op 7.6223 us/op 1.52
OrderedSet add up to 256 items then delete last 18.870 us/op 11.674 us/op 1.62
Set add up to 256 items then delete middle 12.238 us/op 7.5639 us/op 1.62
OrderedSet add up to 256 items then delete middle 45.742 us/op 34.724 us/op 1.32
transfer serialized Status (84 B) 1.5020 us/op 1.3930 us/op 1.08
copy serialized Status (84 B) 1.3650 us/op 1.2000 us/op 1.14
transfer serialized SignedVoluntaryExit (112 B) 1.6480 us/op 1.4940 us/op 1.10
copy serialized SignedVoluntaryExit (112 B) 1.3760 us/op 1.2670 us/op 1.09
transfer serialized ProposerSlashing (416 B) 2.1620 us/op 1.6320 us/op 1.32
copy serialized ProposerSlashing (416 B) 2.3100 us/op 1.8190 us/op 1.27
transfer serialized Attestation (485 B) 2.0140 us/op 1.7710 us/op 1.14
copy serialized Attestation (485 B) 2.1950 us/op 1.6600 us/op 1.32
transfer serialized AttesterSlashing (33232 B) 2.2370 us/op 1.6620 us/op 1.35
copy serialized AttesterSlashing (33232 B) 7.2390 us/op 4.4390 us/op 1.63
transfer serialized Small SignedBeaconBlock (128000 B) 2.7880 us/op 2.4300 us/op 1.15
copy serialized Small SignedBeaconBlock (128000 B) 19.604 us/op 11.083 us/op 1.77
transfer serialized Avg SignedBeaconBlock (200000 B) 3.1780 us/op 3.8480 us/op 0.83
copy serialized Avg SignedBeaconBlock (200000 B) 27.815 us/op 33.781 us/op 0.82
transfer serialized BlobsSidecar (524380 B) 2.7800 us/op 5.0600 us/op 0.55
copy serialized BlobsSidecar (524380 B) 185.38 us/op 96.778 us/op 1.92
transfer serialized Big SignedBeaconBlock (1000000 B) 3.4460 us/op 4.2620 us/op 0.81
copy serialized Big SignedBeaconBlock (1000000 B) 181.53 us/op 170.37 us/op 1.07
pass gossip attestations to forkchoice per slot 3.1297 ms/op 2.6393 ms/op 1.19
forkChoice updateHead vc 100000 bc 64 eq 0 480.50 us/op 362.03 us/op 1.33
forkChoice updateHead vc 600000 bc 64 eq 0 4.2121 ms/op 2.3142 ms/op 1.82
forkChoice updateHead vc 1000000 bc 64 eq 0 5.7683 ms/op 3.7612 ms/op 1.53
forkChoice updateHead vc 600000 bc 320 eq 0 2.9740 ms/op 2.2329 ms/op 1.33
forkChoice updateHead vc 600000 bc 1200 eq 0 3.0373 ms/op 2.2768 ms/op 1.33
forkChoice updateHead vc 600000 bc 7200 eq 0 4.0611 ms/op 3.2816 ms/op 1.24
forkChoice updateHead vc 600000 bc 64 eq 1000 10.603 ms/op 9.8071 ms/op 1.08
forkChoice updateHead vc 600000 bc 64 eq 10000 10.634 ms/op 10.598 ms/op 1.00
forkChoice updateHead vc 600000 bc 64 eq 300000 14.459 ms/op 33.287 ms/op 0.43
computeDeltas 500000 validators 300 proto nodes 4.2177 ms/op 3.9676 ms/op 1.06
computeDeltas 500000 validators 1200 proto nodes 4.4124 ms/op 4.4810 ms/op 0.98
computeDeltas 500000 validators 7200 proto nodes 4.3483 ms/op 4.4845 ms/op 0.97
computeDeltas 750000 validators 300 proto nodes 6.2271 ms/op 5.9240 ms/op 1.05
computeDeltas 750000 validators 1200 proto nodes 6.2004 ms/op 5.9333 ms/op 1.05
computeDeltas 750000 validators 7200 proto nodes 6.2513 ms/op 5.6229 ms/op 1.11
computeDeltas 1400000 validators 300 proto nodes 11.500 ms/op 11.291 ms/op 1.02
computeDeltas 1400000 validators 1200 proto nodes 11.592 ms/op 10.959 ms/op 1.06
computeDeltas 1400000 validators 7200 proto nodes 11.566 ms/op 11.094 ms/op 1.04
computeDeltas 2100000 validators 300 proto nodes 17.742 ms/op 15.617 ms/op 1.14
computeDeltas 2100000 validators 1200 proto nodes 17.317 ms/op 14.922 ms/op 1.16
computeDeltas 2100000 validators 7200 proto nodes 17.809 ms/op 14.947 ms/op 1.19
altair processAttestation - 250000 vs - 7PWei normalcase 2.1508 ms/op 1.8098 ms/op 1.19
altair processAttestation - 250000 vs - 7PWei worstcase 2.9691 ms/op 2.7148 ms/op 1.09
altair processAttestation - setStatus - 1/6 committees join 112.33 us/op 69.999 us/op 1.60
altair processAttestation - setStatus - 1/3 committees join 199.02 us/op 138.01 us/op 1.44
altair processAttestation - setStatus - 1/2 committees join 275.93 us/op 223.06 us/op 1.24
altair processAttestation - setStatus - 2/3 committees join 349.10 us/op 298.36 us/op 1.17
altair processAttestation - setStatus - 4/5 committees join 499.16 us/op 432.70 us/op 1.15
altair processAttestation - setStatus - 100% committees join 624.61 us/op 534.07 us/op 1.17
altair processBlock - 250000 vs - 7PWei normalcase 5.9571 ms/op 6.5070 ms/op 0.92
altair processBlock - 250000 vs - 7PWei normalcase hashState 27.801 ms/op 27.734 ms/op 1.00
altair processBlock - 250000 vs - 7PWei worstcase 37.168 ms/op 39.817 ms/op 0.93
altair processBlock - 250000 vs - 7PWei worstcase hashState 78.422 ms/op 74.339 ms/op 1.05
phase0 processBlock - 250000 vs - 7PWei normalcase 2.6301 ms/op 1.7987 ms/op 1.46
phase0 processBlock - 250000 vs - 7PWei worstcase 28.770 ms/op 24.167 ms/op 1.19
altair processEth1Data - 250000 vs - 7PWei normalcase 544.03 us/op 272.46 us/op 2.00
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 6.5180 us/op 6.8150 us/op 0.96
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 46.137 us/op 35.235 us/op 1.31
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 11.725 us/op 10.994 us/op 1.07
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 6.4220 us/op 7.4530 us/op 0.86
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 183.98 us/op 130.70 us/op 1.41
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 1.3722 ms/op 942.07 us/op 1.46
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 1.9105 ms/op 1.2373 ms/op 1.54
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 1.8432 ms/op 1.1865 ms/op 1.55
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 3.8533 ms/op 2.9855 ms/op 1.29
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 1.9600 ms/op 1.3108 ms/op 1.50
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 4.1380 ms/op 3.3390 ms/op 1.24
Tree 40 250000 create 250.01 ms/op 205.63 ms/op 1.22
Tree 40 250000 get(125000) 162.86 ns/op 120.17 ns/op 1.36
Tree 40 250000 set(125000) 835.09 ns/op 544.81 ns/op 1.53
Tree 40 250000 toArray() 21.218 ms/op 18.600 ms/op 1.14
Tree 40 250000 iterate all - toArray() + loop 21.937 ms/op 19.371 ms/op 1.13
Tree 40 250000 iterate all - get(i) 63.848 ms/op 56.823 ms/op 1.12
Array 250000 create 3.3986 ms/op 3.5482 ms/op 0.96
Array 250000 clone - spread 1.5600 ms/op 1.4740 ms/op 1.06
Array 250000 get(125000) 0.43400 ns/op 0.67400 ns/op 0.64
Array 250000 set(125000) 0.46300 ns/op 0.66100 ns/op 0.70
Array 250000 iterate all - loop 98.202 us/op 84.189 us/op 1.17
phase0 afterProcessEpoch - 250000 vs - 7PWei 53.493 ms/op 46.251 ms/op 1.16
Array.fill - length 1000000 5.0930 ms/op 2.7042 ms/op 1.88
Array push - length 1000000 22.667 ms/op 19.503 ms/op 1.16
Array.get 0.29507 ns/op 0.27241 ns/op 1.08
Uint8Array.get 0.46783 ns/op 0.34837 ns/op 1.34
phase0 beforeProcessEpoch - 250000 vs - 7PWei 22.344 ms/op 14.886 ms/op 1.50
altair processEpoch - mainnet_e81889 303.27 ms/op 239.57 ms/op 1.27
mainnet_e81889 - altair beforeProcessEpoch 21.266 ms/op 18.722 ms/op 1.14
mainnet_e81889 - altair processJustificationAndFinalization 14.692 us/op 16.217 us/op 0.91
mainnet_e81889 - altair processInactivityUpdates 6.7082 ms/op 5.7208 ms/op 1.17
mainnet_e81889 - altair processRewardsAndPenalties 44.317 ms/op 41.907 ms/op 1.06
mainnet_e81889 - altair processRegistryUpdates 2.8340 us/op 2.5940 us/op 1.09
mainnet_e81889 - altair processSlashings 1.0620 us/op 1.2780 us/op 0.83
mainnet_e81889 - altair processEth1DataReset 1.6310 us/op 902.00 ns/op 1.81
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.4213 ms/op 1.1205 ms/op 2.16
mainnet_e81889 - altair processSlashingsReset 6.2920 us/op 4.3230 us/op 1.46
mainnet_e81889 - altair processRandaoMixesReset 6.8200 us/op 6.9260 us/op 0.98
mainnet_e81889 - altair processHistoricalRootsUpdate 794.00 ns/op 1.2960 us/op 0.61
mainnet_e81889 - altair processParticipationFlagUpdates 2.7060 us/op 3.9950 us/op 0.68
mainnet_e81889 - altair processSyncCommitteeUpdates 436.00 ns/op 902.00 ns/op 0.48
mainnet_e81889 - altair afterProcessEpoch 55.021 ms/op 44.422 ms/op 1.24
capella processEpoch - mainnet_e217614 1.2396 s/op 1.0892 s/op 1.14
mainnet_e217614 - capella beforeProcessEpoch 86.561 ms/op 71.670 ms/op 1.21
mainnet_e217614 - capella processJustificationAndFinalization 26.579 us/op 14.652 us/op 1.81
mainnet_e217614 - capella processInactivityUpdates 23.912 ms/op 18.642 ms/op 1.28
mainnet_e217614 - capella processRewardsAndPenalties 280.02 ms/op 231.55 ms/op 1.21
mainnet_e217614 - capella processRegistryUpdates 21.621 us/op 15.166 us/op 1.43
mainnet_e217614 - capella processSlashings 441.00 ns/op 755.00 ns/op 0.58
mainnet_e217614 - capella processEth1DataReset 485.00 ns/op 969.00 ns/op 0.50
mainnet_e217614 - capella processEffectiveBalanceUpdates 18.081 ms/op 17.095 ms/op 1.06
mainnet_e217614 - capella processSlashingsReset 6.4320 us/op 4.8910 us/op 1.32
mainnet_e217614 - capella processRandaoMixesReset 8.8000 us/op 3.7280 us/op 2.36
mainnet_e217614 - capella processHistoricalRootsUpdate 1.1180 us/op 812.00 ns/op 1.38
mainnet_e217614 - capella processParticipationFlagUpdates 2.8180 us/op 1.9490 us/op 1.45
mainnet_e217614 - capella afterProcessEpoch 133.13 ms/op 106.48 ms/op 1.25
phase0 processEpoch - mainnet_e58758 363.25 ms/op 288.65 ms/op 1.26
mainnet_e58758 - phase0 beforeProcessEpoch 101.60 ms/op 89.901 ms/op 1.13
mainnet_e58758 - phase0 processJustificationAndFinalization 23.479 us/op 19.786 us/op 1.19
mainnet_e58758 - phase0 processRewardsAndPenalties 29.067 ms/op 34.784 ms/op 0.84
mainnet_e58758 - phase0 processRegistryUpdates 10.864 us/op 9.6730 us/op 1.12
mainnet_e58758 - phase0 processSlashings 343.00 ns/op 869.00 ns/op 0.39
mainnet_e58758 - phase0 processEth1DataReset 413.00 ns/op 747.00 ns/op 0.55
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.6852 ms/op 814.30 us/op 2.07
mainnet_e58758 - phase0 processSlashingsReset 4.7670 us/op 5.2260 us/op 0.91
mainnet_e58758 - phase0 processRandaoMixesReset 5.4980 us/op 3.6550 us/op 1.50
mainnet_e58758 - phase0 processHistoricalRootsUpdate 568.00 ns/op 846.00 ns/op 0.67
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.6910 us/op 3.2040 us/op 1.15
mainnet_e58758 - phase0 afterProcessEpoch 45.000 ms/op 36.038 ms/op 1.25
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.9364 ms/op 1.0017 ms/op 1.93
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.1425 ms/op 1.8746 ms/op 1.14
altair processInactivityUpdates - 250000 normalcase 20.559 ms/op 17.320 ms/op 1.19
altair processInactivityUpdates - 250000 worstcase 20.177 ms/op 16.996 ms/op 1.19
phase0 processRegistryUpdates - 250000 normalcase 8.4740 us/op 9.2270 us/op 0.92
phase0 processRegistryUpdates - 250000 badcase_full_deposits 348.06 us/op 309.50 us/op 1.12
phase0 processRegistryUpdates - 250000 worstcase 0.5 128.55 ms/op 110.84 ms/op 1.16
altair processRewardsAndPenalties - 250000 normalcase 43.879 ms/op 34.267 ms/op 1.28
altair processRewardsAndPenalties - 250000 worstcase 48.741 ms/op 43.353 ms/op 1.12
phase0 getAttestationDeltas - 250000 normalcase 8.8290 ms/op 6.3054 ms/op 1.40
phase0 getAttestationDeltas - 250000 worstcase 7.7016 ms/op 6.2771 ms/op 1.23
phase0 processSlashings - 250000 worstcase 115.83 us/op 95.435 us/op 1.21
altair processSyncCommitteeUpdates - 250000 132.85 ms/op 104.39 ms/op 1.27
BeaconState.hashTreeRoot - No change 235.00 ns/op 442.00 ns/op 0.53
BeaconState.hashTreeRoot - 1 full validator 148.37 us/op 144.94 us/op 1.02
BeaconState.hashTreeRoot - 32 full validator 1.2066 ms/op 1.4787 ms/op 0.82
BeaconState.hashTreeRoot - 512 full validator 13.440 ms/op 10.679 ms/op 1.26
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 166.97 us/op 142.06 us/op 1.18
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 2.6532 ms/op 1.8790 ms/op 1.41
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 27.963 ms/op 22.012 ms/op 1.27
BeaconState.hashTreeRoot - 1 balances 144.44 us/op 106.58 us/op 1.36
BeaconState.hashTreeRoot - 32 balances 990.40 us/op 1.0473 ms/op 0.95
BeaconState.hashTreeRoot - 512 balances 9.5744 ms/op 7.7566 ms/op 1.23
BeaconState.hashTreeRoot - 250000 balances 194.99 ms/op 129.55 ms/op 1.51
aggregationBits - 2048 els - zipIndexesInBitList 25.763 us/op 50.929 us/op 0.51
byteArrayEquals 32 55.411 ns/op 47.742 ns/op 1.16
Buffer.compare 32 17.385 ns/op 15.808 ns/op 1.10
byteArrayEquals 1024 1.6467 us/op 1.2760 us/op 1.29
Buffer.compare 1024 26.121 ns/op 24.547 ns/op 1.06
byteArrayEquals 16384 25.735 us/op 20.198 us/op 1.27
Buffer.compare 16384 198.62 ns/op 187.46 ns/op 1.06
byteArrayEquals 123687377 194.07 ms/op 146.16 ms/op 1.33
Buffer.compare 123687377 7.2863 ms/op 5.5138 ms/op 1.32
byteArrayEquals 32 - diff last byte 52.792 ns/op 46.076 ns/op 1.15
Buffer.compare 32 - diff last byte 17.089 ns/op 15.199 ns/op 1.12
byteArrayEquals 1024 - diff last byte 1.5841 us/op 1.2391 us/op 1.28
Buffer.compare 1024 - diff last byte 27.304 ns/op 22.217 ns/op 1.23
byteArrayEquals 16384 - diff last byte 25.266 us/op 19.368 us/op 1.30
Buffer.compare 16384 - diff last byte 209.48 ns/op 187.16 ns/op 1.12
byteArrayEquals 123687377 - diff last byte 195.21 ms/op 147.14 ms/op 1.33
Buffer.compare 123687377 - diff last byte 7.9735 ms/op 5.3933 ms/op 1.48
byteArrayEquals 32 - random bytes 5.2570 ns/op 4.7700 ns/op 1.10
Buffer.compare 32 - random bytes 17.579 ns/op 15.169 ns/op 1.16
byteArrayEquals 1024 - random bytes 5.5260 ns/op 4.7960 ns/op 1.15
Buffer.compare 1024 - random bytes 17.375 ns/op 14.900 ns/op 1.17
byteArrayEquals 16384 - random bytes 5.7700 ns/op 4.8320 ns/op 1.19
Buffer.compare 16384 - random bytes 17.792 ns/op 14.986 ns/op 1.19
byteArrayEquals 123687377 - random bytes 6.6200 ns/op 7.5900 ns/op 0.87
Buffer.compare 123687377 - random bytes 18.770 ns/op 17.710 ns/op 1.06
regular array get 100000 times 42.341 us/op 29.810 us/op 1.42
wrappedArray get 100000 times 35.522 us/op 29.748 us/op 1.19
arrayWithProxy get 100000 times 14.693 ms/op 9.6938 ms/op 1.52
ssz.Root.equals 46.740 ns/op 38.378 ns/op 1.22
byteArrayEquals 45.884 ns/op 41.944 ns/op 1.09
Buffer.compare 10.554 ns/op 8.7390 ns/op 1.21
processSlot - 1 slots 18.286 us/op 12.137 us/op 1.51
processSlot - 32 slots 3.2526 ms/op 2.0926 ms/op 1.55
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 39.697 ms/op 38.011 ms/op 1.04
getCommitteeAssignments - req 1 vs - 250000 vc 2.1494 ms/op 1.8038 ms/op 1.19
getCommitteeAssignments - req 100 vs - 250000 vc 4.2042 ms/op 3.5153 ms/op 1.20
getCommitteeAssignments - req 1000 vs - 250000 vc 4.4830 ms/op 3.7587 ms/op 1.19
findModifiedValidators - 10000 modified validators 287.82 ms/op 241.54 ms/op 1.19
findModifiedValidators - 1000 modified validators 180.41 ms/op 143.46 ms/op 1.26
findModifiedValidators - 100 modified validators 180.20 ms/op 135.70 ms/op 1.33
findModifiedValidators - 10 modified validators 175.31 ms/op 144.80 ms/op 1.21
findModifiedValidators - 1 modified validators 145.72 ms/op 135.00 ms/op 1.08
findModifiedValidators - no difference 179.36 ms/op 145.99 ms/op 1.23
compare ViewDUs 3.2619 s/op 3.0838 s/op 1.06
compare each validator Uint8Array 1.8810 s/op 1.1173 s/op 1.68
compare ViewDU to Uint8Array 1.1360 s/op 753.23 ms/op 1.51
migrate state 1000000 validators, 24 modified, 0 new 708.23 ms/op 704.54 ms/op 1.01
migrate state 1000000 validators, 1700 modified, 1000 new 1.0203 s/op 886.77 ms/op 1.15
migrate state 1000000 validators, 3400 modified, 2000 new 1.2443 s/op 1.1221 s/op 1.11
migrate state 1500000 validators, 24 modified, 0 new 854.35 ms/op 662.53 ms/op 1.29
migrate state 1500000 validators, 1700 modified, 1000 new 969.71 ms/op 911.90 ms/op 1.06
migrate state 1500000 validators, 3400 modified, 2000 new 1.2795 s/op 1.1103 s/op 1.15
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.4600 ns/op 6.1000 ns/op 0.73
state getBlockRootAtSlot - 250000 vs - 7PWei 672.97 ns/op 376.82 ns/op 1.79
computeProposers - vc 250000 8.6343 ms/op 5.4958 ms/op 1.57
computeEpochShuffling - vc 250000 42.448 ms/op 35.092 ms/op 1.21
getNextSyncCommittee - vc 250000 133.28 ms/op 106.70 ms/op 1.25
computeSigningRoot for AttestationData 19.421 us/op 22.188 us/op 0.88
hash AttestationData serialized data then Buffer.toString(base64) 1.6133 us/op 1.1877 us/op 1.36
toHexString serialized data 974.32 ns/op 811.85 ns/op 1.20
Buffer.toString(base64) 201.86 ns/op 138.27 ns/op 1.46
nodejs block root to RootHex using toHex 178.23 ns/op 126.28 ns/op 1.41
nodejs block root to RootHex using toRootHex 104.95 ns/op 79.003 ns/op 1.33
browser block root to RootHex using the deprecated toHexString 258.62 ns/op 216.69 ns/op 1.19
browser block root to RootHex using toHex 189.63 ns/op 164.95 ns/op 1.15
browser block root to RootHex using toRootHex 168.04 ns/op 146.22 ns/op 1.15

by benchmarkbot/action

@twoeths twoeths marked this pull request as ready for review November 27, 2024 03:39
@twoeths twoeths requested a review from a team as a code owner November 27, 2024 03:39
@codecov Codecov
Copy link

codecov bot commented Nov 27, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 48.51%. Comparing base (18f4218) to head (7c22054).
Report is 1 commits behind head on unstable.

Additional details and impacted files 
@@            Coverage Diff            @@
##           unstable    #7241   +/-   ##
=========================================
  Coverage     48.51%   48.51%           
=========================================
  Files           600      600           
  Lines         40121    40121           
  Branches       2054     2054           
=========================================
  Hits          19466    19466           
  Misses        20617    20617           
  Partials         38       38

@twoeths twoeths merged commit e86e816 into unstable Nov 27, 2024
20 checks passed
@twoeths twoeths deleted the nft-devnet-0 branch November 27, 2024 08:22
nflaig pushed a commit that referenced this pull request Nov 28, 2024
@twoeths @nflaig
fix: prune checkpoint states at syncing time (#7241)
ffa4bb8 
* fix: prune checkpoint states at syncing time

* fix: lint

* fix: check-types in test
@philknows philknows mentioned this pull request Nov 28, 2024
Merged
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.23.1 🎉

wemeetagain added a commit that referenced this pull request Dec 20, 2024
@wemeetagain @nflaig
@nazarhussain @g11tech @matthewkeil @ensi321 @twoeths @philknows @dependabot @acolytec3 @varunguleriaCodes @ClockworkYuzu
chore: release v1.24.0 (#7325)
11f35a8 
* feat: add keymanager endpoint to retrieve proposer config (#7210)

* feat: add keymanager endpoint to retrieve proposer config

* Do not return empty builder config

* Check all builder proposer config values

* Fix settings builder config if undefined

* Fix builder config parsing

* Use ssz type to handle json serialization

Default parsing can't handle BigInt

* Revert "Use ssz type to handle json serialization"

This reverts commit 01fcea7.

* Fix boost factor json serialization

* Remove unused import

* Update test data

* Update proposer config test

* feat: add mekong network option (#7212)

* chore: fix import order with biome syntax (#7211)

Fix import order

* fix: consistently validate pubkey and throw 404 if not found (#7214)

* Throw error if pubkey is unknown when getting graffiti

* Consistently validate pubkey and throw 404 if not found

* fix: only return local keys from /eth/v1/keystores (#7215)

* fix: only return local keys from /eth/v1/keystores

* Fix fetching remote keys in node assertion

* feat: add and use getBlobsV1 to expedite gossip import (#7134)

* hookup the getblobs api to get bob and proof data from el

remove unused

fix import

metrics overhault, test, debugging testing, some feeback

fix

add nethermind bug dicussion link

fix

resolve conflicts

* deblobs timeout

* fix metric

* chore: revert async aggregate with randomness (#7218)

Revert "feat: asyncAggregateWithRandomness (#7204)"

This reverts commit e31d535.

* fix: update config for relaunched mekong network (#7220)

* fix: light client generating `LightClientUpdate` with wrong length of branches (#7187)

* initial commit

* Rewrite SyncCommitteeWitnessRepository

* Fix finality branch

* Update unit test

* fix e2e

* Review PR

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* fix: archive finalized state when shutting down beacon node (#7221)

* Fix typo

* feat: remove unfinalized pubkey cache (#7230)

* Remove unfinalized pubkey cache

* lint

* Fix unit test

* chore: skip web3_provider unit tests (#7252)

* fix: prune checkpoint states at syncing time (#7241)

* fix: prune checkpoint states at syncing time

* fix: lint

* fix: check-types in test

* fix: sync cached isCompoundingValidatorArr at epoch transition (#7247)

* fix: handle outOfRangeData when range sync Deneb (#7249)

* fix: handle outOfRangeData for beaconBlocksMaybeBlobsByRange()

* fix: lint

* fix: archiveBlocks - handle deneb outOfRangeData block

* fix: sync cached balance when adding new validator to registry (#7255)

* fix: sync cached balance when adding new validator to registry

* chore: add more comments

* fix: remove persisted checkpoint states from the previous run at startup

* fix: do not throw error when trying to prune missing directory (#7257)

* docs: update documentation Oct 2024 (#7178)

* docs update oct 2024 init

* Reconfig quickstart nav and minor fixes

* fix lint

* spelling fixes

* minor fixes and add to wordlist

* prettier fix

* add to wordlist

* sort wordlist

* modify dominance to include lighthouse

* fix typescript casing and add recommendation

* add selection and boost_factor with keymanager notice

* update wordlist

* remove builder enabled and add keymanager api

* spelling

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* chore(deps): bump cross-spawn from 7.0.3 to 7.0.6 in /docs (#7268)

Bumps [cross-spawn](https://github.com/moxystudio/node-cross-spawn) from 7.0.3 to 7.0.6.
- [Changelog](https://github.com/moxystudio/node-cross-spawn/blob/master/CHANGELOG.md)
- [Commits](moxystudio/node-cross-spawn@v7.0.3...v7.0.6)

---
updated-dependencies:
- dependency-name: cross-spawn
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: add error log to notifier if execution client auth failed (#7239)

* feat: add error log to notifier if execution client auth failed

* Update packages/beacon-node/src/node/notifier.ts

---------

Co-authored-by: NC <17676176+ensi321@users.noreply.github.com>

* docs: display rcConfig flag on CLI reference page (#7270)

* docs: display rcConfig flag on CLI reference page

* Update word list

* chore: remove prettier as default formatter for all file types (#7275)

* chore: unhide flags relevant for devnets / testing (#7271)

* feat: debug too many shuffling promises (#7251)

* feat: add asyncShufflingCalculation to StateTransitionOpts

* feat: add asyncShufflingCalculation to all regen / processSlots consumers

* fix: default to false for async shuffling and remove unnecessary props

* fix: remove unnecessary flags from stateTransition

* feat: implement conditional build of shuffling for prepareNextSlot

* fix: spec test bug where shufflingCache is present from BeaconChain constructor

* feat: sync build next shuffling if not queued async

* fix: use getSync to pull next shuffling correctly

* docs: add comment to prepareNextSlot

* refactor: rename StateCloneOpts to StateRegenerationOpts

* feat: pass asyncShufflingCalculation through to afterProcessEpoch and refactor conditional to run purely sync

* docs: add issue number to comment

* chore: lint

* chore: unpin nodejs version from 22.4 (#6982)

* Revert "chore: pin nodejs version to 22.4 (#6964)"

This reverts commit f20484b.

* Don't revert formatting changes

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>
Co-authored-by: Cayman <caymannava@gmail.com>

* chore: update bootnodes file url for holesky and sepolia (#7276)

* feat: add `debug_getHistoricalSummaries` endpoint (#7245)

* feat: add new getHistoricalSummaries endpoint to debug namespace

* Add JSON response

* Restructure to use stateId and add proof to response

* add test scaffolding

* Address feedback

* Move getHistoricalSummaries to lodestar namespace

* add lodestar namespace unit test

* update route name to lodestar namespace

* cast state object as Capella state

* Lint

* json properties need to be lower case

* Make it v1 since it's now part of lodestar namespace

* Group with other /lodestar endpoints

* Simplify beacon node impl

* Rename return type

* Update test description

* Fix variable name

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* chore: log sync committee signature errors as `error` (#7283)

* fix: update engine_getClientVersionV1 commit encoding (#7282)

* fix: check pubkey or validator index known to a state (#7284)

* fix: check pubkey or validator index known to a state

* chore: add more comments

* feat: lodestar script setup (#7254)

* feat: lodestar_setup

* feat: script_updates + docs

* feat: script_addition_in_docs + command_update

* Remove duplicate script from docs folder

* Minor script updates

* Update script to prepare docs and ignore copied file

* Update installation page

* Wording

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* feat: add terminal-sized Electra giraffe banner (#7286)

* Create giraffeBanners.ts

* Wire in banner

* Fix file name

* lint

* Address @nflaig's comment

---------

Co-authored-by: NC <17676176+ensi321@users.noreply.github.com>

* chore: pin nodejs version to 22.4 (#7291)

Revert "chore: unpin nodejs version from 22.4 (#6982)"

This reverts commit 69ae688.

* feat: expose `DOMAIN_APPLICATION_MASK` in config/spec api (#7296)

* feat: expose DOMAIN_APPLICATION_MASK in config/spec api

* Lint

* feat: make `MAX_REQUEST_BLOB_SIDECARS` and `MAX_BLOBS_PER_BLOCK` configurable (#7294)

* Init commit

* Fix check-types

* Add comment on how MAX_REQUEST_BLOB_SIDECARS is calculated

* Ensure proper config object is passed

* Address comment

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* feat: use `BLOB_SIDECAR_SUBNET_COUNT` to configure blob subnets (#7297)

feat: use BLOB_SIDECAR_SUBNET_COUNT to configure blob subnets

* chore: log sync aggregate participants when producing beacon block body (#7300)

* chore: log sync aggregate participants when producing beacon block body

* Use isForkLightClient instead of ForkSeq

* Fix produce block unit tests

* chore: print graffiti when producing beacon block body (#7303)

* fix: warn if engine / builder failed to produce block within cutoff time (#7305)

* feat: add kzg commitment length check when validating gossip blocks (#7302)

* feat: add blob sidecar index check (#7313)

Validate blobSidecar index

* fix: fix blob sidecar index check (#7315)

Fix index check

* chore: fix format of printed graffiti from hex to utf-8 (#7306)

* chore: fix format of printed graffiti from hex to utf-8

* Use Buffer.from no copy with offset

* docs: batch commit typos and update contributor readme (#7312)

* batch commit typos and update contributor readme

* update donation text

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* correct spelling

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* chore: remove trailing null bytes from printed graffiti (#7320)

* chore: remove trailing null bytes from printed graffiti

* Use replaceAll instead of regex

* chore: unpin nodejs version from 22.4 (#7324)

Revert "chore: pin nodejs version to 22.4 (#7291)"

This reverts commit 99794d3.

* chore: bump package versions to 1.24.0

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
Co-authored-by: Nazar Hussain <nazarhussain@gmail.com>
Co-authored-by: g11tech <develop@g11tech.io>
Co-authored-by: Matthew Keil <me@matthewkeil.com>
Co-authored-by: NC <17676176+ensi321@users.noreply.github.com>
Co-authored-by: twoeths <tuyen@chainsafe.io>
Co-authored-by: Phil Ngo <58080811+philknows@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>
Co-authored-by: Varun Guleria <152203177+varunguleriaCodes@users.noreply.github.com>
Co-authored-by: ClockworkYuzu <yvetteheh@gmail.com>
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.24.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nflaig nflaig nflaig approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Lodestar got OOM during big unfinality period
3 participants
@twoeths @wemeetagain @nflaig