Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add and use getBlobsV1 to expedite gossip import #7134

Merged
merged 3 commits into from
Nov 3, 2024
Merged

feat: add and use getBlobsV1 to expedite gossip import #7134

merged 3 commits into from
Nov 3, 2024

Conversation

g11tech
Copy link
Contributor

@g11tech g11tech commented Oct 8, 2024 •
edited
Loading

one may review but still

  • undergoing testing/observation
    image

Closes #7114

pending todo for a followup PR

  • add dashboard panels for the metrics

@g11tech g11tech requested a review from a team as a code owner October 8, 2024 13:38
@g11tech g11tech closed this Oct 8, 2024
@g11tech g11tech reopened this Oct 8, 2024
@codecov Codecov
Copy link

codecov bot commented Oct 8, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 2.60870% with 224 lines in your changes missing coverage. Please review.

Project coverage is 49.09%. Comparing base (06b4c2d) to head (11e21f0).
Report is 3 commits behind head on unstable.

Additional details and impacted files 
@@             Coverage Diff              @@
##           unstable    #7134      +/-   ##
============================================
- Coverage     49.38%   49.09%   -0.29%     
============================================
  Files           599      600       +1     
  Lines         39956    40214     +258     
  Branches       2097     2101       +4     
============================================
+ Hits          19733    19745      +12     
- Misses        20185    20431     +246     
  Partials         38       38

@philknows philknows added this to the v1.23.0 milestone Oct 8, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 8, 2024 •
edited
Loading

Performance Report

✔️ no performance regression detected

🚀🚀 Significant benchmark improvement detected

Benchmark suite Current: a7b8430 Previous: 06b4c2d Ratio
getSlashingsAndExits - default max 63.370 us/op 215.80 us/op 0.29
Full benchmark results
Benchmark suite Current: a7b8430 Previous: 06b4c2d Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 1.8611 ms/op 2.2642 ms/op 0.82
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 38.824 us/op 69.775 us/op 0.56
BLS verify - blst 872.35 us/op 1.0233 ms/op 0.85
BLS verifyMultipleSignatures 3 - blst 1.2989 ms/op 1.5842 ms/op 0.82
BLS verifyMultipleSignatures 8 - blst 2.1245 ms/op 2.5115 ms/op 0.85
BLS verifyMultipleSignatures 32 - blst 4.4832 ms/op 6.5633 ms/op 0.68
BLS verifyMultipleSignatures 64 - blst 8.4386 ms/op 11.133 ms/op 0.76
BLS verifyMultipleSignatures 128 - blst 16.030 ms/op 20.299 ms/op 0.79
BLS deserializing 10000 signatures 627.04 ms/op 752.85 ms/op 0.83
BLS deserializing 100000 signatures 6.2135 s/op 7.2695 s/op 0.85
BLS verifyMultipleSignatures - same message - 3 - blst 926.76 us/op 935.86 us/op 0.99
BLS verifyMultipleSignatures - same message - 8 - blst 1.1008 ms/op 1.0577 ms/op 1.04
BLS verifyMultipleSignatures - same message - 32 - blst 1.6187 ms/op 1.8107 ms/op 0.89
BLS verifyMultipleSignatures - same message - 64 - blst 2.4018 ms/op 2.7299 ms/op 0.88
BLS verifyMultipleSignatures - same message - 128 - blst 3.9289 ms/op 4.7750 ms/op 0.82
BLS aggregatePubkeys 32 - blst 17.351 us/op 21.806 us/op 0.80
BLS aggregatePubkeys 128 - blst 60.429 us/op 77.726 us/op 0.78
notSeenSlots=1 numMissedVotes=1 numBadVotes=10 64.543 ms/op 79.474 ms/op 0.81
notSeenSlots=1 numMissedVotes=0 numBadVotes=4 43.745 ms/op 68.149 ms/op 0.64
notSeenSlots=2 numMissedVotes=1 numBadVotes=10 37.204 ms/op 54.022 ms/op 0.69
getSlashingsAndExits - default max 63.370 us/op 215.80 us/op 0.29
getSlashingsAndExits - 2k 229.61 us/op 500.51 us/op 0.46
proposeBlockBody type=full, size=empty 4.7047 ms/op 8.3371 ms/op 0.56
isKnown best case - 1 super set check 484.00 ns/op 705.00 ns/op 0.69
isKnown normal case - 2 super set checks 468.00 ns/op 741.00 ns/op 0.63
isKnown worse case - 16 super set checks 465.00 ns/op 734.00 ns/op 0.63
InMemoryCheckpointStateCache - add get delete 2.9700 us/op 5.1650 us/op 0.58
updateUnfinalizedPubkeys - updating 10 pubkeys 685.51 us/op 1.6693 ms/op 0.41
updateUnfinalizedPubkeys - updating 100 pubkeys 2.6770 ms/op 6.3416 ms/op 0.42
updateUnfinalizedPubkeys - updating 1000 pubkeys 36.759 ms/op 84.247 ms/op 0.44
validate api signedAggregateAndProof - struct 1.4537 ms/op 1.9358 ms/op 0.75
validate gossip signedAggregateAndProof - struct 1.4285 ms/op 1.8343 ms/op 0.78
batch validate gossip attestation - vc 640000 - chunk 32 119.76 us/op 159.07 us/op 0.75
batch validate gossip attestation - vc 640000 - chunk 64 102.22 us/op 134.14 us/op 0.76
batch validate gossip attestation - vc 640000 - chunk 128 95.034 us/op 119.56 us/op 0.79
batch validate gossip attestation - vc 640000 - chunk 256 93.433 us/op 108.45 us/op 0.86
pickEth1Vote - no votes 919.88 us/op 1.0872 ms/op 0.85
pickEth1Vote - max votes 5.3303 ms/op 6.8299 ms/op 0.78
pickEth1Vote - Eth1Data hashTreeRoot value x2048 16.010 ms/op 14.225 ms/op 1.13
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 18.309 ms/op 17.707 ms/op 1.03
pickEth1Vote - Eth1Data fastSerialize value x2048 374.90 us/op 489.10 us/op 0.77
pickEth1Vote - Eth1Data fastSerialize tree x2048 4.3485 ms/op 2.9889 ms/op 1.45
bytes32 toHexString 597.00 ns/op 489.00 ns/op 1.22
bytes32 Buffer.toString(hex) 434.00 ns/op 284.00 ns/op 1.53
bytes32 Buffer.toString(hex) from Uint8Array 535.00 ns/op 470.00 ns/op 1.14
bytes32 Buffer.toString(hex) + 0x 439.00 ns/op 288.00 ns/op 1.52
Object access 1 prop 0.32500 ns/op 0.20000 ns/op 1.63
Map access 1 prop 0.32800 ns/op 0.14100 ns/op 2.33
Object get x1000 4.9740 ns/op 6.2420 ns/op 0.80
Map get x1000 5.5530 ns/op 6.5660 ns/op 0.85
Object set x1000 22.250 ns/op 42.821 ns/op 0.52
Map set x1000 18.647 ns/op 30.148 ns/op 0.62
Return object 10000 times 0.28560 ns/op 0.31060 ns/op 0.92
Throw Error 10000 times 2.5684 us/op 3.6302 us/op 0.71
toHex 102.44 ns/op 178.57 ns/op 0.57
Buffer.from 95.224 ns/op 158.32 ns/op 0.60
shared Buffer 62.799 ns/op 99.318 ns/op 0.63
fastMsgIdFn sha256 / 200 bytes 1.8920 us/op 2.3800 us/op 0.79
fastMsgIdFn h32 xxhash / 200 bytes 406.00 ns/op 292.00 ns/op 1.39
fastMsgIdFn h64 xxhash / 200 bytes 459.00 ns/op 283.00 ns/op 1.62
fastMsgIdFn sha256 / 1000 bytes 5.9800 us/op 7.8470 us/op 0.76
fastMsgIdFn h32 xxhash / 1000 bytes 529.00 ns/op 404.00 ns/op 1.31
fastMsgIdFn h64 xxhash / 1000 bytes 518.00 ns/op 360.00 ns/op 1.44
fastMsgIdFn sha256 / 10000 bytes 50.270 us/op 66.916 us/op 0.75
fastMsgIdFn h32 xxhash / 10000 bytes 1.9030 us/op 1.8780 us/op 1.01
fastMsgIdFn h64 xxhash / 10000 bytes 1.3360 us/op 1.2410 us/op 1.08
send data - 1000 256B messages 9.7000 ms/op 13.163 ms/op 0.74
send data - 1000 512B messages 13.464 ms/op 17.538 ms/op 0.77
send data - 1000 1024B messages 22.352 ms/op 27.848 ms/op 0.80
send data - 1000 1200B messages 23.463 ms/op 28.765 ms/op 0.82
send data - 1000 2048B messages 28.970 ms/op 34.714 ms/op 0.83
send data - 1000 4096B messages 25.522 ms/op 33.672 ms/op 0.76
send data - 1000 16384B messages 64.648 ms/op 75.829 ms/op 0.85
send data - 1000 65536B messages 251.87 ms/op 226.25 ms/op 1.11
enrSubnets - fastDeserialize 64 bits 1.1440 us/op 1.2320 us/op 0.93
enrSubnets - ssz BitVector 64 bits 517.00 ns/op 416.00 ns/op 1.24
enrSubnets - fastDeserialize 4 bits 327.00 ns/op 178.00 ns/op 1.84
enrSubnets - ssz BitVector 4 bits 505.00 ns/op 439.00 ns/op 1.15
prioritizePeers score -10:0 att 32-0.1 sync 2-0 126.53 us/op 171.78 us/op 0.74
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 177.20 us/op 174.10 us/op 1.02
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 281.09 us/op 271.11 us/op 1.04
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 471.22 us/op 488.27 us/op 0.97
prioritizePeers score 0:0 att 64-1 sync 4-1 564.90 us/op 663.24 us/op 0.85
array of 16000 items push then shift 1.2448 us/op 1.6774 us/op 0.74
LinkedList of 16000 items push then shift 7.3080 ns/op 7.9920 ns/op 0.91
array of 16000 items push then pop 86.760 ns/op 128.96 ns/op 0.67
LinkedList of 16000 items push then pop 6.1530 ns/op 7.7530 ns/op 0.79
array of 24000 items push then shift 1.7678 us/op 2.4967 us/op 0.71
LinkedList of 24000 items push then shift 6.9310 ns/op 8.6550 ns/op 0.80
array of 24000 items push then pop 114.41 ns/op 174.31 ns/op 0.66
LinkedList of 24000 items push then pop 6.1590 ns/op 8.6110 ns/op 0.72
intersect bitArray bitLen 8 5.1750 ns/op 7.1120 ns/op 0.73
intersect array and set length 8 42.358 ns/op 57.282 ns/op 0.74
intersect bitArray bitLen 128 25.471 ns/op 31.722 ns/op 0.80
intersect array and set length 128 617.21 ns/op 848.98 ns/op 0.73
bitArray.getTrueBitIndexes() bitLen 128 2.1560 us/op 1.6110 us/op 1.34
bitArray.getTrueBitIndexes() bitLen 248 3.9670 us/op 2.8010 us/op 1.42
bitArray.getTrueBitIndexes() bitLen 512 6.8980 us/op 6.1080 us/op 1.13
Buffer.concat 32 items 1.0210 us/op 1.0810 us/op 0.94
Uint8Array.set 32 items 1.5720 us/op 2.2080 us/op 0.71
Buffer.copy 1.8230 us/op 2.4250 us/op 0.75
Uint8Array.set - with subarray 2.3530 us/op 3.3440 us/op 0.70
Uint8Array.set - without subarray 1.7190 us/op 1.6840 us/op 1.02
getUint32 - dataview 420.00 ns/op 298.00 ns/op 1.41
getUint32 - manual 348.00 ns/op 233.00 ns/op 1.49
Set add up to 64 items then delete first 1.8609 us/op 2.5983 us/op 0.72
OrderedSet add up to 64 items then delete first 2.9184 us/op 3.9284 us/op 0.74
Set add up to 64 items then delete last 2.0981 us/op 3.1416 us/op 0.67
OrderedSet add up to 64 items then delete last 3.4011 us/op 4.8114 us/op 0.71
Set add up to 64 items then delete middle 2.1517 us/op 3.1527 us/op 0.68
OrderedSet add up to 64 items then delete middle 4.7145 us/op 5.7934 us/op 0.81
Set add up to 128 items then delete first 4.1073 us/op 6.4026 us/op 0.64
OrderedSet add up to 128 items then delete first 6.1976 us/op 9.7599 us/op 0.63
Set add up to 128 items then delete last 4.1133 us/op 6.2224 us/op 0.66
OrderedSet add up to 128 items then delete last 6.3946 us/op 9.1490 us/op 0.70
Set add up to 128 items then delete middle 4.0790 us/op 6.0866 us/op 0.67
OrderedSet add up to 128 items then delete middle 12.232 us/op 16.456 us/op 0.74
Set add up to 256 items then delete first 8.0418 us/op 12.862 us/op 0.63
OrderedSet add up to 256 items then delete first 12.263 us/op 20.081 us/op 0.61
Set add up to 256 items then delete last 8.2146 us/op 11.574 us/op 0.71
OrderedSet add up to 256 items then delete last 12.774 us/op 17.467 us/op 0.73
Set add up to 256 items then delete middle 7.9913 us/op 11.422 us/op 0.70
OrderedSet add up to 256 items then delete middle 35.656 us/op 46.571 us/op 0.77
transfer serialized Status (84 B) 1.4530 us/op 1.6230 us/op 0.90
copy serialized Status (84 B) 1.2030 us/op 1.3300 us/op 0.90
transfer serialized SignedVoluntaryExit (112 B) 1.4260 us/op 1.7150 us/op 0.83
copy serialized SignedVoluntaryExit (112 B) 1.1940 us/op 1.4230 us/op 0.84
transfer serialized ProposerSlashing (416 B) 1.5730 us/op 2.1350 us/op 0.74
copy serialized ProposerSlashing (416 B) 1.3990 us/op 2.5770 us/op 0.54
transfer serialized Attestation (485 B) 1.6100 us/op 2.4090 us/op 0.67
copy serialized Attestation (485 B) 1.4020 us/op 2.6450 us/op 0.53
transfer serialized AttesterSlashing (33232 B) 1.7110 us/op 2.7110 us/op 0.63
copy serialized AttesterSlashing (33232 B) 4.4340 us/op 8.0310 us/op 0.55
transfer serialized Small SignedBeaconBlock (128000 B) 2.5130 us/op 2.5170 us/op 1.00
copy serialized Small SignedBeaconBlock (128000 B) 10.375 us/op 15.688 us/op 0.66
transfer serialized Avg SignedBeaconBlock (200000 B) 3.0940 us/op 3.0360 us/op 1.02
copy serialized Avg SignedBeaconBlock (200000 B) 14.123 us/op 22.966 us/op 0.61
transfer serialized BlobsSidecar (524380 B) 3.2110 us/op 3.5780 us/op 0.90
copy serialized BlobsSidecar (524380 B) 75.227 us/op 75.650 us/op 0.99
transfer serialized Big SignedBeaconBlock (1000000 B) 3.3040 us/op 4.2210 us/op 0.78
copy serialized Big SignedBeaconBlock (1000000 B) 149.67 us/op 171.55 us/op 0.87
pass gossip attestations to forkchoice per slot 2.4528 ms/op 3.0034 ms/op 0.82
forkChoice updateHead vc 100000 bc 64 eq 0 403.28 us/op 480.74 us/op 0.84
forkChoice updateHead vc 600000 bc 64 eq 0 2.6059 ms/op 3.8343 ms/op 0.68
forkChoice updateHead vc 1000000 bc 64 eq 0 4.1758 ms/op 5.1756 ms/op 0.81
forkChoice updateHead vc 600000 bc 320 eq 0 2.1373 ms/op 3.0472 ms/op 0.70
forkChoice updateHead vc 600000 bc 1200 eq 0 2.1975 ms/op 2.9577 ms/op 0.74
forkChoice updateHead vc 600000 bc 7200 eq 0 2.7534 ms/op 5.6023 ms/op 0.49
forkChoice updateHead vc 600000 bc 64 eq 1000 9.0637 ms/op 10.512 ms/op 0.86
forkChoice updateHead vc 600000 bc 64 eq 10000 8.9875 ms/op 10.411 ms/op 0.86
forkChoice updateHead vc 600000 bc 64 eq 300000 11.448 ms/op 14.587 ms/op 0.78
computeDeltas 500000 validators 300 proto nodes 3.3323 ms/op 4.2527 ms/op 0.78
computeDeltas 500000 validators 1200 proto nodes 3.2853 ms/op 4.2477 ms/op 0.77
computeDeltas 500000 validators 7200 proto nodes 3.3061 ms/op 4.2624 ms/op 0.78
computeDeltas 750000 validators 300 proto nodes 4.9856 ms/op 6.1556 ms/op 0.81
computeDeltas 750000 validators 1200 proto nodes 4.9044 ms/op 6.1428 ms/op 0.80
computeDeltas 750000 validators 7200 proto nodes 5.0393 ms/op 6.2998 ms/op 0.80
computeDeltas 1400000 validators 300 proto nodes 12.674 ms/op 11.790 ms/op 1.07
computeDeltas 1400000 validators 1200 proto nodes 9.1214 ms/op 11.638 ms/op 0.78
computeDeltas 1400000 validators 7200 proto nodes 9.1220 ms/op 11.521 ms/op 0.79
computeDeltas 2100000 validators 300 proto nodes 13.471 ms/op 16.647 ms/op 0.81
computeDeltas 2100000 validators 1200 proto nodes 13.309 ms/op 16.560 ms/op 0.80
computeDeltas 2100000 validators 7200 proto nodes 13.400 ms/op 16.483 ms/op 0.81
altair processAttestation - 250000 vs - 7PWei normalcase 1.4468 ms/op 1.7019 ms/op 0.85
altair processAttestation - 250000 vs - 7PWei worstcase 2.1489 ms/op 2.5084 ms/op 0.86
altair processAttestation - setStatus - 1/6 committees join 71.504 us/op 84.545 us/op 0.85
altair processAttestation - setStatus - 1/3 committees join 146.04 us/op 171.03 us/op 0.85
altair processAttestation - setStatus - 1/2 committees join 212.65 us/op 238.41 us/op 0.89
altair processAttestation - setStatus - 2/3 committees join 272.36 us/op 308.21 us/op 0.88
altair processAttestation - setStatus - 4/5 committees join 388.77 us/op 454.40 us/op 0.86
altair processAttestation - setStatus - 100% committees join 471.35 us/op 537.41 us/op 0.88
altair processBlock - 250000 vs - 7PWei normalcase 4.2278 ms/op 3.7211 ms/op 1.14
altair processBlock - 250000 vs - 7PWei normalcase hashState 26.152 ms/op 26.016 ms/op 1.01
altair processBlock - 250000 vs - 7PWei worstcase 38.029 ms/op 36.113 ms/op 1.05
altair processBlock - 250000 vs - 7PWei worstcase hashState 83.125 ms/op 65.995 ms/op 1.26
phase0 processBlock - 250000 vs - 7PWei normalcase 1.5843 ms/op 1.6803 ms/op 0.94
phase0 processBlock - 250000 vs - 7PWei worstcase 21.277 ms/op 19.411 ms/op 1.10
altair processEth1Data - 250000 vs - 7PWei normalcase 250.34 us/op 301.64 us/op 0.83
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 6.1260 us/op 4.6350 us/op 1.32
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 27.005 us/op 26.285 us/op 1.03
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 7.5880 us/op 7.9790 us/op 0.95
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 4.8540 us/op 4.7920 us/op 1.01
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 82.382 us/op 111.10 us/op 0.74
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 1.3453 ms/op 1.0840 ms/op 1.24
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 1.2241 ms/op 1.4656 ms/op 0.84
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 1.1171 ms/op 1.5203 ms/op 0.73
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 2.6885 ms/op 3.6401 ms/op 0.74
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 1.2496 ms/op 1.4701 ms/op 0.85
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 2.9547 ms/op 3.4920 ms/op 0.85
Tree 40 250000 create 175.28 ms/op 224.07 ms/op 0.78
Tree 40 250000 get(125000) 113.81 ns/op 147.44 ns/op 0.77
Tree 40 250000 set(125000) 542.52 ns/op 605.79 ns/op 0.90
Tree 40 250000 toArray() 9.8988 ms/op 14.249 ms/op 0.69
Tree 40 250000 iterate all - toArray() + loop 9.9203 ms/op 14.330 ms/op 0.69
Tree 40 250000 iterate all - get(i) 41.422 ms/op 50.218 ms/op 0.82
Array 250000 create 2.2950 ms/op 2.7128 ms/op 0.85
Array 250000 clone - spread 1.2515 ms/op 1.3091 ms/op 0.96
Array 250000 get(125000) 0.58200 ns/op 0.40300 ns/op 1.44
Array 250000 set(125000) 0.59000 ns/op 0.41500 ns/op 1.42
Array 250000 iterate all - loop 78.121 us/op 101.67 us/op 0.77
phase0 afterProcessEpoch - 250000 vs - 7PWei 41.201 ms/op 49.259 ms/op 0.84
Array.fill - length 1000000 2.5364 ms/op 3.4132 ms/op 0.74
Array push - length 1000000 13.992 ms/op 12.149 ms/op 1.15
Array.get 0.25043 ns/op 0.26555 ns/op 0.94
Uint8Array.get 0.32861 ns/op 0.41810 ns/op 0.79
phase0 beforeProcessEpoch - 250000 vs - 7PWei 15.342 ms/op 17.489 ms/op 0.88
altair processEpoch - mainnet_e81889 266.47 ms/op 276.35 ms/op 0.96
mainnet_e81889 - altair beforeProcessEpoch 16.134 ms/op 17.980 ms/op 0.90
mainnet_e81889 - altair processJustificationAndFinalization 8.7810 us/op 12.355 us/op 0.71
mainnet_e81889 - altair processInactivityUpdates 3.9890 ms/op 4.8404 ms/op 0.82
mainnet_e81889 - altair processRewardsAndPenalties 50.591 ms/op 40.039 ms/op 1.26
mainnet_e81889 - altair processRegistryUpdates 1.8730 us/op 1.7880 us/op 1.05
mainnet_e81889 - altair processSlashings 698.00 ns/op 405.00 ns/op 1.72
mainnet_e81889 - altair processEth1DataReset 696.00 ns/op 335.00 ns/op 2.08
mainnet_e81889 - altair processEffectiveBalanceUpdates 978.74 us/op 1.8790 ms/op 0.52
mainnet_e81889 - altair processSlashingsReset 2.3760 us/op 2.5660 us/op 0.93
mainnet_e81889 - altair processRandaoMixesReset 2.5450 us/op 4.0640 us/op 0.63
mainnet_e81889 - altair processHistoricalRootsUpdate 649.00 ns/op 659.00 ns/op 0.98
mainnet_e81889 - altair processParticipationFlagUpdates 1.3080 us/op 1.7330 us/op 0.75
mainnet_e81889 - altair processSyncCommitteeUpdates 638.00 ns/op 427.00 ns/op 1.49
mainnet_e81889 - altair afterProcessEpoch 43.242 ms/op 50.772 ms/op 0.85
capella processEpoch - mainnet_e217614 1.0198 s/op 1.0135 s/op 1.01
mainnet_e217614 - capella beforeProcessEpoch 64.906 ms/op 82.383 ms/op 0.79
mainnet_e217614 - capella processJustificationAndFinalization 6.6810 us/op 15.842 us/op 0.42
mainnet_e217614 - capella processInactivityUpdates 12.226 ms/op 17.543 ms/op 0.70
mainnet_e217614 - capella processRewardsAndPenalties 246.06 ms/op 223.33 ms/op 1.10
mainnet_e217614 - capella processRegistryUpdates 7.6050 us/op 13.418 us/op 0.57
mainnet_e217614 - capella processSlashings 408.00 ns/op 452.00 ns/op 0.90
mainnet_e217614 - capella processEth1DataReset 375.00 ns/op 354.00 ns/op 1.06
mainnet_e217614 - capella processEffectiveBalanceUpdates 10.755 ms/op 14.621 ms/op 0.74
mainnet_e217614 - capella processSlashingsReset 1.4670 us/op 4.1750 us/op 0.35
mainnet_e217614 - capella processRandaoMixesReset 1.9940 us/op 4.7530 us/op 0.42
mainnet_e217614 - capella processHistoricalRootsUpdate 403.00 ns/op 964.00 ns/op 0.42
mainnet_e217614 - capella processParticipationFlagUpdates 812.00 ns/op 2.1010 us/op 0.39
mainnet_e217614 - capella afterProcessEpoch 101.20 ms/op 127.70 ms/op 0.79
phase0 processEpoch - mainnet_e58758 296.84 ms/op 326.50 ms/op 0.91
mainnet_e58758 - phase0 beforeProcessEpoch 73.219 ms/op 77.405 ms/op 0.95
mainnet_e58758 - phase0 processJustificationAndFinalization 12.130 us/op 16.775 us/op 0.72
mainnet_e58758 - phase0 processRewardsAndPenalties 27.423 ms/op 28.938 ms/op 0.95
mainnet_e58758 - phase0 processRegistryUpdates 4.1390 us/op 9.8060 us/op 0.42
mainnet_e58758 - phase0 processSlashings 690.00 ns/op 466.00 ns/op 1.48
mainnet_e58758 - phase0 processEth1DataReset 644.00 ns/op 366.00 ns/op 1.76
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.3419 ms/op 1.2291 ms/op 1.09
mainnet_e58758 - phase0 processSlashingsReset 2.2240 us/op 2.5480 us/op 0.87
mainnet_e58758 - phase0 processRandaoMixesReset 2.8640 us/op 3.7430 us/op 0.77
mainnet_e58758 - phase0 processHistoricalRootsUpdate 679.00 ns/op 376.00 ns/op 1.81
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.8950 us/op 3.1270 us/op 1.25
mainnet_e58758 - phase0 afterProcessEpoch 35.842 ms/op 43.937 ms/op 0.82
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.0022 ms/op 1.9365 ms/op 0.52
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.8979 ms/op 2.2963 ms/op 0.83
altair processInactivityUpdates - 250000 normalcase 15.494 ms/op 14.566 ms/op 1.06
altair processInactivityUpdates - 250000 worstcase 16.152 ms/op 16.141 ms/op 1.00
phase0 processRegistryUpdates - 250000 normalcase 3.1840 us/op 7.0250 us/op 0.45
phase0 processRegistryUpdates - 250000 badcase_full_deposits 297.07 us/op 312.36 us/op 0.95
phase0 processRegistryUpdates - 250000 worstcase 0.5 116.17 ms/op 127.93 ms/op 0.91
altair processRewardsAndPenalties - 250000 normalcase 45.970 ms/op 37.573 ms/op 1.22
altair processRewardsAndPenalties - 250000 worstcase 50.386 ms/op 35.955 ms/op 1.40
phase0 getAttestationDeltas - 250000 normalcase 5.7497 ms/op 7.7508 ms/op 0.74
phase0 getAttestationDeltas - 250000 worstcase 5.6861 ms/op 7.4666 ms/op 0.76
phase0 processSlashings - 250000 worstcase 86.765 us/op 89.296 us/op 0.97
altair processSyncCommitteeUpdates - 250000 94.751 ms/op 124.33 ms/op 0.76
BeaconState.hashTreeRoot - No change 457.00 ns/op 267.00 ns/op 1.71
BeaconState.hashTreeRoot - 1 full validator 138.02 us/op 93.586 us/op 1.47
BeaconState.hashTreeRoot - 32 full validator 1.1320 ms/op 1.1361 ms/op 1.00
BeaconState.hashTreeRoot - 512 full validator 9.3452 ms/op 10.097 ms/op 0.93
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 164.93 us/op 114.49 us/op 1.44
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.9356 ms/op 1.4589 ms/op 1.33
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 20.836 ms/op 20.835 ms/op 1.00
BeaconState.hashTreeRoot - 1 balances 96.198 us/op 93.184 us/op 1.03
BeaconState.hashTreeRoot - 32 balances 1.0287 ms/op 840.87 us/op 1.22
BeaconState.hashTreeRoot - 512 balances 7.4150 ms/op 7.6803 ms/op 0.97
BeaconState.hashTreeRoot - 250000 balances 169.20 ms/op 153.77 ms/op 1.10
aggregationBits - 2048 els - zipIndexesInBitList 19.371 us/op 26.247 us/op 0.74
byteArrayEquals 32 48.745 ns/op 54.540 ns/op 0.89
Buffer.compare 32 16.901 ns/op 17.373 ns/op 0.97
byteArrayEquals 1024 1.2188 us/op 1.6156 us/op 0.75
Buffer.compare 1024 22.929 ns/op 24.840 ns/op 0.92
byteArrayEquals 16384 19.344 us/op 25.658 us/op 0.75
Buffer.compare 16384 167.19 ns/op 194.48 ns/op 0.86
byteArrayEquals 123687377 146.39 ms/op 194.38 ms/op 0.75
Buffer.compare 123687377 5.2876 ms/op 7.5644 ms/op 0.70
byteArrayEquals 32 - diff last byte 45.527 ns/op 53.624 ns/op 0.85
Buffer.compare 32 - diff last byte 15.264 ns/op 17.465 ns/op 0.87
byteArrayEquals 1024 - diff last byte 1.2234 us/op 1.6308 us/op 0.75
Buffer.compare 1024 - diff last byte 21.867 ns/op 25.450 ns/op 0.86
byteArrayEquals 16384 - diff last byte 19.441 us/op 25.829 us/op 0.75
Buffer.compare 16384 - diff last byte 161.66 ns/op 202.71 ns/op 0.80
byteArrayEquals 123687377 - diff last byte 146.97 ms/op 189.78 ms/op 0.77
Buffer.compare 123687377 - diff last byte 5.3262 ms/op 6.1649 ms/op 0.86
byteArrayEquals 32 - random bytes 4.7690 ns/op 5.0800 ns/op 0.94
Buffer.compare 32 - random bytes 15.602 ns/op 17.132 ns/op 0.91
byteArrayEquals 1024 - random bytes 4.8010 ns/op 5.1070 ns/op 0.94
Buffer.compare 1024 - random bytes 15.142 ns/op 17.092 ns/op 0.89
byteArrayEquals 16384 - random bytes 5.6780 ns/op 5.0930 ns/op 1.11
Buffer.compare 16384 - random bytes 15.125 ns/op 17.412 ns/op 0.87
byteArrayEquals 123687377 - random bytes 7.6000 ns/op 6.4500 ns/op 1.18
Buffer.compare 123687377 - random bytes 17.970 ns/op 18.490 ns/op 0.97
regular array get 100000 times 29.960 us/op 34.731 us/op 0.86
wrappedArray get 100000 times 29.955 us/op 33.183 us/op 0.90
arrayWithProxy get 100000 times 9.3040 ms/op 13.341 ms/op 0.70
ssz.Root.equals 43.064 ns/op 45.720 ns/op 0.94
byteArrayEquals 38.231 ns/op 45.116 ns/op 0.85
Buffer.compare 8.8490 ns/op 10.330 ns/op 0.86
processSlot - 1 slots 14.832 us/op 11.679 us/op 1.27
processSlot - 32 slots 2.6877 ms/op 2.1561 ms/op 1.25
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 44.211 ms/op 36.599 ms/op 1.21
getCommitteeAssignments - req 1 vs - 250000 vc 1.8445 ms/op 2.1613 ms/op 0.85
getCommitteeAssignments - req 100 vs - 250000 vc 3.6041 ms/op 4.2952 ms/op 0.84
getCommitteeAssignments - req 1000 vs - 250000 vc 3.8663 ms/op 4.4900 ms/op 0.86
findModifiedValidators - 10000 modified validators 248.35 ms/op 258.03 ms/op 0.96
findModifiedValidators - 1000 modified validators 138.43 ms/op 176.68 ms/op 0.78
findModifiedValidators - 100 modified validators 128.68 ms/op 160.38 ms/op 0.80
findModifiedValidators - 10 modified validators 128.13 ms/op 152.13 ms/op 0.84
findModifiedValidators - 1 modified validators 122.81 ms/op 153.14 ms/op 0.80
findModifiedValidators - no difference 132.97 ms/op 167.39 ms/op 0.79
compare ViewDUs 3.0359 s/op 3.3091 s/op 0.92
compare each validator Uint8Array 1.5803 s/op 1.4704 s/op 1.07
compare ViewDU to Uint8Array 668.25 ms/op 1.2361 s/op 0.54
migrate state 1000000 validators, 24 modified, 0 new 690.76 ms/op 757.73 ms/op 0.91
migrate state 1000000 validators, 1700 modified, 1000 new 938.95 ms/op 923.05 ms/op 1.02
migrate state 1000000 validators, 3400 modified, 2000 new 1.1521 s/op 1.1167 s/op 1.03
migrate state 1500000 validators, 24 modified, 0 new 740.72 ms/op 750.72 ms/op 0.99
migrate state 1500000 validators, 1700 modified, 1000 new 927.57 ms/op 994.39 ms/op 0.93
migrate state 1500000 validators, 3400 modified, 2000 new 1.1518 s/op 1.1480 s/op 1.00
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 6.0700 ns/op 4.3200 ns/op 1.41
state getBlockRootAtSlot - 250000 vs - 7PWei 748.83 ns/op 449.20 ns/op 1.67
computeProposers - vc 250000 6.7343 ms/op 7.8054 ms/op 0.86
computeEpochShuffling - vc 250000 33.791 ms/op 42.055 ms/op 0.80
getNextSyncCommittee - vc 250000 116.43 ms/op 124.04 ms/op 0.94
computeSigningRoot for AttestationData 27.905 us/op 16.864 us/op 1.65
hash AttestationData serialized data then Buffer.toString(base64) 1.1353 us/op 1.5935 us/op 0.71
toHexString serialized data 723.47 ns/op 904.86 ns/op 0.80
Buffer.toString(base64) 134.99 ns/op 185.40 ns/op 0.73
nodejs block root to RootHex using toHex 109.17 ns/op 150.48 ns/op 0.73
nodejs block root to RootHex using toRootHex 68.458 ns/op 95.790 ns/op 0.71
browser block root to RootHex using the deprecated toHexString 196.26 ns/op 228.65 ns/op 0.86
browser block root to RootHex using toHex 158.35 ns/op 183.14 ns/op 0.86
browser block root to RootHex using toRootHex 140.22 ns/op 163.18 ns/op 0.86

by benchmarkbot/action

@ensi321 ensi321 mentioned this pull request Oct 23, 2024
Open
12 tasks
@g11tech
hookup the getblobs api to get bob and proof data from el
c5a0c76 
remove unused

fix import

metrics overhault, test, debugging testing, some feeback

fix

add nethermind bug dicussion link

fix

resolve conflicts
@g11tech g11tech merged commit 2e5265b into unstable Nov 3, 2024
23 checks passed
@g11tech g11tech deleted the getblobsv1 branch November 3, 2024 09:35
@twoeths
Copy link
Contributor

twoeths commented Nov 6, 2024

mesh peers on the unstable mainnet node dropped consistently since this PR merged

Screenshot 2024-11-06 at 10 17 05

same to blob_sidecar topic, it does not happen on holesky through

twoeths added a commit that referenced this pull request Dec 13, 2024
@twoeths
Revert "feat: add and use getBlobsV1 to expedite gossip import (#7134)"
1247d2e 
This reverts commit 2e5265b.
This was referenced Dec 13, 2024
Open
Closed
wemeetagain added a commit that referenced this pull request Dec 20, 2024
@wemeetagain @nflaig
@nazarhussain @g11tech @matthewkeil @ensi321 @twoeths @philknows @dependabot @acolytec3 @varunguleriaCodes @ClockworkYuzu
chore: release v1.24.0 (#7325)
11f35a8 
* feat: add keymanager endpoint to retrieve proposer config (#7210)

* feat: add keymanager endpoint to retrieve proposer config

* Do not return empty builder config

* Check all builder proposer config values

* Fix settings builder config if undefined

* Fix builder config parsing

* Use ssz type to handle json serialization

Default parsing can't handle BigInt

* Revert "Use ssz type to handle json serialization"

This reverts commit 01fcea7.

* Fix boost factor json serialization

* Remove unused import

* Update test data

* Update proposer config test

* feat: add mekong network option (#7212)

* chore: fix import order with biome syntax (#7211)

Fix import order

* fix: consistently validate pubkey and throw 404 if not found (#7214)

* Throw error if pubkey is unknown when getting graffiti

* Consistently validate pubkey and throw 404 if not found

* fix: only return local keys from /eth/v1/keystores (#7215)

* fix: only return local keys from /eth/v1/keystores

* Fix fetching remote keys in node assertion

* feat: add and use getBlobsV1 to expedite gossip import (#7134)

* hookup the getblobs api to get bob and proof data from el

remove unused

fix import

metrics overhault, test, debugging testing, some feeback

fix

add nethermind bug dicussion link

fix

resolve conflicts

* deblobs timeout

* fix metric

* chore: revert async aggregate with randomness (#7218)

Revert "feat: asyncAggregateWithRandomness (#7204)"

This reverts commit e31d535.

* fix: update config for relaunched mekong network (#7220)

* fix: light client generating `LightClientUpdate` with wrong length of branches (#7187)

* initial commit

* Rewrite SyncCommitteeWitnessRepository

* Fix finality branch

* Update unit test

* fix e2e

* Review PR

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* fix: archive finalized state when shutting down beacon node (#7221)

* Fix typo

* feat: remove unfinalized pubkey cache (#7230)

* Remove unfinalized pubkey cache

* lint

* Fix unit test

* chore: skip web3_provider unit tests (#7252)

* fix: prune checkpoint states at syncing time (#7241)

* fix: prune checkpoint states at syncing time

* fix: lint

* fix: check-types in test

* fix: sync cached isCompoundingValidatorArr at epoch transition (#7247)

* fix: handle outOfRangeData when range sync Deneb (#7249)

* fix: handle outOfRangeData for beaconBlocksMaybeBlobsByRange()

* fix: lint

* fix: archiveBlocks - handle deneb outOfRangeData block

* fix: sync cached balance when adding new validator to registry (#7255)

* fix: sync cached balance when adding new validator to registry

* chore: add more comments

* fix: remove persisted checkpoint states from the previous run at startup

* fix: do not throw error when trying to prune missing directory (#7257)

* docs: update documentation Oct 2024 (#7178)

* docs update oct 2024 init

* Reconfig quickstart nav and minor fixes

* fix lint

* spelling fixes

* minor fixes and add to wordlist

* prettier fix

* add to wordlist

* sort wordlist

* modify dominance to include lighthouse

* fix typescript casing and add recommendation

* add selection and boost_factor with keymanager notice

* update wordlist

* remove builder enabled and add keymanager api

* spelling

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* chore(deps): bump cross-spawn from 7.0.3 to 7.0.6 in /docs (#7268)

Bumps [cross-spawn](https://github.com/moxystudio/node-cross-spawn) from 7.0.3 to 7.0.6.
- [Changelog](https://github.com/moxystudio/node-cross-spawn/blob/master/CHANGELOG.md)
- [Commits](moxystudio/node-cross-spawn@v7.0.3...v7.0.6)

---
updated-dependencies:
- dependency-name: cross-spawn
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: add error log to notifier if execution client auth failed (#7239)

* feat: add error log to notifier if execution client auth failed

* Update packages/beacon-node/src/node/notifier.ts

---------

Co-authored-by: NC <17676176+ensi321@users.noreply.github.com>

* docs: display rcConfig flag on CLI reference page (#7270)

* docs: display rcConfig flag on CLI reference page

* Update word list

* chore: remove prettier as default formatter for all file types (#7275)

* chore: unhide flags relevant for devnets / testing (#7271)

* feat: debug too many shuffling promises (#7251)

* feat: add asyncShufflingCalculation to StateTransitionOpts

* feat: add asyncShufflingCalculation to all regen / processSlots consumers

* fix: default to false for async shuffling and remove unnecessary props

* fix: remove unnecessary flags from stateTransition

* feat: implement conditional build of shuffling for prepareNextSlot

* fix: spec test bug where shufflingCache is present from BeaconChain constructor

* feat: sync build next shuffling if not queued async

* fix: use getSync to pull next shuffling correctly

* docs: add comment to prepareNextSlot

* refactor: rename StateCloneOpts to StateRegenerationOpts

* feat: pass asyncShufflingCalculation through to afterProcessEpoch and refactor conditional to run purely sync

* docs: add issue number to comment

* chore: lint

* chore: unpin nodejs version from 22.4 (#6982)

* Revert "chore: pin nodejs version to 22.4 (#6964)"

This reverts commit f20484b.

* Don't revert formatting changes

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>
Co-authored-by: Cayman <caymannava@gmail.com>

* chore: update bootnodes file url for holesky and sepolia (#7276)

* feat: add `debug_getHistoricalSummaries` endpoint (#7245)

* feat: add new getHistoricalSummaries endpoint to debug namespace

* Add JSON response

* Restructure to use stateId and add proof to response

* add test scaffolding

* Address feedback

* Move getHistoricalSummaries to lodestar namespace

* add lodestar namespace unit test

* update route name to lodestar namespace

* cast state object as Capella state

* Lint

* json properties need to be lower case

* Make it v1 since it's now part of lodestar namespace

* Group with other /lodestar endpoints

* Simplify beacon node impl

* Rename return type

* Update test description

* Fix variable name

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* chore: log sync committee signature errors as `error` (#7283)

* fix: update engine_getClientVersionV1 commit encoding (#7282)

* fix: check pubkey or validator index known to a state (#7284)

* fix: check pubkey or validator index known to a state

* chore: add more comments

* feat: lodestar script setup (#7254)

* feat: lodestar_setup

* feat: script_updates + docs

* feat: script_addition_in_docs + command_update

* Remove duplicate script from docs folder

* Minor script updates

* Update script to prepare docs and ignore copied file

* Update installation page

* Wording

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* feat: add terminal-sized Electra giraffe banner (#7286)

* Create giraffeBanners.ts

* Wire in banner

* Fix file name

* lint

* Address @nflaig's comment

---------

Co-authored-by: NC <17676176+ensi321@users.noreply.github.com>

* chore: pin nodejs version to 22.4 (#7291)

Revert "chore: unpin nodejs version from 22.4 (#6982)"

This reverts commit 69ae688.

* feat: expose `DOMAIN_APPLICATION_MASK` in config/spec api (#7296)

* feat: expose DOMAIN_APPLICATION_MASK in config/spec api

* Lint

* feat: make `MAX_REQUEST_BLOB_SIDECARS` and `MAX_BLOBS_PER_BLOCK` configurable (#7294)

* Init commit

* Fix check-types

* Add comment on how MAX_REQUEST_BLOB_SIDECARS is calculated

* Ensure proper config object is passed

* Address comment

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* feat: use `BLOB_SIDECAR_SUBNET_COUNT` to configure blob subnets (#7297)

feat: use BLOB_SIDECAR_SUBNET_COUNT to configure blob subnets

* chore: log sync aggregate participants when producing beacon block body (#7300)

* chore: log sync aggregate participants when producing beacon block body

* Use isForkLightClient instead of ForkSeq

* Fix produce block unit tests

* chore: print graffiti when producing beacon block body (#7303)

* fix: warn if engine / builder failed to produce block within cutoff time (#7305)

* feat: add kzg commitment length check when validating gossip blocks (#7302)

* feat: add blob sidecar index check (#7313)

Validate blobSidecar index

* fix: fix blob sidecar index check (#7315)

Fix index check

* chore: fix format of printed graffiti from hex to utf-8 (#7306)

* chore: fix format of printed graffiti from hex to utf-8

* Use Buffer.from no copy with offset

* docs: batch commit typos and update contributor readme (#7312)

* batch commit typos and update contributor readme

* update donation text

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* correct spelling

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* chore: remove trailing null bytes from printed graffiti (#7320)

* chore: remove trailing null bytes from printed graffiti

* Use replaceAll instead of regex

* chore: unpin nodejs version from 22.4 (#7324)

Revert "chore: pin nodejs version to 22.4 (#7291)"

This reverts commit 99794d3.

* chore: bump package versions to 1.24.0

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
Co-authored-by: Nazar Hussain <nazarhussain@gmail.com>
Co-authored-by: g11tech <develop@g11tech.io>
Co-authored-by: Matthew Keil <me@matthewkeil.com>
Co-authored-by: NC <17676176+ensi321@users.noreply.github.com>
Co-authored-by: twoeths <tuyen@chainsafe.io>
Co-authored-by: Phil Ngo <58080811+philknows@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com>
Co-authored-by: Varun Guleria <152203177+varunguleriaCodes@users.noreply.github.com>
Co-authored-by: ClockworkYuzu <yvetteheh@gmail.com>
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.24.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nflaig nflaig nflaig left review comments

@twoeths twoeths twoeths left review comments

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v1.23.0
Development

Successfully merging this pull request may close these issues.

Implement engine_getBlobsV1
5 participants
@g11tech @twoeths @wemeetagain @nflaig @philknows