Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: introduce beacon api test ignore list #6171

Merged
merged 27 commits into from
Dec 19, 2023

Conversation

jeluard
Copy link
Contributor

@jeluard jeluard commented Dec 8, 2023

Motivation

Make sure lodestar supports latest Beacon APIs.

Description

To allow for easier testing against latest Beacon API, a black-list of not-yet supported features/examples is introduced.

Relates to #5710
Builds on top of unstable...nflaig/update-beacon-spec-version

Steps to test or reproduce

All unit tests in packages/api should pass.

@jeluard jeluard self-assigned this Dec 8, 2023
@jeluard jeluard changed the title Jeluard/beacon api ignore list feature: introduce beacon api ignore list Dec 8, 2023
@jeluard jeluard changed the title feature: introduce beacon api ignore list feature: introduce beacon api test ignore list Dec 8, 2023
@jeluard jeluard changed the title feature: introduce beacon api test ignore list feat: introduce beacon api test ignore list Dec 8, 2023
Copy link

codecov bot commented Dec 8, 2023

Codecov Report

Merging #6171 (25e8aa5) into unstable (4b9647f) will increase coverage by 90.35%.
Report is 30 commits behind head on unstable.
The diff coverage is n/a.

Additional details and impacted files
@@              Coverage Diff              @@
##           unstable    #6171       +/-   ##
=============================================
+ Coverage          0   90.35%   +90.35%     
=============================================
  Files             0       78       +78     
  Lines             0     8087     +8087     
  Branches          0      490      +490     
=============================================
+ Hits              0     7307     +7307     
- Misses            0      772      +772     
- Partials          0        8        +8     

@jeluard jeluard changed the title feat: introduce beacon api test ignore list test: introduce beacon api test ignore list Dec 8, 2023
Copy link
Contributor

github-actions bot commented Dec 8, 2023

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 47d0dea Previous: d3005bf Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 590.15 us/op 617.22 us/op 0.96
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 118.00 us/op 84.778 us/op 1.39
BLS verify - blst-native 1.3700 ms/op 1.3312 ms/op 1.03
BLS verifyMultipleSignatures 3 - blst-native 2.8670 ms/op 2.8139 ms/op 1.02
BLS verifyMultipleSignatures 8 - blst-native 6.2722 ms/op 6.1753 ms/op 1.02
BLS verifyMultipleSignatures 32 - blst-native 23.039 ms/op 22.547 ms/op 1.02
BLS verifyMultipleSignatures 64 - blst-native 45.338 ms/op 44.476 ms/op 1.02
BLS verifyMultipleSignatures 128 - blst-native 89.623 ms/op 88.753 ms/op 1.01
BLS deserializing 10000 signatures 995.96 ms/op 920.68 ms/op 1.08
BLS deserializing 100000 signatures 9.2069 s/op 9.6734 s/op 0.95
BLS verifyMultipleSignatures - same message - 3 - blst-native 1.3875 ms/op 1.3640 ms/op 1.02
BLS verifyMultipleSignatures - same message - 8 - blst-native 1.5756 ms/op 1.6998 ms/op 0.93
BLS verifyMultipleSignatures - same message - 32 - blst-native 2.4116 ms/op 2.9494 ms/op 0.82
BLS verifyMultipleSignatures - same message - 64 - blst-native 3.6016 ms/op 4.2028 ms/op 0.86
BLS verifyMultipleSignatures - same message - 128 - blst-native 5.8346 ms/op 7.2895 ms/op 0.80
BLS aggregatePubkeys 32 - blst-native 26.945 us/op 27.930 us/op 0.96
BLS aggregatePubkeys 128 - blst-native 105.13 us/op 101.14 us/op 1.04
getAttestationsForBlock 53.471 ms/op 53.125 ms/op 1.01
getSlashingsAndExits - default max 198.91 us/op 159.65 us/op 1.25
getSlashingsAndExits - 2k 559.37 us/op 423.24 us/op 1.32
proposeBlockBody type=full, size=empty 5.6660 ms/op 5.4065 ms/op 1.05
isKnown best case - 1 super set check 327.00 ns/op 328.00 ns/op 1.00
isKnown normal case - 2 super set checks 308.00 ns/op 309.00 ns/op 1.00
isKnown worse case - 16 super set checks 303.00 ns/op 312.00 ns/op 0.97
CheckpointStateCache - add get delete 5.2160 us/op 5.6810 us/op 0.92
validate api signedAggregateAndProof - struct 2.8068 ms/op 2.7903 ms/op 1.01
validate gossip signedAggregateAndProof - struct 2.8203 ms/op 2.8286 ms/op 1.00
validate gossip attestation - vc 640000 1.3330 ms/op 1.3615 ms/op 0.98
batch validate gossip attestation - vc 640000 - chunk 32 165.43 us/op 164.73 us/op 1.00
batch validate gossip attestation - vc 640000 - chunk 64 150.60 us/op 144.69 us/op 1.04
batch validate gossip attestation - vc 640000 - chunk 128 146.38 us/op 136.49 us/op 1.07
batch validate gossip attestation - vc 640000 - chunk 256 142.51 us/op 138.43 us/op 1.03
pickEth1Vote - no votes 1.3582 ms/op 1.1773 ms/op 1.15
pickEth1Vote - max votes 8.7582 ms/op 10.561 ms/op 0.83
pickEth1Vote - Eth1Data hashTreeRoot value x2048 17.385 ms/op 21.590 ms/op 0.81
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 30.997 ms/op 29.080 ms/op 1.07
pickEth1Vote - Eth1Data fastSerialize value x2048 635.44 us/op 613.04 us/op 1.04
pickEth1Vote - Eth1Data fastSerialize tree x2048 4.8094 ms/op 4.5670 ms/op 1.05
bytes32 toHexString 615.00 ns/op 506.00 ns/op 1.22
bytes32 Buffer.toString(hex) 336.00 ns/op 293.00 ns/op 1.15
bytes32 Buffer.toString(hex) from Uint8Array 585.00 ns/op 451.00 ns/op 1.30
bytes32 Buffer.toString(hex) + 0x 312.00 ns/op 285.00 ns/op 1.09
Object access 1 prop 0.18900 ns/op 0.15900 ns/op 1.19
Map access 1 prop 0.16300 ns/op 0.15000 ns/op 1.09
Object get x1000 7.3840 ns/op 7.4470 ns/op 0.99
Map get x1000 0.86200 ns/op 0.78500 ns/op 1.10
Object set x1000 72.038 ns/op 52.329 ns/op 1.38
Map set x1000 53.667 ns/op 40.201 ns/op 1.33
Return object 10000 times 0.27090 ns/op 0.24360 ns/op 1.11
Throw Error 10000 times 4.0567 us/op 3.8881 us/op 1.04
fastMsgIdFn sha256 / 200 bytes 3.4460 us/op 3.3250 us/op 1.04
fastMsgIdFn h32 xxhash / 200 bytes 326.00 ns/op 278.00 ns/op 1.17
fastMsgIdFn h64 xxhash / 200 bytes 382.00 ns/op 338.00 ns/op 1.13
fastMsgIdFn sha256 / 1000 bytes 11.666 us/op 11.364 us/op 1.03
fastMsgIdFn h32 xxhash / 1000 bytes 495.00 ns/op 409.00 ns/op 1.21
fastMsgIdFn h64 xxhash / 1000 bytes 490.00 ns/op 408.00 ns/op 1.20
fastMsgIdFn sha256 / 10000 bytes 108.79 us/op 106.47 us/op 1.02
fastMsgIdFn h32 xxhash / 10000 bytes 2.0800 us/op 2.0190 us/op 1.03
fastMsgIdFn h64 xxhash / 10000 bytes 1.4320 us/op 1.3740 us/op 1.04
send data - 1000 256B messages 21.845 ms/op 19.153 ms/op 1.14
send data - 1000 512B messages 28.245 ms/op 25.787 ms/op 1.10
send data - 1000 1024B messages 43.014 ms/op 41.459 ms/op 1.04
send data - 1000 1200B messages 37.337 ms/op 36.869 ms/op 1.01
send data - 1000 2048B messages 44.460 ms/op 43.427 ms/op 1.02
send data - 1000 4096B messages 40.672 ms/op 40.855 ms/op 1.00
send data - 1000 16384B messages 122.87 ms/op 110.65 ms/op 1.11
send data - 1000 65536B messages 509.51 ms/op 393.80 ms/op 1.29
enrSubnets - fastDeserialize 64 bits 1.2730 us/op 1.3990 us/op 0.91
enrSubnets - ssz BitVector 64 bits 424.00 ns/op 436.00 ns/op 0.97
enrSubnets - fastDeserialize 4 bits 172.00 ns/op 178.00 ns/op 0.97
enrSubnets - ssz BitVector 4 bits 423.00 ns/op 498.00 ns/op 0.85
prioritizePeers score -10:0 att 32-0.1 sync 2-0 102.40 us/op 112.36 us/op 0.91
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 129.88 us/op 138.33 us/op 0.94
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 172.70 us/op 210.82 us/op 0.82
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 313.69 us/op 328.38 us/op 0.96
prioritizePeers score 0:0 att 64-1 sync 4-1 350.62 us/op 363.63 us/op 0.96
array of 16000 items push then shift 1.6525 us/op 1.6982 us/op 0.97
LinkedList of 16000 items push then shift 8.7300 ns/op 9.2030 ns/op 0.95
array of 16000 items push then pop 94.862 ns/op 92.952 ns/op 1.02
LinkedList of 16000 items push then pop 9.0730 ns/op 9.1290 ns/op 0.99
array of 24000 items push then shift 2.6356 us/op 2.5526 us/op 1.03
LinkedList of 24000 items push then shift 9.2230 ns/op 9.1850 ns/op 1.00
array of 24000 items push then pop 129.48 ns/op 139.72 ns/op 0.93
LinkedList of 24000 items push then pop 8.8890 ns/op 9.3410 ns/op 0.95
intersect bitArray bitLen 8 6.6480 ns/op 6.5000 ns/op 1.02
intersect array and set length 8 68.698 ns/op 64.549 ns/op 1.06
intersect bitArray bitLen 128 34.780 ns/op 34.527 ns/op 1.01
intersect array and set length 128 920.57 ns/op 913.79 ns/op 1.01
bitArray.getTrueBitIndexes() bitLen 128 1.5490 us/op 1.5750 us/op 0.98
bitArray.getTrueBitIndexes() bitLen 248 2.4810 us/op 2.6710 us/op 0.93
bitArray.getTrueBitIndexes() bitLen 512 4.8970 us/op 5.0820 us/op 0.96
Buffer.concat 32 items 964.00 ns/op 995.00 ns/op 0.97
Uint8Array.set 32 items 1.9120 us/op 1.9390 us/op 0.99
Set add up to 64 items then delete first 4.3796 us/op 4.3020 us/op 1.02
OrderedSet add up to 64 items then delete first 5.5434 us/op 5.4105 us/op 1.02
Set add up to 64 items then delete last 4.7079 us/op 4.5840 us/op 1.03
OrderedSet add up to 64 items then delete last 5.8151 us/op 5.7861 us/op 1.01
Set add up to 64 items then delete middle 4.6624 us/op 4.6169 us/op 1.01
OrderedSet add up to 64 items then delete middle 7.2989 us/op 7.0188 us/op 1.04
Set add up to 128 items then delete first 10.093 us/op 9.3565 us/op 1.08
OrderedSet add up to 128 items then delete first 12.785 us/op 12.271 us/op 1.04
Set add up to 128 items then delete last 9.2485 us/op 9.2541 us/op 1.00
OrderedSet add up to 128 items then delete last 11.783 us/op 11.698 us/op 1.01
Set add up to 128 items then delete middle 10.304 us/op 9.1948 us/op 1.12
OrderedSet add up to 128 items then delete middle 17.803 us/op 16.720 us/op 1.06
Set add up to 256 items then delete first 20.648 us/op 18.887 us/op 1.09
OrderedSet add up to 256 items then delete first 26.496 us/op 25.468 us/op 1.04
Set add up to 256 items then delete last 18.658 us/op 18.273 us/op 1.02
OrderedSet add up to 256 items then delete last 22.950 us/op 23.172 us/op 0.99
Set add up to 256 items then delete middle 17.700 us/op 18.396 us/op 0.96
OrderedSet add up to 256 items then delete middle 43.832 us/op 44.760 us/op 0.98
transfer serialized Status (84 B) 1.7130 us/op 1.7380 us/op 0.99
copy serialized Status (84 B) 1.4190 us/op 1.5060 us/op 0.94
transfer serialized SignedVoluntaryExit (112 B) 1.8100 us/op 1.9820 us/op 0.91
copy serialized SignedVoluntaryExit (112 B) 1.4770 us/op 1.5850 us/op 0.93
transfer serialized ProposerSlashing (416 B) 2.3010 us/op 2.2780 us/op 1.01
copy serialized ProposerSlashing (416 B) 1.8200 us/op 3.1300 us/op 0.58
transfer serialized Attestation (485 B) 2.0590 us/op 3.2250 us/op 0.64
copy serialized Attestation (485 B) 1.9240 us/op 3.1250 us/op 0.62
transfer serialized AttesterSlashing (33232 B) 2.1630 us/op 3.2180 us/op 0.67
copy serialized AttesterSlashing (33232 B) 5.3740 us/op 6.6130 us/op 0.81
transfer serialized Small SignedBeaconBlock (128000 B) 2.6590 us/op 3.0560 us/op 0.87
copy serialized Small SignedBeaconBlock (128000 B) 14.810 us/op 17.824 us/op 0.83
transfer serialized Avg SignedBeaconBlock (200000 B) 2.9960 us/op 3.1870 us/op 0.94
copy serialized Avg SignedBeaconBlock (200000 B) 23.249 us/op 21.438 us/op 1.08
transfer serialized BlobsSidecar (524380 B) 2.9180 us/op 3.1260 us/op 0.93
copy serialized BlobsSidecar (524380 B) 83.456 us/op 97.942 us/op 0.85
transfer serialized Big SignedBeaconBlock (1000000 B) 3.1030 us/op 3.2070 us/op 0.97
copy serialized Big SignedBeaconBlock (1000000 B) 146.11 us/op 219.30 us/op 0.67
pass gossip attestations to forkchoice per slot 4.3316 ms/op 4.2704 ms/op 1.01
forkChoice updateHead vc 100000 bc 64 eq 0 704.77 us/op 686.62 us/op 1.03
forkChoice updateHead vc 600000 bc 64 eq 0 4.3342 ms/op 4.6733 ms/op 0.93
forkChoice updateHead vc 1000000 bc 64 eq 0 7.3571 ms/op 7.1046 ms/op 1.04
forkChoice updateHead vc 600000 bc 320 eq 0 4.3317 ms/op 4.2697 ms/op 1.01
forkChoice updateHead vc 600000 bc 1200 eq 0 4.4014 ms/op 4.3061 ms/op 1.02
forkChoice updateHead vc 600000 bc 7200 eq 0 5.3573 ms/op 5.5790 ms/op 0.96
forkChoice updateHead vc 600000 bc 64 eq 1000 11.434 ms/op 11.315 ms/op 1.01
forkChoice updateHead vc 600000 bc 64 eq 10000 12.164 ms/op 12.085 ms/op 1.01
forkChoice updateHead vc 600000 bc 64 eq 300000 16.196 ms/op 16.275 ms/op 1.00
computeDeltas 500000 validators 300 proto nodes 6.7317 ms/op 6.8184 ms/op 0.99
computeDeltas 500000 validators 1200 proto nodes 6.6955 ms/op 6.6462 ms/op 1.01
computeDeltas 500000 validators 7200 proto nodes 6.6508 ms/op 6.4632 ms/op 1.03
computeDeltas 750000 validators 300 proto nodes 9.7646 ms/op 9.7178 ms/op 1.00
computeDeltas 750000 validators 1200 proto nodes 9.9305 ms/op 9.6766 ms/op 1.03
computeDeltas 750000 validators 7200 proto nodes 9.9080 ms/op 9.9186 ms/op 1.00
computeDeltas 1400000 validators 300 proto nodes 19.766 ms/op 18.761 ms/op 1.05
computeDeltas 1400000 validators 1200 proto nodes 19.328 ms/op 19.513 ms/op 0.99
computeDeltas 1400000 validators 7200 proto nodes 19.228 ms/op 19.919 ms/op 0.97
computeDeltas 2100000 validators 300 proto nodes 30.010 ms/op 29.037 ms/op 1.03
computeDeltas 2100000 validators 1200 proto nodes 29.950 ms/op 29.040 ms/op 1.03
computeDeltas 2100000 validators 7200 proto nodes 29.286 ms/op 29.548 ms/op 0.99
computeProposerBoostScoreFromBalances 500000 validators 3.8973 ms/op 3.8326 ms/op 1.02
computeProposerBoostScoreFromBalances 750000 validators 3.9095 ms/op 3.8195 ms/op 1.02
computeProposerBoostScoreFromBalances 1400000 validators 3.9518 ms/op 3.8045 ms/op 1.04
computeProposerBoostScoreFromBalances 2100000 validators 3.9201 ms/op 3.7577 ms/op 1.04
altair processAttestation - 250000 vs - 7PWei normalcase 3.0764 ms/op 2.2627 ms/op 1.36
altair processAttestation - 250000 vs - 7PWei worstcase 3.4710 ms/op 3.4085 ms/op 1.02
altair processAttestation - setStatus - 1/6 committees join 207.79 us/op 144.29 us/op 1.44
altair processAttestation - setStatus - 1/3 committees join 409.93 us/op 294.06 us/op 1.39
altair processAttestation - setStatus - 1/2 committees join 516.93 us/op 383.54 us/op 1.35
altair processAttestation - setStatus - 2/3 committees join 685.06 us/op 481.10 us/op 1.42
altair processAttestation - setStatus - 4/5 committees join 855.24 us/op 676.40 us/op 1.26
altair processAttestation - setStatus - 100% committees join 1.0313 ms/op 811.41 us/op 1.27
altair processBlock - 250000 vs - 7PWei normalcase 14.178 ms/op 9.9211 ms/op 1.43
altair processBlock - 250000 vs - 7PWei normalcase hashState 42.367 ms/op 34.756 ms/op 1.22
altair processBlock - 250000 vs - 7PWei worstcase 42.573 ms/op 37.009 ms/op 1.15
altair processBlock - 250000 vs - 7PWei worstcase hashState 112.36 ms/op 95.838 ms/op 1.17
phase0 processBlock - 250000 vs - 7PWei normalcase 4.3248 ms/op 3.2664 ms/op 1.32
phase0 processBlock - 250000 vs - 7PWei worstcase 37.911 ms/op 33.923 ms/op 1.12
altair processEth1Data - 250000 vs - 7PWei normalcase 757.65 us/op 599.97 us/op 1.26
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 19.155 us/op 11.181 us/op 1.71
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 72.876 us/op 62.899 us/op 1.16
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 28.080 us/op 16.665 us/op 1.68
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 24.206 us/op 11.500 us/op 2.10
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 257.62 us/op 189.38 us/op 1.36
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 1.9985 ms/op 1.4632 ms/op 1.37
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 2.0780 ms/op 1.6798 ms/op 1.24
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 2.0538 ms/op 2.9176 ms/op 0.70
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 4.6160 ms/op 3.9795 ms/op 1.16
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 2.8389 ms/op 2.6734 ms/op 1.06
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 6.0865 ms/op 5.7619 ms/op 1.06
Tree 40 250000 create 407.21 ms/op 369.36 ms/op 1.10
Tree 40 250000 get(125000) 215.77 ns/op 211.84 ns/op 1.02
Tree 40 250000 set(125000) 1.1838 us/op 1.0979 us/op 1.08
Tree 40 250000 toArray() 24.835 ms/op 21.738 ms/op 1.14
Tree 40 250000 iterate all - toArray() + loop 24.886 ms/op 20.062 ms/op 1.24
Tree 40 250000 iterate all - get(i) 77.367 ms/op 67.584 ms/op 1.14
MutableVector 250000 create 15.770 ms/op 17.852 ms/op 0.88
MutableVector 250000 get(125000) 6.8030 ns/op 6.6380 ns/op 1.02
MutableVector 250000 set(125000) 337.98 ns/op 281.89 ns/op 1.20
MutableVector 250000 toArray() 4.5686 ms/op 3.3645 ms/op 1.36
MutableVector 250000 iterate all - toArray() + loop 4.5016 ms/op 3.9873 ms/op 1.13
MutableVector 250000 iterate all - get(i) 1.6053 ms/op 1.5522 ms/op 1.03
Array 250000 create 4.0064 ms/op 4.0630 ms/op 0.99
Array 250000 clone - spread 1.3398 ms/op 1.2261 ms/op 1.09
Array 250000 get(125000) 1.1180 ns/op 1.0260 ns/op 1.09
Array 250000 set(125000) 4.6220 ns/op 4.1370 ns/op 1.12
Array 250000 iterate all - loop 174.13 us/op 167.68 us/op 1.04
effectiveBalanceIncrements clone Uint8Array 300000 42.115 us/op 30.217 us/op 1.39
effectiveBalanceIncrements clone MutableVector 300000 395.00 ns/op 346.00 ns/op 1.14
effectiveBalanceIncrements rw all Uint8Array 300000 208.84 us/op 202.79 us/op 1.03
effectiveBalanceIncrements rw all MutableVector 300000 91.497 ms/op 83.258 ms/op 1.10
phase0 afterProcessEpoch - 250000 vs - 7PWei 120.60 ms/op 114.51 ms/op 1.05
phase0 beforeProcessEpoch - 250000 vs - 7PWei 58.882 ms/op 54.807 ms/op 1.07
altair processEpoch - mainnet_e81889 587.26 ms/op 557.44 ms/op 1.05
mainnet_e81889 - altair beforeProcessEpoch 96.111 ms/op 86.784 ms/op 1.11
mainnet_e81889 - altair processJustificationAndFinalization 14.897 us/op 15.000 us/op 0.99
mainnet_e81889 - altair processInactivityUpdates 6.5456 ms/op 6.5800 ms/op 0.99
mainnet_e81889 - altair processRewardsAndPenalties 44.576 ms/op 73.486 ms/op 0.61
mainnet_e81889 - altair processRegistryUpdates 2.9120 us/op 2.2140 us/op 1.32
mainnet_e81889 - altair processSlashings 498.00 ns/op 504.00 ns/op 0.99
mainnet_e81889 - altair processEth1DataReset 535.00 ns/op 596.00 ns/op 0.90
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.4392 ms/op 1.7281 ms/op 0.83
mainnet_e81889 - altair processSlashingsReset 3.5930 us/op 7.3270 us/op 0.49
mainnet_e81889 - altair processRandaoMixesReset 6.5230 us/op 7.5840 us/op 0.86
mainnet_e81889 - altair processHistoricalRootsUpdate 744.00 ns/op 820.00 ns/op 0.91
mainnet_e81889 - altair processParticipationFlagUpdates 2.6290 us/op 1.8720 us/op 1.40
mainnet_e81889 - altair processSyncCommitteeUpdates 1.4100 us/op 744.00 ns/op 1.90
mainnet_e81889 - altair afterProcessEpoch 127.21 ms/op 121.97 ms/op 1.04
capella processEpoch - mainnet_e217614 2.6756 s/op 2.3516 s/op 1.14
mainnet_e217614 - capella beforeProcessEpoch 619.89 ms/op 509.63 ms/op 1.22
mainnet_e217614 - capella processJustificationAndFinalization 29.600 us/op 25.358 us/op 1.17
mainnet_e217614 - capella processInactivityUpdates 28.347 ms/op 20.848 ms/op 1.36
mainnet_e217614 - capella processRewardsAndPenalties 461.08 ms/op 424.85 ms/op 1.09
mainnet_e217614 - capella processRegistryUpdates 25.425 us/op 30.864 us/op 0.82
mainnet_e217614 - capella processSlashings 499.00 ns/op 651.00 ns/op 0.77
mainnet_e217614 - capella processEth1DataReset 436.00 ns/op 500.00 ns/op 0.87
mainnet_e217614 - capella processEffectiveBalanceUpdates 9.0358 ms/op 5.5342 ms/op 1.63
mainnet_e217614 - capella processSlashingsReset 3.6100 us/op 4.7140 us/op 0.77
mainnet_e217614 - capella processRandaoMixesReset 4.0170 us/op 5.2700 us/op 0.76
mainnet_e217614 - capella processHistoricalRootsUpdate 568.00 ns/op 506.00 ns/op 1.12
mainnet_e217614 - capella processParticipationFlagUpdates 1.8520 us/op 1.8630 us/op 0.99
mainnet_e217614 - capella afterProcessEpoch 345.24 ms/op 336.52 ms/op 1.03
phase0 processEpoch - mainnet_e58758 492.12 ms/op 519.63 ms/op 0.95
mainnet_e58758 - phase0 beforeProcessEpoch 122.29 ms/op 159.54 ms/op 0.77
mainnet_e58758 - phase0 processJustificationAndFinalization 16.387 us/op 21.727 us/op 0.75
mainnet_e58758 - phase0 processRewardsAndPenalties 52.984 ms/op 67.684 ms/op 0.78
mainnet_e58758 - phase0 processRegistryUpdates 12.657 us/op 16.654 us/op 0.76
mainnet_e58758 - phase0 processSlashings 632.00 ns/op 674.00 ns/op 0.94
mainnet_e58758 - phase0 processEth1DataReset 395.00 ns/op 638.00 ns/op 0.62
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.1783 ms/op 2.1456 ms/op 0.55
mainnet_e58758 - phase0 processSlashingsReset 6.5020 us/op 4.3970 us/op 1.48
mainnet_e58758 - phase0 processRandaoMixesReset 5.8800 us/op 7.6070 us/op 0.77
mainnet_e58758 - phase0 processHistoricalRootsUpdate 823.00 ns/op 678.00 ns/op 1.21
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.8330 us/op 5.8900 us/op 0.65
mainnet_e58758 - phase0 afterProcessEpoch 101.35 ms/op 93.071 ms/op 1.09
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.4290 ms/op 2.5930 ms/op 0.55
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.4513 ms/op 1.5247 ms/op 0.95
altair processInactivityUpdates - 250000 normalcase 24.668 ms/op 30.993 ms/op 0.80
altair processInactivityUpdates - 250000 worstcase 27.905 ms/op 32.809 ms/op 0.85
phase0 processRegistryUpdates - 250000 normalcase 9.8130 us/op 12.953 us/op 0.76
phase0 processRegistryUpdates - 250000 badcase_full_deposits 334.00 us/op 643.69 us/op 0.52
phase0 processRegistryUpdates - 250000 worstcase 0.5 127.13 ms/op 138.76 ms/op 0.92
altair processRewardsAndPenalties - 250000 normalcase 59.169 ms/op 72.214 ms/op 0.82
altair processRewardsAndPenalties - 250000 worstcase 57.116 ms/op 69.276 ms/op 0.82
phase0 getAttestationDeltas - 250000 normalcase 9.3448 ms/op 9.3050 ms/op 1.00
phase0 getAttestationDeltas - 250000 worstcase 9.7124 ms/op 9.3875 ms/op 1.03
phase0 processSlashings - 250000 worstcase 111.84 us/op 131.98 us/op 0.85
altair processSyncCommitteeUpdates - 250000 167.61 ms/op 152.37 ms/op 1.10
BeaconState.hashTreeRoot - No change 262.00 ns/op 255.00 ns/op 1.03
BeaconState.hashTreeRoot - 1 full validator 130.71 us/op 189.73 us/op 0.69
BeaconState.hashTreeRoot - 32 full validator 1.3012 ms/op 1.6575 ms/op 0.79
BeaconState.hashTreeRoot - 512 full validator 13.229 ms/op 18.921 ms/op 0.70
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 148.41 us/op 176.72 us/op 0.84
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 2.0486 ms/op 2.9053 ms/op 0.71
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 26.456 ms/op 32.017 ms/op 0.83
BeaconState.hashTreeRoot - 1 balances 140.24 us/op 139.72 us/op 1.00
BeaconState.hashTreeRoot - 32 balances 1.2027 ms/op 1.2460 ms/op 0.97
BeaconState.hashTreeRoot - 512 balances 13.762 ms/op 13.688 ms/op 1.01
BeaconState.hashTreeRoot - 250000 balances 211.90 ms/op 226.81 ms/op 0.93
aggregationBits - 2048 els - zipIndexesInBitList 18.321 us/op 17.145 us/op 1.07
byteArrayEquals 32 77.968 ns/op 75.458 ns/op 1.03
Buffer.compare 32 58.620 ns/op 55.700 ns/op 1.05
byteArrayEquals 1024 2.1183 us/op 2.0285 us/op 1.04
Buffer.compare 1024 76.112 ns/op 71.235 ns/op 1.07
byteArrayEquals 16384 34.756 us/op 32.605 us/op 1.07
Buffer.compare 16384 262.26 ns/op 262.48 ns/op 1.00
byteArrayEquals 123687377 255.86 ms/op 260.36 ms/op 0.98
Buffer.compare 123687377 9.4019 ms/op 9.0453 ms/op 1.04
byteArrayEquals 32 - diff last byte 88.404 ns/op 81.448 ns/op 1.09
Buffer.compare 32 - diff last byte 60.000 ns/op 62.322 ns/op 0.96
byteArrayEquals 1024 - diff last byte 2.1394 us/op 2.2273 us/op 0.96
Buffer.compare 1024 - diff last byte 75.241 ns/op 75.671 ns/op 0.99
byteArrayEquals 16384 - diff last byte 34.467 us/op 35.119 us/op 0.98
Buffer.compare 16384 - diff last byte 287.23 ns/op 311.74 ns/op 0.92
byteArrayEquals 123687377 - diff last byte 281.15 ms/op 268.16 ms/op 1.05
Buffer.compare 123687377 - diff last byte 9.9734 ms/op 8.8174 ms/op 1.13
byteArrayEquals 32 - random bytes 7.0130 ns/op 6.5850 ns/op 1.06
Buffer.compare 32 - random bytes 67.079 ns/op 64.080 ns/op 1.05
byteArrayEquals 1024 - random bytes 6.8740 ns/op 6.5200 ns/op 1.05
Buffer.compare 1024 - random bytes 66.246 ns/op 63.601 ns/op 1.04
byteArrayEquals 16384 - random bytes 7.0510 ns/op 6.5320 ns/op 1.08
Buffer.compare 16384 - random bytes 66.097 ns/op 63.880 ns/op 1.03
byteArrayEquals 123687377 - random bytes 11.850 ns/op 10.140 ns/op 1.17
Buffer.compare 123687377 - random bytes 80.660 ns/op 77.620 ns/op 1.04
regular array get 100000 times 48.793 us/op 47.681 us/op 1.02
wrappedArray get 100000 times 48.716 us/op 50.827 us/op 0.96
arrayWithProxy get 100000 times 15.109 ms/op 14.774 ms/op 1.02
ssz.Root.equals 58.161 ns/op 60.890 ns/op 0.96
byteArrayEquals 57.995 ns/op 59.525 ns/op 0.97
Buffer.compare 12.816 ns/op 12.994 ns/op 0.99
shuffle list - 16384 els 7.3548 ms/op 7.4534 ms/op 0.99
shuffle list - 250000 els 109.52 ms/op 113.34 ms/op 0.97
processSlot - 1 slots 21.095 us/op 21.388 us/op 0.99
processSlot - 32 slots 3.8637 ms/op 4.1495 ms/op 0.93
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 61.537 ms/op 60.627 ms/op 1.02
getCommitteeAssignments - req 1 vs - 250000 vc 2.6024 ms/op 2.6066 ms/op 1.00
getCommitteeAssignments - req 100 vs - 250000 vc 3.8472 ms/op 3.8584 ms/op 1.00
getCommitteeAssignments - req 1000 vs - 250000 vc 4.2063 ms/op 4.1734 ms/op 1.01
findModifiedValidators - 10000 modified validators 551.87 ms/op 597.54 ms/op 0.92
findModifiedValidators - 1000 modified validators 459.00 ms/op 496.45 ms/op 0.92
findModifiedValidators - 100 modified validators 393.46 ms/op 492.26 ms/op 0.80
findModifiedValidators - 10 modified validators 438.86 ms/op 470.94 ms/op 0.93
findModifiedValidators - 1 modified validators 425.36 ms/op 473.01 ms/op 0.90
findModifiedValidators - no difference 421.66 ms/op 488.08 ms/op 0.86
compare ViewDUs 4.7415 s/op 5.1864 s/op 0.91
compare each validator Uint8Array 1.8914 s/op 1.7828 s/op 1.06
compare ViewDU to Uint8Array 1.4042 s/op 1.3830 s/op 1.02
migrate state 1000000 validators, 24 modified, 0 new 822.02 ms/op 900.06 ms/op 0.91
migrate state 1000000 validators, 1700 modified, 1000 new 1.0588 s/op 1.3101 s/op 0.81
migrate state 1000000 validators, 3400 modified, 2000 new 1.2910 s/op 1.5017 s/op 0.86
migrate state 1500000 validators, 24 modified, 0 new 820.02 ms/op 1.0023 s/op 0.82
migrate state 1500000 validators, 1700 modified, 1000 new 1.0959 s/op 1.2620 s/op 0.87
migrate state 1500000 validators, 3400 modified, 2000 new 1.4208 s/op 1.3991 s/op 1.02
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.2600 ns/op 5.2500 ns/op 1.00
state getBlockRootAtSlot - 250000 vs - 7PWei 1.0243 us/op 939.23 ns/op 1.09
computeProposers - vc 250000 10.671 ms/op 10.659 ms/op 1.00
computeEpochShuffling - vc 250000 109.35 ms/op 109.01 ms/op 1.00
getNextSyncCommittee - vc 250000 169.22 ms/op 159.40 ms/op 1.06
computeSigningRoot for AttestationData 26.133 us/op 30.713 us/op 0.85
hash AttestationData serialized data then Buffer.toString(base64) 2.4283 us/op 2.3574 us/op 1.03
toHexString serialized data 1.1613 us/op 1.1396 us/op 1.02
Buffer.toString(base64) 263.92 ns/op 233.54 ns/op 1.13

by benchmarkbot/action

@jeluard jeluard force-pushed the jeluard/beacon-api-ignore-list branch from 7d7e4ca to 12c7ef4 Compare December 12, 2023 12:53
@jeluard jeluard marked this pull request as ready for review December 12, 2023 15:14
@jeluard jeluard requested a review from a team as a code owner December 12, 2023 15:14
@jeluard jeluard requested a review from nflaig December 12, 2023 15:14
@jeluard jeluard requested a review from nflaig December 13, 2023 11:21
packages/api/test/unit/beacon/oapiSpec.test.ts Outdated Show resolved Hide resolved
packages/api/test/utils/checkAgainstSpec.ts Outdated Show resolved Hide resolved
packages/api/test/unit/beacon/oapiSpec.test.ts Outdated Show resolved Hide resolved
Copy link
Member

@nflaig nflaig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty good, I like the semantics. Just some minor nits

packages/api/test/unit/beacon/oapiSpec.test.ts Outdated Show resolved Hide resolved
packages/api/test/unit/beacon/oapiSpec.test.ts Outdated Show resolved Hide resolved
packages/api/test/utils/checkAgainstSpec.ts Outdated Show resolved Hide resolved
packages/api/test/utils/checkAgainstSpec.ts Outdated Show resolved Hide resolved
jeluard and others added 4 commits December 14, 2023 08:20
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
nflaig
nflaig previously approved these changes Dec 14, 2023
packages/api/test/utils/checkAgainstSpec.ts Outdated Show resolved Hide resolved
packages/api/test/utils/checkAgainstSpec.ts Outdated Show resolved Hide resolved
packages/api/test/unit/beacon/oapiSpec.test.ts Outdated Show resolved Hide resolved
@jeluard
Copy link
Contributor Author

jeluard commented Dec 19, 2023

@nazarhussain your points have been addressed

Copy link
Member

@wemeetagain wemeetagain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm 👍

@wemeetagain wemeetagain merged commit 6d1dc61 into unstable Dec 19, 2023
14 of 15 checks passed
@wemeetagain wemeetagain deleted the jeluard/beacon-api-ignore-list branch December 19, 2023 14:09
twoeths pushed a commit that referenced this pull request Dec 22, 2023
* Fix operationId of light_client routes

* Fix operationId of bls_to_execution_changes routes

* Update beacon api spec version to 2.4.0

* Push case change

* Remove now useless file

* fix: lints

* fix: filter broken test

* Revert "Fix operationId of light_client routes"

This reverts commit 91cd2af.

* Revert "Fix operationId of bls_to_execution_changes routes"

This reverts commit ad53c2d.

* test: ignore missing routes

* test: allow to filter required properties from testing

* fix: incorrect case

* test: fixed incorrect test filtering

* fix: lints

* fix: cleanup

* test: allow more fine grain API tests filtering

* fix: lints

* test: increase JSON schema validation strictness

* fix: restore removed keyword implementation

* test: improve filtering semantic

* test: add support for JSONPath syntax to filtering

* fix: typo

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* fix: wording

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* test: improve semantic

* test: added issue for context

* fix: improved issues references

* fix: incorrect dotted property parsing

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.14.0 🎉

ensi321 pushed a commit to ensi321/lodestar that referenced this pull request Jan 22, 2024
* Fix operationId of light_client routes

* Fix operationId of bls_to_execution_changes routes

* Update beacon api spec version to 2.4.0

* Push case change

* Remove now useless file

* fix: lints

* fix: filter broken test

* Revert "Fix operationId of light_client routes"

This reverts commit 91cd2af.

* Revert "Fix operationId of bls_to_execution_changes routes"

This reverts commit ad53c2d.

* test: ignore missing routes

* test: allow to filter required properties from testing

* fix: incorrect case

* test: fixed incorrect test filtering

* fix: lints

* fix: cleanup

* test: allow more fine grain API tests filtering

* fix: lints

* test: increase JSON schema validation strictness

* fix: restore removed keyword implementation

* test: improve filtering semantic

* test: add support for JSONPath syntax to filtering

* fix: typo

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* fix: wording

Co-authored-by: Nico Flaig <nflaig@protonmail.com>

* test: improve semantic

* test: added issue for context

* fix: improved issues references

* fix: incorrect dotted property parsing

---------

Co-authored-by: Nico Flaig <nflaig@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants