Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add compact proofs API #5155

Merged
merged 5 commits into from
Feb 20, 2023
Merged

Add compact proofs API #5155

merged 5 commits into from
Feb 20, 2023

Conversation

wemeetagain
Copy link
Member

@wemeetagain wemeetagain commented Feb 14, 2023 •
edited
Loading

Description

This is a breaking change to our existing getStateProof endpoint.

Before:

const proof = api.debug.getStateProof(paths);

After:

import {computeDescriptor} from "@chainsafe/persistent-merkle-tree";
import {ssz} from "@lodestar/types";

const gindices = paths.map((path) => ssz.bellatrix.BeaconState.getPathInfo(path).gindex);
const descriptor = computeDescriptor(gindices);
const proof = api.debug.getStateProof(descriptor);

@wemeetagain wemeetagain requested a review from a team as a code owner February 14, 2023 23:06
@github-actions
Copy link
Contributor

github-actions bot commented Feb 14, 2023 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: cf8a3b0 Previous: 63479b4 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 499.92 us/op 701.74 us/op 0.71
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 45.486 us/op 43.588 us/op 1.04
BLS verify - blst-native 1.2180 ms/op 1.1722 ms/op 1.04
BLS verifyMultipleSignatures 3 - blst-native 2.4880 ms/op 2.3940 ms/op 1.04
BLS verifyMultipleSignatures 8 - blst-native 5.2779 ms/op 5.1279 ms/op 1.03
BLS verifyMultipleSignatures 32 - blst-native 19.492 ms/op 18.512 ms/op 1.05
BLS aggregatePubkeys 32 - blst-native 25.650 us/op 24.782 us/op 1.04
BLS aggregatePubkeys 128 - blst-native 100.02 us/op 97.190 us/op 1.03
getAttestationsForBlock 57.525 ms/op 53.304 ms/op 1.08
isKnown best case - 1 super set check 289.00 ns/op 264.00 ns/op 1.09
isKnown normal case - 2 super set checks 278.00 ns/op 248.00 ns/op 1.12
isKnown worse case - 16 super set checks 260.00 ns/op 248.00 ns/op 1.05
CheckpointStateCache - add get delete 5.6390 us/op 5.1650 us/op 1.09
validate gossip signedAggregateAndProof - struct 2.7767 ms/op 2.7542 ms/op 1.01
validate gossip attestation - struct 1.3237 ms/op 1.3186 ms/op 1.00
pickEth1Vote - no votes 1.3266 ms/op 1.2033 ms/op 1.10
pickEth1Vote - max votes 9.3635 ms/op 10.044 ms/op 0.93
pickEth1Vote - Eth1Data hashTreeRoot value x2048 9.3621 ms/op 8.6356 ms/op 1.08
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 16.486 ms/op 14.425 ms/op 1.14
pickEth1Vote - Eth1Data fastSerialize value x2048 660.93 us/op 626.86 us/op 1.05
pickEth1Vote - Eth1Data fastSerialize tree x2048 6.4233 ms/op 7.4927 ms/op 0.86
bytes32 toHexString 549.00 ns/op 481.00 ns/op 1.14
bytes32 Buffer.toString(hex) 398.00 ns/op 331.00 ns/op 1.20
bytes32 Buffer.toString(hex) from Uint8Array 623.00 ns/op 536.00 ns/op 1.16
bytes32 Buffer.toString(hex) + 0x 394.00 ns/op 333.00 ns/op 1.18
Object access 1 prop 0.17500 ns/op 0.17300 ns/op 1.01
Map access 1 prop 0.16200 ns/op 0.15600 ns/op 1.04
Object get x1000 6.5390 ns/op 6.3600 ns/op 1.03
Map get x1000 0.58400 ns/op 0.61600 ns/op 0.95
Object set x1000 57.954 ns/op 50.990 ns/op 1.14
Map set x1000 46.875 ns/op 42.109 ns/op 1.11
Return object 10000 times 0.24350 ns/op 0.23340 ns/op 1.04
Throw Error 10000 times 4.2267 us/op 4.0960 us/op 1.03
fastMsgIdFn sha256 / 200 bytes 3.5160 us/op 3.4210 us/op 1.03
fastMsgIdFn h32 xxhash / 200 bytes 306.00 ns/op 301.00 ns/op 1.02
fastMsgIdFn h64 xxhash / 200 bytes 450.00 ns/op 391.00 ns/op 1.15
fastMsgIdFn sha256 / 1000 bytes 11.690 us/op 11.573 us/op 1.01
fastMsgIdFn h32 xxhash / 1000 bytes 442.00 ns/op 426.00 ns/op 1.04
fastMsgIdFn h64 xxhash / 1000 bytes 492.00 ns/op 467.00 ns/op 1.05
fastMsgIdFn sha256 / 10000 bytes 106.70 us/op 103.73 us/op 1.03
fastMsgIdFn h32 xxhash / 10000 bytes 2.0060 us/op 1.9490 us/op 1.03
fastMsgIdFn h64 xxhash / 10000 bytes 1.4590 us/op 1.3810 us/op 1.06
enrSubnets - fastDeserialize 64 bits 1.3980 us/op 1.3130 us/op 1.06
enrSubnets - ssz BitVector 64 bits 491.00 ns/op 487.00 ns/op 1.01
enrSubnets - fastDeserialize 4 bits 201.00 ns/op 185.00 ns/op 1.09
enrSubnets - ssz BitVector 4 bits 625.00 ns/op 495.00 ns/op 1.26
prioritizePeers score -10:0 att 32-0.1 sync 2-0 100.70 us/op 95.372 us/op 1.06
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 141.67 us/op 120.90 us/op 1.17
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 194.36 us/op 166.15 us/op 1.17
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 370.66 us/op 301.96 us/op 1.23
prioritizePeers score 0:0 att 64-1 sync 4-1 381.58 us/op 359.57 us/op 1.06
array of 16000 items push then shift 1.6487 us/op 1.6317 us/op 1.01
LinkedList of 16000 items push then shift 8.9970 ns/op 8.8230 ns/op 1.02
array of 16000 items push then pop 98.863 ns/op 83.191 ns/op 1.19
LinkedList of 16000 items push then pop 8.7330 ns/op 8.5630 ns/op 1.02
array of 24000 items push then shift 2.3892 us/op 2.3791 us/op 1.00
LinkedList of 24000 items push then shift 9.1210 ns/op 8.6570 ns/op 1.05
array of 24000 items push then pop 92.149 ns/op 72.186 ns/op 1.28
LinkedList of 24000 items push then pop 8.9430 ns/op 8.4830 ns/op 1.05
intersect bitArray bitLen 8 13.586 ns/op 13.258 ns/op 1.02
intersect array and set length 8 81.402 ns/op 77.928 ns/op 1.04
intersect bitArray bitLen 128 44.845 ns/op 44.024 ns/op 1.02
intersect array and set length 128 1.1794 us/op 1.0504 us/op 1.12
Buffer.concat 32 items 2.7430 us/op 2.8500 us/op 0.96
Uint8Array.set 32 items 2.6310 us/op 2.7320 us/op 0.96
pass gossip attestations to forkchoice per slot 2.3668 ms/op 3.0895 ms/op 0.77
computeDeltas 3.0018 ms/op 3.4573 ms/op 0.87
computeProposerBoostScoreFromBalances 1.7933 ms/op 1.7061 ms/op 1.05
altair processAttestation - 250000 vs - 7PWei normalcase 2.7532 ms/op 2.1296 ms/op 1.29
altair processAttestation - 250000 vs - 7PWei worstcase 3.9866 ms/op 3.1553 ms/op 1.26
altair processAttestation - setStatus - 1/6 committees join 144.53 us/op 131.87 us/op 1.10
altair processAttestation - setStatus - 1/3 committees join 285.06 us/op 260.07 us/op 1.10
altair processAttestation - setStatus - 1/2 committees join 374.42 us/op 362.71 us/op 1.03
altair processAttestation - setStatus - 2/3 committees join 473.44 us/op 449.06 us/op 1.05
altair processAttestation - setStatus - 4/5 committees join 665.31 us/op 632.50 us/op 1.05
altair processAttestation - setStatus - 100% committees join 789.37 us/op 744.07 us/op 1.06
altair processBlock - 250000 vs - 7PWei normalcase 17.844 ms/op 18.026 ms/op 0.99
altair processBlock - 250000 vs - 7PWei normalcase hashState 26.368 ms/op 26.599 ms/op 0.99
altair processBlock - 250000 vs - 7PWei worstcase 51.229 ms/op 54.955 ms/op 0.93
altair processBlock - 250000 vs - 7PWei worstcase hashState 71.167 ms/op 68.414 ms/op 1.04
phase0 processBlock - 250000 vs - 7PWei normalcase 2.4474 ms/op 1.9878 ms/op 1.23
phase0 processBlock - 250000 vs - 7PWei worstcase 32.648 ms/op 27.525 ms/op 1.19
altair processEth1Data - 250000 vs - 7PWei normalcase 576.77 us/op 462.36 us/op 1.25
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 7.9380 us/op 6.7070 us/op 1.18
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 26.933 us/op 19.407 us/op 1.39
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 11.409 us/op 8.4130 us/op 1.36
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 9.3850 us/op 6.4270 us/op 1.46
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 113.66 us/op 74.386 us/op 1.53
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 658.10 us/op 597.85 us/op 1.10
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 921.03 us/op 869.94 us/op 1.06
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 1.0077 ms/op 862.79 us/op 1.17
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.4118 ms/op 2.2955 ms/op 1.05
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.6175 ms/op 1.5169 ms/op 1.07
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 4.4666 ms/op 3.5716 ms/op 1.25
Tree 40 250000 create 383.54 ms/op 303.84 ms/op 1.26
Tree 40 250000 get(125000) 196.76 ns/op 176.39 ns/op 1.12
Tree 40 250000 set(125000) 1.1289 us/op 895.75 ns/op 1.26
Tree 40 250000 toArray() 23.078 ms/op 16.925 ms/op 1.36
Tree 40 250000 iterate all - toArray() + loop 23.500 ms/op 18.179 ms/op 1.29
Tree 40 250000 iterate all - get(i) 77.681 ms/op 63.130 ms/op 1.23
MutableVector 250000 create 10.914 ms/op 9.4880 ms/op 1.15
MutableVector 250000 get(125000) 6.4200 ns/op 6.2420 ns/op 1.03
MutableVector 250000 set(125000) 286.25 ns/op 242.52 ns/op 1.18
MutableVector 250000 toArray() 3.5735 ms/op 2.6488 ms/op 1.35
MutableVector 250000 iterate all - toArray() + loop 3.8341 ms/op 2.7402 ms/op 1.40
MutableVector 250000 iterate all - get(i) 1.5545 ms/op 1.5016 ms/op 1.04
Array 250000 create 3.1138 ms/op 2.4654 ms/op 1.26
Array 250000 clone - spread 1.2184 ms/op 1.1472 ms/op 1.06
Array 250000 get(125000) 0.61200 ns/op 0.53700 ns/op 1.14
Array 250000 set(125000) 0.68800 ns/op 0.62000 ns/op 1.11
Array 250000 iterate all - loop 85.652 us/op 80.333 us/op 1.07
effectiveBalanceIncrements clone Uint8Array 300000 36.544 us/op 25.666 us/op 1.42
effectiveBalanceIncrements clone MutableVector 300000 378.00 ns/op 352.00 ns/op 1.07
effectiveBalanceIncrements rw all Uint8Array 300000 170.34 us/op 157.67 us/op 1.08
effectiveBalanceIncrements rw all MutableVector 300000 86.657 ms/op 76.188 ms/op 1.14
phase0 afterProcessEpoch - 250000 vs - 7PWei 117.14 ms/op 108.29 ms/op 1.08
phase0 beforeProcessEpoch - 250000 vs - 7PWei 37.020 ms/op 32.185 ms/op 1.15
altair processEpoch - mainnet_e81889 331.55 ms/op 329.81 ms/op 1.01
mainnet_e81889 - altair beforeProcessEpoch 67.357 ms/op 62.858 ms/op 1.07
mainnet_e81889 - altair processJustificationAndFinalization 18.857 us/op 18.323 us/op 1.03
mainnet_e81889 - altair processInactivityUpdates 6.3949 ms/op 5.2772 ms/op 1.21
mainnet_e81889 - altair processRewardsAndPenalties 68.062 ms/op 66.205 ms/op 1.03
mainnet_e81889 - altair processRegistryUpdates 2.6460 us/op 2.9120 us/op 0.91
mainnet_e81889 - altair processSlashings 424.00 ns/op 458.00 ns/op 0.93
mainnet_e81889 - altair processEth1DataReset 505.00 ns/op 504.00 ns/op 1.00
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2454 ms/op 1.1979 ms/op 1.04
mainnet_e81889 - altair processSlashingsReset 4.7270 us/op 4.6760 us/op 1.01
mainnet_e81889 - altair processRandaoMixesReset 4.5500 us/op 4.4510 us/op 1.02
mainnet_e81889 - altair processHistoricalRootsUpdate 1.0970 us/op 606.00 ns/op 1.81
mainnet_e81889 - altair processParticipationFlagUpdates 2.5690 us/op 2.5020 us/op 1.03
mainnet_e81889 - altair processSyncCommitteeUpdates 529.00 ns/op 437.00 ns/op 1.21
mainnet_e81889 - altair afterProcessEpoch 127.53 ms/op 112.12 ms/op 1.14
phase0 processEpoch - mainnet_e58758 361.80 ms/op 318.57 ms/op 1.14
mainnet_e58758 - phase0 beforeProcessEpoch 139.13 ms/op 123.94 ms/op 1.12
mainnet_e58758 - phase0 processJustificationAndFinalization 18.649 us/op 15.656 us/op 1.19
mainnet_e58758 - phase0 processRewardsAndPenalties 67.704 ms/op 59.063 ms/op 1.15
mainnet_e58758 - phase0 processRegistryUpdates 7.8820 us/op 7.4840 us/op 1.05
mainnet_e58758 - phase0 processSlashings 500.00 ns/op 566.00 ns/op 0.88
mainnet_e58758 - phase0 processEth1DataReset 510.00 ns/op 541.00 ns/op 0.94
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.3455 ms/op 939.77 us/op 1.43
mainnet_e58758 - phase0 processSlashingsReset 4.9880 us/op 3.5920 us/op 1.39
mainnet_e58758 - phase0 processRandaoMixesReset 5.1130 us/op 4.5580 us/op 1.12
mainnet_e58758 - phase0 processHistoricalRootsUpdate 898.00 ns/op 585.00 ns/op 1.54
mainnet_e58758 - phase0 processParticipationRecordUpdates 4.1680 us/op 4.5880 us/op 0.91
mainnet_e58758 - phase0 afterProcessEpoch 99.424 ms/op 91.431 ms/op 1.09
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2455 ms/op 1.2343 ms/op 1.01
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.4121 ms/op 1.4330 ms/op 0.99
altair processInactivityUpdates - 250000 normalcase 26.342 ms/op 19.719 ms/op 1.34
altair processInactivityUpdates - 250000 worstcase 27.241 ms/op 22.179 ms/op 1.23
phase0 processRegistryUpdates - 250000 normalcase 8.1200 us/op 6.9090 us/op 1.18
phase0 processRegistryUpdates - 250000 badcase_full_deposits 305.78 us/op 227.25 us/op 1.35
phase0 processRegistryUpdates - 250000 worstcase 0.5 127.33 ms/op 123.30 ms/op 1.03
altair processRewardsAndPenalties - 250000 normalcase 68.698 ms/op 57.900 ms/op 1.19
altair processRewardsAndPenalties - 250000 worstcase 66.942 ms/op 59.128 ms/op 1.13
phase0 getAttestationDeltas - 250000 normalcase 6.8600 ms/op 6.3608 ms/op 1.08
phase0 getAttestationDeltas - 250000 worstcase 6.9086 ms/op 6.4081 ms/op 1.08
phase0 processSlashings - 250000 worstcase 3.4422 ms/op 3.2932 ms/op 1.05
altair processSyncCommitteeUpdates - 250000 185.58 ms/op 171.19 ms/op 1.08
BeaconState.hashTreeRoot - No change 315.00 ns/op 255.00 ns/op 1.24
BeaconState.hashTreeRoot - 1 full validator 52.917 us/op 50.889 us/op 1.04
BeaconState.hashTreeRoot - 32 full validator 532.14 us/op 496.19 us/op 1.07
BeaconState.hashTreeRoot - 512 full validator 5.2918 ms/op 5.0401 ms/op 1.05
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 66.748 us/op 64.732 us/op 1.03
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.0400 ms/op 867.30 us/op 1.20
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 13.880 ms/op 11.134 ms/op 1.25
BeaconState.hashTreeRoot - 1 balances 60.584 us/op 48.682 us/op 1.24
BeaconState.hashTreeRoot - 32 balances 506.12 us/op 433.97 us/op 1.17
BeaconState.hashTreeRoot - 512 balances 5.8013 ms/op 4.2307 ms/op 1.37
BeaconState.hashTreeRoot - 250000 balances 84.964 ms/op 71.382 ms/op 1.19
aggregationBits - 2048 els - zipIndexesInBitList 22.804 us/op 16.415 us/op 1.39
regular array get 100000 times 35.300 us/op 32.875 us/op 1.07
wrappedArray get 100000 times 36.596 us/op 32.965 us/op 1.11
arrayWithProxy get 100000 times 18.279 ms/op 15.429 ms/op 1.18
ssz.Root.equals 708.00 ns/op 548.00 ns/op 1.29
byteArrayEquals 654.00 ns/op 537.00 ns/op 1.22
shuffle list - 16384 els 7.3730 ms/op 6.8114 ms/op 1.08
shuffle list - 250000 els 106.41 ms/op 99.736 ms/op 1.07
processSlot - 1 slots 10.523 us/op 8.6260 us/op 1.22
processSlot - 32 slots 1.4996 ms/op 1.3250 ms/op 1.13
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 232.84 us/op 178.08 us/op 1.31
getCommitteeAssignments - req 1 vs - 250000 vc 3.1513 ms/op 2.9135 ms/op 1.08
getCommitteeAssignments - req 100 vs - 250000 vc 4.3894 ms/op 4.1559 ms/op 1.06
getCommitteeAssignments - req 1000 vs - 250000 vc 4.6635 ms/op 4.4562 ms/op 1.05
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.1900 ns/op 4.4700 ns/op 1.16
state getBlockRootAtSlot - 250000 vs - 7PWei 929.56 ns/op 955.37 ns/op 0.97
computeProposers - vc 250000 11.616 ms/op 9.8966 ms/op 1.17
computeEpochShuffling - vc 250000 110.39 ms/op 96.589 ms/op 1.14
getNextSyncCommittee - vc 250000 192.16 ms/op 166.13 ms/op 1.16

by benchmarkbot/action

@wemeetagain
Copy link
Member Author

Tested by @zsfelfoldi

Hi! After fixing the format encoding on my side, the new state api seems to work fine 🙂
I updated my "blsync" geth branch that can already be used to test it, and it works fine now:
ethereum/go-ethereum#25901

twoeths
twoeths approved these changes Feb 20, 2023
View reviewed changes
Copy link
Contributor

@twoeths twoeths left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me!

@wemeetagain wemeetagain merged commit f4057cf into unstable Feb 20, 2023
@wemeetagain wemeetagain deleted the cayman/proof-apis branch February 20, 2023 02:21
@wemeetagain
Copy link
Member Author

🎉 This PR is included in v1.6.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@twoeths twoeths twoeths approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@wemeetagain @twoeths