Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discv5 worker #4988

Merged
merged 15 commits into from
Jan 20, 2023
Merged

Discv5 worker #4988

merged 15 commits into from
Jan 20, 2023

Conversation

wemeetagain
Copy link
Member

@wemeetagain wemeetagain commented Jan 9, 2023

WIP, opening for initial review / discussion.

Deployed on nogroup-hzax41-0 and the beta group

Comment on lines +72 to +73
// TODO there is a bug in enr encoding(?) that causes many enrs to be incorrectly encoded (and thus incorrectly decoded)
// this.logger.error("Unable to decode enr", {enr: Buffer.from(enrStr).toString("hex")}, e as Error);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There seems to be a bug in ENR encoding?
Uncommenting this results in a flood of error logs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any updates here?

@@ -35,7 +34,7 @@ export interface IMetadataModules {
* https://github.com/ethereum/consensus-specs/blob/v1.1.10/specs/phase0/p2p-interface.md#metadata
*/
export class MetadataController {
private enr?: ENR;
private setEnrValue?: (key: string, value: Uint8Array) => Promise<void>;
Copy link
Member Author

@wemeetagain wemeetagain Jan 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Someone has to 'own' the ENR.
Current choice is for the discv5 worker to own the ENR and just pass a function here to update it.

The FileENR should also be removed and we just write the ENR to disk on close.

@dapplion dapplion mentioned this pull request Jan 9, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Jan 9, 2023

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: f27f406 Previous: 460ba59 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 1.2742 ms/op 1.2661 ms/op 1.01
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 75.156 us/op 97.021 us/op 0.77
BLS verify - blst-native 1.8636 ms/op 2.1162 ms/op 0.88
BLS verifyMultipleSignatures 3 - blst-native 3.8093 ms/op 4.2819 ms/op 0.89
BLS verifyMultipleSignatures 8 - blst-native 8.2047 ms/op 9.1750 ms/op 0.89
BLS verifyMultipleSignatures 32 - blst-native 29.766 ms/op 33.750 ms/op 0.88
BLS aggregatePubkeys 32 - blst-native 39.345 us/op 45.526 us/op 0.86
BLS aggregatePubkeys 128 - blst-native 154.08 us/op 178.94 us/op 0.86
getAttestationsForBlock 91.553 ms/op 108.29 ms/op 0.85
isKnown best case - 1 super set check 454.00 ns/op 462.00 ns/op 0.98
isKnown normal case - 2 super set checks 445.00 ns/op 454.00 ns/op 0.98
isKnown worse case - 16 super set checks 451.00 ns/op 451.00 ns/op 1.00
CheckpointStateCache - add get delete 8.6130 us/op 8.8610 us/op 0.97
validate gossip signedAggregateAndProof - struct 4.2699 ms/op 4.6003 ms/op 0.93
validate gossip attestation - struct 2.0222 ms/op 2.2384 ms/op 0.90
pickEth1Vote - no votes 2.2010 ms/op 2.4093 ms/op 0.91
pickEth1Vote - max votes 18.632 ms/op 17.915 ms/op 1.04
pickEth1Vote - Eth1Data hashTreeRoot value x2048 12.950 ms/op 14.286 ms/op 0.91
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 22.585 ms/op 23.572 ms/op 0.96
pickEth1Vote - Eth1Data fastSerialize value x2048 1.2619 ms/op 1.2933 ms/op 0.98
pickEth1Vote - Eth1Data fastSerialize tree x2048 11.172 ms/op 10.040 ms/op 1.11
bytes32 toHexString 1.1550 us/op 1.0050 us/op 1.15
bytes32 Buffer.toString(hex) 638.00 ns/op 639.00 ns/op 1.00
bytes32 Buffer.toString(hex) from Uint8Array 917.00 ns/op 1.0550 us/op 0.87
bytes32 Buffer.toString(hex) + 0x 627.00 ns/op 677.00 ns/op 0.93
Object access 1 prop 0.29100 ns/op 0.31200 ns/op 0.93
Map access 1 prop 0.26200 ns/op 0.28600 ns/op 0.92
Object get x1000 16.261 ns/op 14.900 ns/op 1.09
Map get x1000 1.0320 ns/op 0.86400 ns/op 1.19
Object set x1000 114.11 ns/op 94.390 ns/op 1.21
Map set x1000 73.640 ns/op 70.447 ns/op 1.05
Return object 10000 times 0.37370 ns/op 0.37850 ns/op 0.99
Throw Error 10000 times 7.0459 us/op 7.1180 us/op 0.99
fastMsgIdFn sha256 / 200 bytes 4.3240 us/op 5.0310 us/op 0.86
fastMsgIdFn h32 xxhash / 200 bytes 487.00 ns/op 615.00 ns/op 0.79
fastMsgIdFn h64 xxhash / 200 bytes 693.00 ns/op 733.00 ns/op 0.95
fastMsgIdFn sha256 / 1000 bytes 14.083 us/op 16.170 us/op 0.87
fastMsgIdFn h32 xxhash / 1000 bytes 627.00 ns/op 812.00 ns/op 0.77
fastMsgIdFn h64 xxhash / 1000 bytes 760.00 ns/op 861.00 ns/op 0.88
fastMsgIdFn sha256 / 10000 bytes 121.19 us/op 137.49 us/op 0.88
fastMsgIdFn h32 xxhash / 10000 bytes 2.3910 us/op 2.5370 us/op 0.94
fastMsgIdFn h64 xxhash / 10000 bytes 1.7030 us/op 1.7890 us/op 0.95
enrSubnets - fastDeserialize 64 bits 2.5890 us/op 2.3900 us/op 1.08
enrSubnets - ssz BitVector 64 bits 907.00 ns/op 840.00 ns/op 1.08
enrSubnets - fastDeserialize 4 bits 307.00 ns/op 331.00 ns/op 0.93
enrSubnets - ssz BitVector 4 bits 927.00 ns/op 922.00 ns/op 1.01
prioritizePeers score -10:0 att 32-0.1 sync 2-0 175.74 us/op 191.32 us/op 0.92
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 202.54 us/op 228.81 us/op 0.89
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 310.30 us/op 341.49 us/op 0.91
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 537.23 us/op 597.76 us/op 0.90
prioritizePeers score 0:0 att 64-1 sync 4-1 626.52 us/op 661.10 us/op 0.95
array of 16000 items push then shift 2.8044 us/op 5.0717 us/op 0.55
LinkedList of 16000 items push then shift 17.718 ns/op 18.218 ns/op 0.97
array of 16000 items push then pop 222.11 ns/op 248.54 ns/op 0.89
LinkedList of 16000 items push then pop 16.677 ns/op 17.163 ns/op 0.97
array of 24000 items push then shift 3.9778 us/op 7.3968 us/op 0.54
LinkedList of 24000 items push then shift 20.746 ns/op 19.797 ns/op 1.05
array of 24000 items push then pop 218.75 ns/op 227.77 ns/op 0.96
LinkedList of 24000 items push then pop 17.961 ns/op 17.251 ns/op 1.04
intersect bitArray bitLen 8 29.582 ns/op 26.035 ns/op 1.14
intersect array and set length 8 176.64 ns/op 178.78 ns/op 0.99
intersect bitArray bitLen 128 108.73 ns/op 86.303 ns/op 1.26
intersect array and set length 128 2.2978 us/op 2.0275 us/op 1.13
Buffer.concat 32 items 5.1800 us/op 5.5730 us/op 0.93
Uint8Array.set 32 items 4.3720 us/op 4.6000 us/op 0.95
pass gossip attestations to forkchoice per slot 6.4677 ms/op 4.4090 ms/op 1.47
computeDeltas 6.4139 ms/op 5.1017 ms/op 1.26
computeProposerBoostScoreFromBalances 2.8455 ms/op 2.7736 ms/op 1.03
altair processAttestation - 250000 vs - 7PWei normalcase 4.3955 ms/op 4.2293 ms/op 1.04
altair processAttestation - 250000 vs - 7PWei worstcase 6.6671 ms/op 13.160 ms/op 0.51
altair processAttestation - setStatus - 1/6 committees join 212.87 us/op 258.21 us/op 0.82
altair processAttestation - setStatus - 1/3 committees join 403.63 us/op 471.45 us/op 0.86
altair processAttestation - setStatus - 1/2 committees join 569.81 us/op 688.01 us/op 0.83
altair processAttestation - setStatus - 2/3 committees join 729.89 us/op 829.18 us/op 0.88
altair processAttestation - setStatus - 4/5 committees join 1.0228 ms/op 1.2341 ms/op 0.83
altair processAttestation - setStatus - 100% committees join 1.2083 ms/op 1.5262 ms/op 0.79
altair processBlock - 250000 vs - 7PWei normalcase 27.235 ms/op 29.541 ms/op 0.92
altair processBlock - 250000 vs - 7PWei normalcase hashState 34.627 ms/op 40.080 ms/op 0.86
altair processBlock - 250000 vs - 7PWei worstcase 81.063 ms/op 103.79 ms/op 0.78
altair processBlock - 250000 vs - 7PWei worstcase hashState 98.200 ms/op 113.90 ms/op 0.86
phase0 processBlock - 250000 vs - 7PWei normalcase 3.7473 ms/op 4.0435 ms/op 0.93
phase0 processBlock - 250000 vs - 7PWei worstcase 49.795 ms/op 52.020 ms/op 0.96
altair processEth1Data - 250000 vs - 7PWei normalcase 773.37 us/op 908.42 us/op 0.85
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 10.533 us/op 20.683 us/op 0.51
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 28.683 us/op 45.483 us/op 0.63
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 12.385 us/op 27.903 us/op 0.44
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 10.826 us/op 19.924 us/op 0.54
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 106.41 us/op 143.15 us/op 0.74
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 1.0199 ms/op 1.1053 ms/op 0.92
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 1.3645 ms/op 1.5431 ms/op 0.88
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 1.3504 ms/op 1.4154 ms/op 0.95
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 4.6822 ms/op 4.8716 ms/op 0.96
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 2.8430 ms/op 3.0487 ms/op 0.93
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 7.3851 ms/op 7.9863 ms/op 0.92
Tree 40 250000 create 730.02 ms/op 787.18 ms/op 0.93
Tree 40 250000 get(125000) 287.40 ns/op 288.35 ns/op 1.00
Tree 40 250000 set(125000) 2.6386 us/op 2.3746 us/op 1.11
Tree 40 250000 toArray() 31.946 ms/op 32.915 ms/op 0.97
Tree 40 250000 iterate all - toArray() + loop 31.817 ms/op 33.251 ms/op 0.96
Tree 40 250000 iterate all - get(i) 105.11 ms/op 112.34 ms/op 0.94
MutableVector 250000 create 15.580 ms/op 15.282 ms/op 1.02
MutableVector 250000 get(125000) 13.194 ns/op 13.327 ns/op 0.99
MutableVector 250000 set(125000) 736.56 ns/op 637.60 ns/op 1.16
MutableVector 250000 toArray() 6.7500 ms/op 6.8879 ms/op 0.98
MutableVector 250000 iterate all - toArray() + loop 6.9732 ms/op 6.9890 ms/op 1.00
MutableVector 250000 iterate all - get(i) 3.4748 ms/op 3.4810 ms/op 1.00
Array 250000 create 6.5996 ms/op 6.0847 ms/op 1.08
Array 250000 clone - spread 3.1488 ms/op 2.5833 ms/op 1.22
Array 250000 get(125000) 1.4890 ns/op 1.0980 ns/op 1.36
Array 250000 set(125000) 1.4830 ns/op 1.1200 ns/op 1.32
Array 250000 iterate all - loop 167.81 us/op 136.28 us/op 1.23
effectiveBalanceIncrements clone Uint8Array 300000 92.224 us/op 67.194 us/op 1.37
effectiveBalanceIncrements clone MutableVector 300000 981.00 ns/op 730.00 ns/op 1.34
effectiveBalanceIncrements rw all Uint8Array 300000 252.97 us/op 253.75 us/op 1.00
effectiveBalanceIncrements rw all MutableVector 300000 239.73 ms/op 183.43 ms/op 1.31
phase0 afterProcessEpoch - 250000 vs - 7PWei 204.80 ms/op 207.66 ms/op 0.99
phase0 beforeProcessEpoch - 250000 vs - 7PWei 72.117 ms/op 75.916 ms/op 0.95
altair processEpoch - mainnet_e81889 498.08 ms/op 601.46 ms/op 0.83
mainnet_e81889 - altair beforeProcessEpoch 144.70 ms/op 159.36 ms/op 0.91
mainnet_e81889 - altair processJustificationAndFinalization 21.110 us/op 69.713 us/op 0.30
mainnet_e81889 - altair processInactivityUpdates 11.445 ms/op 10.545 ms/op 1.09
mainnet_e81889 - altair processRewardsAndPenalties 91.618 ms/op 99.808 ms/op 0.92
mainnet_e81889 - altair processRegistryUpdates 3.5970 us/op 11.448 us/op 0.31
mainnet_e81889 - altair processSlashings 1.0110 us/op 2.6400 us/op 0.38
mainnet_e81889 - altair processEth1DataReset 925.00 ns/op 2.7240 us/op 0.34
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.3497 ms/op 2.3008 ms/op 1.02
mainnet_e81889 - altair processSlashingsReset 5.8660 us/op 23.401 us/op 0.25
mainnet_e81889 - altair processRandaoMixesReset 6.7710 us/op 24.934 us/op 0.27
mainnet_e81889 - altair processHistoricalRootsUpdate 1.1820 us/op 3.4540 us/op 0.34
mainnet_e81889 - altair processParticipationFlagUpdates 3.5140 us/op 13.361 us/op 0.26
mainnet_e81889 - altair processSyncCommitteeUpdates 712.00 ns/op 2.4990 us/op 0.28
mainnet_e81889 - altair afterProcessEpoch 195.13 ms/op 186.95 ms/op 1.04
phase0 processEpoch - mainnet_e58758 534.49 ms/op 572.45 ms/op 0.93
mainnet_e58758 - phase0 beforeProcessEpoch 238.76 ms/op 281.31 ms/op 0.85
mainnet_e58758 - phase0 processJustificationAndFinalization 23.169 us/op 57.542 us/op 0.40
mainnet_e58758 - phase0 processRewardsAndPenalties 127.06 ms/op 82.261 ms/op 1.54
mainnet_e58758 - phase0 processRegistryUpdates 10.501 us/op 31.917 us/op 0.33
mainnet_e58758 - phase0 processSlashings 933.00 ns/op 3.0290 us/op 0.31
mainnet_e58758 - phase0 processEth1DataReset 1.0750 us/op 2.9780 us/op 0.36
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.9369 ms/op 1.7764 ms/op 1.09
mainnet_e58758 - phase0 processSlashingsReset 5.5070 us/op 13.367 us/op 0.41
mainnet_e58758 - phase0 processRandaoMixesReset 8.5320 us/op 23.874 us/op 0.36
mainnet_e58758 - phase0 processHistoricalRootsUpdate 889.00 ns/op 3.4640 us/op 0.26
mainnet_e58758 - phase0 processParticipationRecordUpdates 6.2650 us/op 20.996 us/op 0.30
mainnet_e58758 - phase0 afterProcessEpoch 161.38 ms/op 158.64 ms/op 1.02
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.7943 ms/op 2.2313 ms/op 1.25
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.9803 ms/op 2.6240 ms/op 1.14
altair processInactivityUpdates - 250000 normalcase 38.262 ms/op 45.166 ms/op 0.85
altair processInactivityUpdates - 250000 worstcase 42.288 ms/op 46.477 ms/op 0.91
phase0 processRegistryUpdates - 250000 normalcase 8.9900 us/op 26.058 us/op 0.34
phase0 processRegistryUpdates - 250000 badcase_full_deposits 441.39 us/op 536.18 us/op 0.82
phase0 processRegistryUpdates - 250000 worstcase 0.5 215.70 ms/op 258.60 ms/op 0.83
altair processRewardsAndPenalties - 250000 normalcase 124.16 ms/op 141.67 ms/op 0.88
altair processRewardsAndPenalties - 250000 worstcase 84.817 ms/op 129.45 ms/op 0.66
phase0 getAttestationDeltas - 250000 normalcase 13.726 ms/op 12.124 ms/op 1.13
phase0 getAttestationDeltas - 250000 worstcase 13.914 ms/op 11.729 ms/op 1.19
phase0 processSlashings - 250000 worstcase 5.9241 ms/op 6.3192 ms/op 0.94
altair processSyncCommitteeUpdates - 250000 272.03 ms/op 299.04 ms/op 0.91
BeaconState.hashTreeRoot - No change 493.00 ns/op 477.00 ns/op 1.03
BeaconState.hashTreeRoot - 1 full validator 62.592 us/op 69.701 us/op 0.90
BeaconState.hashTreeRoot - 32 full validator 741.92 us/op 800.14 us/op 0.93
BeaconState.hashTreeRoot - 512 full validator 7.0436 ms/op 8.1304 ms/op 0.87
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 80.448 us/op 100.70 us/op 0.80
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.1999 ms/op 1.4128 ms/op 0.85
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 16.544 ms/op 17.820 ms/op 0.93
BeaconState.hashTreeRoot - 1 balances 66.099 us/op 69.504 us/op 0.95
BeaconState.hashTreeRoot - 32 balances 582.98 us/op 785.23 us/op 0.74
BeaconState.hashTreeRoot - 512 balances 5.9709 ms/op 6.2146 ms/op 0.96
BeaconState.hashTreeRoot - 250000 balances 99.285 ms/op 98.916 ms/op 1.00
aggregationBits - 2048 els - zipIndexesInBitList 33.974 us/op 25.995 us/op 1.31
regular array get 100000 times 67.384 us/op 53.274 us/op 1.26
wrappedArray get 100000 times 67.391 us/op 55.494 us/op 1.21
arrayWithProxy get 100000 times 33.167 ms/op 33.168 ms/op 1.00
ssz.Root.equals 1.1010 us/op 1.1410 us/op 0.96
byteArrayEquals 1.0820 us/op 1.1150 us/op 0.97
shuffle list - 16384 els 11.360 ms/op 10.582 ms/op 1.07
shuffle list - 250000 els 166.83 ms/op 161.39 ms/op 1.03
processSlot - 1 slots 12.991 us/op 16.208 us/op 0.80
processSlot - 32 slots 1.8033 ms/op 2.0926 ms/op 0.86
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 339.87 us/op 391.68 us/op 0.87
getCommitteeAssignments - req 1 vs - 250000 vc 5.6559 ms/op 5.0360 ms/op 1.12
getCommitteeAssignments - req 100 vs - 250000 vc 7.6064 ms/op 7.3096 ms/op 1.04
getCommitteeAssignments - req 1000 vs - 250000 vc 8.1193 ms/op 7.6537 ms/op 1.06
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 9.8400 ns/op 8.6400 ns/op 1.14
state getBlockRootAtSlot - 250000 vs - 7PWei 1.5252 us/op 1.2200 us/op 1.25
computeProposers - vc 250000 16.302 ms/op 16.980 ms/op 0.96
computeEpochShuffling - vc 250000 173.14 ms/op 167.54 ms/op 1.03
getNextSyncCommittee - vc 250000 270.47 ms/op 294.12 ms/op 0.92

by benchmarkbot/action

@dapplion
Copy link
Contributor

dapplion commented Jan 9, 2023

Good starting point! observables seems sufficient to fullfil the need of events here. To safely merge this we need high observability of the metrics of the worker. This blog is great at how to collect basically all data points we need from the worker https://prog.world/monitoring-multithreaded-node-js-applications/

Regarding metrics the approach of just fetching the metrics str is good for now. Also I think we should move to the pattern of the discv5 library creating the metrics names itself, like we do in gossipsub now.

@wemeetagain
Copy link
Member Author

Screenshot from 2023-01-09 12-36-33

Added some vm-level metrics and appended to the discv5 dashboard

@wemeetagain
Copy link
Member Author

@dapplion @tuyennhv let me know what kind of other metrics you'd like to see

@wemeetagain
Copy link
Member Author

Also I think we should move to the pattern of the discv5 library creating the metrics names itself, like we do in gossipsub now.

ChainSafe/discv5#233 :)

@twoeths
Copy link
Contributor

twoeths commented Jan 16, 2023

@dapplion @tuyennhv let me know what kind of other metrics you'd like to see

looks great 👍 , can't think of other metrics to add

@wemeetagain wemeetagain marked this pull request as ready for review January 17, 2023 21:26
@wemeetagain wemeetagain requested a review from a team as a code owner January 17, 2023 21:26
@wemeetagain wemeetagain merged commit c49ee3c into unstable Jan 20, 2023
@wemeetagain wemeetagain deleted the cayman/threaded-discv5 branch January 20, 2023 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants