Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add historical state regen #6033

Merged
merged 29 commits into from
Jul 30, 2024
Merged

feat: add historical state regen #6033

merged 29 commits into from
Jul 30, 2024

Conversation

wemeetagain
Copy link
Member

@wemeetagain wemeetagain commented Oct 11, 2023 •
edited
Loading

Motivation

Description

  • Add a historical state regen module which can regenerate old states
  • Optional to avoid breaking unrelated tests / mocks
  • Uses a dedicated worker thread with a JobFnQueue inside
  • nodejs, state transition, job queue metrics wired up
  • Add historical state dashboard, which includes VM, job queue, and stfn panels
  • Update VM dashboard to include historical state worker panels and update cross-worker panels
  • PubkeyIndexMap persisted between runs for faster repeated use
  • Deployed to feat3 on Jan 22

TODO

  • pull out any necessary configurability to CLI level - no CLI configurability is needed (?)
  • tests
  • dashboard

@wemeetagain wemeetagain requested a review from a team as a code owner October 11, 2023 16:57
@github-actions
Copy link
Contributor

github-actions bot commented Oct 11, 2023 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: bb8fa49 Previous: 59f72d0 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.0047 ms/op 2.8144 ms/op 0.71
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 44.379 us/op 58.733 us/op 0.76
BLS verify - blst 1.4394 ms/op 1.0293 ms/op 1.40
BLS verifyMultipleSignatures 3 - blst 2.2384 ms/op 1.5085 ms/op 1.48
BLS verifyMultipleSignatures 8 - blst 2.4675 ms/op 2.2218 ms/op 1.11
BLS verifyMultipleSignatures 32 - blst 7.4301 ms/op 7.0741 ms/op 1.05
BLS verifyMultipleSignatures 64 - blst 10.596 ms/op 12.137 ms/op 0.87
BLS verifyMultipleSignatures 128 - blst 16.714 ms/op 20.810 ms/op 0.80
BLS deserializing 10000 signatures 682.46 ms/op 739.80 ms/op 0.92
BLS deserializing 100000 signatures 6.8180 s/op 7.6259 s/op 0.89
BLS verifyMultipleSignatures - same message - 3 - blst 1.2572 ms/op 1.0784 ms/op 1.17
BLS verifyMultipleSignatures - same message - 8 - blst 1.1432 ms/op 1.2278 ms/op 0.93
BLS verifyMultipleSignatures - same message - 32 - blst 1.6899 ms/op 1.7701 ms/op 0.95
BLS verifyMultipleSignatures - same message - 64 - blst 2.6680 ms/op 2.6861 ms/op 0.99
BLS verifyMultipleSignatures - same message - 128 - blst 4.4026 ms/op 4.3882 ms/op 1.00
BLS aggregatePubkeys 32 - blst 19.475 us/op 21.434 us/op 0.91
BLS aggregatePubkeys 128 - blst 68.147 us/op 76.979 us/op 0.89
notSeenSlots=1 numMissedVotes=1 numBadVotes=10 75.714 ms/op 101.51 ms/op 0.75
notSeenSlots=1 numMissedVotes=0 numBadVotes=4 68.768 ms/op 67.655 ms/op 1.02
notSeenSlots=2 numMissedVotes=1 numBadVotes=10 37.164 ms/op 40.916 ms/op 0.91
getSlashingsAndExits - default max 95.665 us/op 166.98 us/op 0.57
getSlashingsAndExits - 2k 397.85 us/op 496.12 us/op 0.80
proposeBlockBody type=full, size=empty 5.8558 ms/op 6.7414 ms/op 0.87
isKnown best case - 1 super set check 304.00 ns/op 500.00 ns/op 0.61
isKnown normal case - 2 super set checks 288.00 ns/op 535.00 ns/op 0.54
isKnown worse case - 16 super set checks 304.00 ns/op 442.00 ns/op 0.69
InMemoryCheckpointStateCache - add get delete 5.4830 us/op 6.7200 us/op 0.82
validate api signedAggregateAndProof - struct 1.8052 ms/op 1.7594 ms/op 1.03
validate gossip signedAggregateAndProof - struct 1.9093 ms/op 1.6286 ms/op 1.17
validate gossip attestation - vc 640000 1.0628 ms/op 1.1567 ms/op 0.92
batch validate gossip attestation - vc 640000 - chunk 32 129.02 us/op 141.22 us/op 0.91
batch validate gossip attestation - vc 640000 - chunk 64 111.53 us/op 125.17 us/op 0.89
batch validate gossip attestation - vc 640000 - chunk 128 109.03 us/op 114.95 us/op 0.95
batch validate gossip attestation - vc 640000 - chunk 256 108.69 us/op 117.75 us/op 0.92
pickEth1Vote - no votes 1.0222 ms/op 1.1676 ms/op 0.88
pickEth1Vote - max votes 7.6123 ms/op 12.274 ms/op 0.62
pickEth1Vote - Eth1Data hashTreeRoot value x2048 11.643 ms/op 17.013 ms/op 0.68
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 19.004 ms/op 29.311 ms/op 0.65
pickEth1Vote - Eth1Data fastSerialize value x2048 467.84 us/op 558.93 us/op 0.84
pickEth1Vote - Eth1Data fastSerialize tree x2048 3.6400 ms/op 4.5172 ms/op 0.81
bytes32 toHexString 430.00 ns/op 483.00 ns/op 0.89
bytes32 Buffer.toString(hex) 240.00 ns/op 256.00 ns/op 0.94
bytes32 Buffer.toString(hex) from Uint8Array 360.00 ns/op 384.00 ns/op 0.94
bytes32 Buffer.toString(hex) + 0x 251.00 ns/op 278.00 ns/op 0.90
Object access 1 prop 0.13700 ns/op 0.14400 ns/op 0.95
Map access 1 prop 0.13100 ns/op 0.13400 ns/op 0.98
Object get x1000 5.9030 ns/op 6.0310 ns/op 0.98
Map get x1000 6.3930 ns/op 6.7980 ns/op 0.94
Object set x1000 33.167 ns/op 77.701 ns/op 0.43
Map set x1000 24.307 ns/op 51.335 ns/op 0.47
Return object 10000 times 0.28960 ns/op 0.32530 ns/op 0.89
Throw Error 10000 times 3.3706 us/op 4.0236 us/op 0.84
fastMsgIdFn sha256 / 200 bytes 2.1630 us/op 2.5380 us/op 0.85
fastMsgIdFn h32 xxhash / 200 bytes 236.00 ns/op 331.00 ns/op 0.71
fastMsgIdFn h64 xxhash / 200 bytes 270.00 ns/op 312.00 ns/op 0.87
fastMsgIdFn sha256 / 1000 bytes 7.1470 us/op 8.2050 us/op 0.87
fastMsgIdFn h32 xxhash / 1000 bytes 365.00 ns/op 423.00 ns/op 0.86
fastMsgIdFn h64 xxhash / 1000 bytes 341.00 ns/op 369.00 ns/op 0.92
fastMsgIdFn sha256 / 10000 bytes 63.363 us/op 68.185 us/op 0.93
fastMsgIdFn h32 xxhash / 10000 bytes 1.8070 us/op 1.9830 us/op 0.91
fastMsgIdFn h64 xxhash / 10000 bytes 1.1940 us/op 1.3210 us/op 0.90
send data - 1000 256B messages 12.196 ms/op 14.074 ms/op 0.87
send data - 1000 512B messages 16.340 ms/op 18.766 ms/op 0.87
send data - 1000 1024B messages 26.685 ms/op 27.920 ms/op 0.96
send data - 1000 1200B messages 26.413 ms/op 33.507 ms/op 0.79
send data - 1000 2048B messages 30.980 ms/op 36.791 ms/op 0.84
send data - 1000 4096B messages 31.351 ms/op 37.187 ms/op 0.84
send data - 1000 16384B messages 70.633 ms/op 73.300 ms/op 0.96
send data - 1000 65536B messages 216.99 ms/op 244.60 ms/op 0.89
enrSubnets - fastDeserialize 64 bits 1.0370 us/op 1.3780 us/op 0.75
enrSubnets - ssz BitVector 64 bits 349.00 ns/op 466.00 ns/op 0.75
enrSubnets - fastDeserialize 4 bits 152.00 ns/op 186.00 ns/op 0.82
enrSubnets - ssz BitVector 4 bits 343.00 ns/op 411.00 ns/op 0.83
prioritizePeers score -10:0 att 32-0.1 sync 2-0 155.68 us/op 225.14 us/op 0.69
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 139.51 us/op 189.71 us/op 0.74
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 290.91 us/op 398.79 us/op 0.73
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 531.92 us/op 595.66 us/op 0.89
prioritizePeers score 0:0 att 64-1 sync 4-1 659.40 us/op 1.1148 ms/op 0.59
array of 16000 items push then shift 1.6620 us/op 1.8577 us/op 0.89
LinkedList of 16000 items push then shift 7.2680 ns/op 8.7960 ns/op 0.83
array of 16000 items push then pop 114.98 ns/op 155.81 ns/op 0.74
LinkedList of 16000 items push then pop 6.9230 ns/op 8.2190 ns/op 0.84
array of 24000 items push then shift 2.3600 us/op 3.2274 us/op 0.73
LinkedList of 24000 items push then shift 6.9110 ns/op 17.365 ns/op 0.40
array of 24000 items push then pop 129.91 ns/op 268.90 ns/op 0.48
LinkedList of 24000 items push then pop 6.7250 ns/op 10.238 ns/op 0.66
intersect bitArray bitLen 8 6.2520 ns/op 7.8480 ns/op 0.80
intersect array and set length 8 43.939 ns/op 112.91 ns/op 0.39
intersect bitArray bitLen 128 29.073 ns/op 34.438 ns/op 0.84
intersect array and set length 128 650.22 ns/op 1.3290 us/op 0.49
bitArray.getTrueBitIndexes() bitLen 128 2.4560 us/op 2.7600 us/op 0.89
bitArray.getTrueBitIndexes() bitLen 248 3.4650 us/op 3.9640 us/op 0.87
bitArray.getTrueBitIndexes() bitLen 512 10.167 us/op 8.1840 us/op 1.24
Buffer.concat 32 items 907.00 ns/op 988.00 ns/op 0.92
Uint8Array.set 32 items 2.0360 us/op 1.6320 us/op 1.25
Buffer.copy 2.1340 us/op 1.7750 us/op 1.20
Uint8Array.set - with subarray 2.8080 us/op 2.7890 us/op 1.01
Uint8Array.set - without subarray 1.6290 us/op 2.0330 us/op 0.80
getUint32 - dataview 230.00 ns/op 315.00 ns/op 0.73
getUint32 - manual 145.00 ns/op 260.00 ns/op 0.56
Set add up to 64 items then delete first 2.1090 us/op 2.7753 us/op 0.76
OrderedSet add up to 64 items then delete first 3.1543 us/op 3.9113 us/op 0.81
Set add up to 64 items then delete last 2.3932 us/op 3.6145 us/op 0.66
OrderedSet add up to 64 items then delete last 3.7489 us/op 5.1222 us/op 0.73
Set add up to 64 items then delete middle 2.4483 us/op 2.7298 us/op 0.90
OrderedSet add up to 64 items then delete middle 5.0520 us/op 6.0286 us/op 0.84
Set add up to 128 items then delete first 4.9201 us/op 6.8297 us/op 0.72
OrderedSet add up to 128 items then delete first 7.0971 us/op 10.442 us/op 0.68
Set add up to 128 items then delete last 4.8271 us/op 5.5755 us/op 0.87
OrderedSet add up to 128 items then delete last 7.3519 us/op 9.9522 us/op 0.74
Set add up to 128 items then delete middle 4.7540 us/op 5.2590 us/op 0.90
OrderedSet add up to 128 items then delete middle 13.080 us/op 14.926 us/op 0.88
Set add up to 256 items then delete first 10.102 us/op 13.789 us/op 0.73
OrderedSet add up to 256 items then delete first 14.544 us/op 20.545 us/op 0.71
Set add up to 256 items then delete last 9.5392 us/op 12.554 us/op 0.76
OrderedSet add up to 256 items then delete last 14.909 us/op 15.758 us/op 0.95
Set add up to 256 items then delete middle 9.5571 us/op 10.667 us/op 0.90
OrderedSet add up to 256 items then delete middle 38.959 us/op 46.727 us/op 0.83
transfer serialized Status (84 B) 1.3950 us/op 1.5170 us/op 0.92
copy serialized Status (84 B) 1.1360 us/op 1.3080 us/op 0.87
transfer serialized SignedVoluntaryExit (112 B) 1.5340 us/op 1.6990 us/op 0.90
copy serialized SignedVoluntaryExit (112 B) 1.2230 us/op 1.3360 us/op 0.92
transfer serialized ProposerSlashing (416 B) 1.9780 us/op 2.1890 us/op 0.90
copy serialized ProposerSlashing (416 B) 2.0020 us/op 2.2500 us/op 0.89
transfer serialized Attestation (485 B) 2.0080 us/op 2.4170 us/op 0.83
copy serialized Attestation (485 B) 2.0440 us/op 2.2720 us/op 0.90
transfer serialized AttesterSlashing (33232 B) 2.8310 us/op 2.7010 us/op 1.05
copy serialized AttesterSlashing (33232 B) 6.0700 us/op 8.8250 us/op 0.69
transfer serialized Small SignedBeaconBlock (128000 B) 3.8330 us/op 2.8150 us/op 1.36
copy serialized Small SignedBeaconBlock (128000 B) 14.040 us/op 24.634 us/op 0.57
transfer serialized Avg SignedBeaconBlock (200000 B) 4.2260 us/op 3.6840 us/op 1.15
copy serialized Avg SignedBeaconBlock (200000 B) 20.134 us/op 25.755 us/op 0.78
transfer serialized BlobsSidecar (524380 B) 2.9910 us/op 2.6350 us/op 1.14
copy serialized BlobsSidecar (524380 B) 77.520 us/op 96.379 us/op 0.80
transfer serialized Big SignedBeaconBlock (1000000 B) 3.1340 us/op 3.8180 us/op 0.82
copy serialized Big SignedBeaconBlock (1000000 B) 159.48 us/op 183.77 us/op 0.87
pass gossip attestations to forkchoice per slot 3.2519 ms/op 3.5957 ms/op 0.90
forkChoice updateHead vc 100000 bc 64 eq 0 594.34 us/op 552.73 us/op 1.08
forkChoice updateHead vc 600000 bc 64 eq 0 3.0001 ms/op 3.2941 ms/op 0.91
forkChoice updateHead vc 1000000 bc 64 eq 0 5.1774 ms/op 6.1188 ms/op 0.85
forkChoice updateHead vc 600000 bc 320 eq 0 3.0030 ms/op 3.3470 ms/op 0.90
forkChoice updateHead vc 600000 bc 1200 eq 0 3.0512 ms/op 4.8470 ms/op 0.63
forkChoice updateHead vc 600000 bc 7200 eq 0 3.9113 ms/op 4.5896 ms/op 0.85
forkChoice updateHead vc 600000 bc 64 eq 1000 10.500 ms/op 11.543 ms/op 0.91
forkChoice updateHead vc 600000 bc 64 eq 10000 10.647 ms/op 11.883 ms/op 0.90
forkChoice updateHead vc 600000 bc 64 eq 300000 13.843 ms/op 23.222 ms/op 0.60
computeDeltas 500000 validators 300 proto nodes 3.4649 ms/op 4.0673 ms/op 0.85
computeDeltas 500000 validators 1200 proto nodes 3.4483 ms/op 4.2413 ms/op 0.81
computeDeltas 500000 validators 7200 proto nodes 3.4200 ms/op 4.6100 ms/op 0.74
computeDeltas 750000 validators 300 proto nodes 5.1447 ms/op 6.7896 ms/op 0.76
computeDeltas 750000 validators 1200 proto nodes 5.2273 ms/op 6.7521 ms/op 0.77
computeDeltas 750000 validators 7200 proto nodes 5.1799 ms/op 6.3722 ms/op 0.81
computeDeltas 1400000 validators 300 proto nodes 9.6368 ms/op 12.043 ms/op 0.80
computeDeltas 1400000 validators 1200 proto nodes 9.5534 ms/op 11.148 ms/op 0.86
computeDeltas 1400000 validators 7200 proto nodes 9.7561 ms/op 11.517 ms/op 0.85
computeDeltas 2100000 validators 300 proto nodes 14.647 ms/op 15.815 ms/op 0.93
computeDeltas 2100000 validators 1200 proto nodes 14.788 ms/op 15.677 ms/op 0.94
computeDeltas 2100000 validators 7200 proto nodes 14.860 ms/op 15.222 ms/op 0.98
altair processAttestation - 250000 vs - 7PWei normalcase 1.5750 ms/op 1.7476 ms/op 0.90
altair processAttestation - 250000 vs - 7PWei worstcase 2.3775 ms/op 2.6841 ms/op 0.89
altair processAttestation - setStatus - 1/6 committees join 87.642 us/op 92.773 us/op 0.94
altair processAttestation - setStatus - 1/3 committees join 178.88 us/op 180.33 us/op 0.99
altair processAttestation - setStatus - 1/2 committees join 247.18 us/op 265.68 us/op 0.93
altair processAttestation - setStatus - 2/3 committees join 332.73 us/op 333.22 us/op 1.00
altair processAttestation - setStatus - 4/5 committees join 496.10 us/op 507.92 us/op 0.98
altair processAttestation - setStatus - 100% committees join 586.84 us/op 604.03 us/op 0.97
altair processBlock - 250000 vs - 7PWei normalcase 5.1146 ms/op 4.5631 ms/op 1.12
altair processBlock - 250000 vs - 7PWei normalcase hashState 24.950 ms/op 32.407 ms/op 0.77
altair processBlock - 250000 vs - 7PWei worstcase 39.152 ms/op 39.460 ms/op 0.99
altair processBlock - 250000 vs - 7PWei worstcase hashState 83.361 ms/op 85.532 ms/op 0.97
phase0 processBlock - 250000 vs - 7PWei normalcase 2.8043 ms/op 2.7890 ms/op 1.01
phase0 processBlock - 250000 vs - 7PWei worstcase 31.735 ms/op 28.085 ms/op 1.13
altair processEth1Data - 250000 vs - 7PWei normalcase 308.31 us/op 467.26 us/op 0.66
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 8.0680 us/op 7.1460 us/op 1.13
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 30.391 us/op 30.398 us/op 1.00
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 11.093 us/op 13.232 us/op 0.84
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 8.7520 us/op 8.4650 us/op 1.03
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 70.060 us/op 119.36 us/op 0.59
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 1.0778 ms/op 916.20 us/op 1.18
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 1.5569 ms/op 1.2645 ms/op 1.23
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 904.86 us/op 1.1508 ms/op 0.79
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 2.4844 ms/op 3.0927 ms/op 0.80
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 1.5291 ms/op 2.1243 ms/op 0.72
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 3.8221 ms/op 4.8473 ms/op 0.79
Tree 40 250000 create 203.04 ms/op 483.00 ms/op 0.42
Tree 40 250000 get(125000) 140.51 ns/op 179.48 ns/op 0.78
Tree 40 250000 set(125000) 627.13 ns/op 1.6828 us/op 0.37
Tree 40 250000 toArray() 15.328 ms/op 25.753 ms/op 0.60
Tree 40 250000 iterate all - toArray() + loop 15.475 ms/op 26.396 ms/op 0.59
Tree 40 250000 iterate all - get(i) 50.189 ms/op 74.884 ms/op 0.67
MutableVector 250000 create 10.827 ms/op 16.958 ms/op 0.64
MutableVector 250000 get(125000) 6.5120 ns/op 7.9280 ns/op 0.82
MutableVector 250000 set(125000) 218.74 ns/op 300.55 ns/op 0.73
MutableVector 250000 toArray() 3.8412 ms/op 5.0849 ms/op 0.76
MutableVector 250000 iterate all - toArray() + loop 3.7757 ms/op 5.4807 ms/op 0.69
MutableVector 250000 iterate all - get(i) 1.5639 ms/op 1.7584 ms/op 0.89
Array 250000 create 2.9853 ms/op 4.8963 ms/op 0.61
Array 250000 clone - spread 1.5255 ms/op 3.6722 ms/op 0.42
Array 250000 get(125000) 0.39200 ns/op 0.51300 ns/op 0.76
Array 250000 set(125000) 0.41800 ns/op 0.53800 ns/op 0.78
Array 250000 iterate all - loop 106.08 us/op 123.48 us/op 0.86
effectiveBalanceIncrements clone Uint8Array 300000 39.044 us/op 78.460 us/op 0.50
effectiveBalanceIncrements clone MutableVector 300000 124.00 ns/op 167.00 ns/op 0.74
effectiveBalanceIncrements rw all Uint8Array 300000 195.72 us/op 221.00 us/op 0.89
effectiveBalanceIncrements rw all MutableVector 300000 70.850 ms/op 179.53 ms/op 0.39
phase0 afterProcessEpoch - 250000 vs - 7PWei 87.138 ms/op 99.330 ms/op 0.88
Array.fill - length 1000000 3.4439 ms/op 7.9360 ms/op 0.43
Array push - length 1000000 21.190 ms/op 26.478 ms/op 0.80
Array.get 0.28615 ns/op 0.31746 ns/op 0.90
Uint8Array.get 0.46409 ns/op 0.49242 ns/op 0.94
phase0 beforeProcessEpoch - 250000 vs - 7PWei 27.160 ms/op 30.087 ms/op 0.90
altair processEpoch - mainnet_e81889 425.38 ms/op 385.46 ms/op 1.10
mainnet_e81889 - altair beforeProcessEpoch 38.605 ms/op 36.538 ms/op 1.06
mainnet_e81889 - altair processJustificationAndFinalization 13.293 us/op 18.921 us/op 0.70
mainnet_e81889 - altair processInactivityUpdates 7.1656 ms/op 6.3351 ms/op 1.13
mainnet_e81889 - altair processRewardsAndPenalties 48.096 ms/op 56.362 ms/op 0.85
mainnet_e81889 - altair processRegistryUpdates 2.3470 us/op 3.3880 us/op 0.69
mainnet_e81889 - altair processSlashings 1.0980 us/op 847.00 ns/op 1.30
mainnet_e81889 - altair processEth1DataReset 930.00 ns/op 732.00 ns/op 1.27
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.1365 ms/op 2.3132 ms/op 0.92
mainnet_e81889 - altair processSlashingsReset 2.9310 us/op 8.3150 us/op 0.35
mainnet_e81889 - altair processRandaoMixesReset 3.6070 us/op 8.5160 us/op 0.42
mainnet_e81889 - altair processHistoricalRootsUpdate 411.00 ns/op 1.2700 us/op 0.32
mainnet_e81889 - altair processParticipationFlagUpdates 2.5380 us/op 3.6310 us/op 0.70
mainnet_e81889 - altair processSyncCommitteeUpdates 520.00 ns/op 1.0030 us/op 0.52
mainnet_e81889 - altair afterProcessEpoch 92.329 ms/op 106.20 ms/op 0.87
capella processEpoch - mainnet_e217614 1.1733 s/op 1.9567 s/op 0.60
mainnet_e217614 - capella beforeProcessEpoch 135.51 ms/op 174.03 ms/op 0.78
mainnet_e217614 - capella processJustificationAndFinalization 23.774 us/op 32.672 us/op 0.73
mainnet_e217614 - capella processInactivityUpdates 18.297 ms/op 23.799 ms/op 0.77
mainnet_e217614 - capella processRewardsAndPenalties 259.80 ms/op 293.13 ms/op 0.89
mainnet_e217614 - capella processRegistryUpdates 18.942 us/op 22.909 us/op 0.83
mainnet_e217614 - capella processSlashings 387.00 ns/op 821.00 ns/op 0.47
mainnet_e217614 - capella processEth1DataReset 320.00 ns/op 1.2620 us/op 0.25
mainnet_e217614 - capella processEffectiveBalanceUpdates 12.707 ms/op 25.334 ms/op 0.50
mainnet_e217614 - capella processSlashingsReset 5.3460 us/op 8.8370 us/op 0.60
mainnet_e217614 - capella processRandaoMixesReset 6.8520 us/op 14.103 us/op 0.49
mainnet_e217614 - capella processHistoricalRootsUpdate 586.00 ns/op 1.2750 us/op 0.46
mainnet_e217614 - capella processParticipationFlagUpdates 2.3660 us/op 8.0520 us/op 0.29
mainnet_e217614 - capella afterProcessEpoch 274.58 ms/op 412.89 ms/op 0.67
phase0 processEpoch - mainnet_e58758 435.24 ms/op 800.70 ms/op 0.54
mainnet_e58758 - phase0 beforeProcessEpoch 110.77 ms/op 218.61 ms/op 0.51
mainnet_e58758 - phase0 processJustificationAndFinalization 21.505 us/op 41.852 us/op 0.51
mainnet_e58758 - phase0 processRewardsAndPenalties 39.973 ms/op 58.574 ms/op 0.68
mainnet_e58758 - phase0 processRegistryUpdates 10.091 us/op 20.839 us/op 0.48
mainnet_e58758 - phase0 processSlashings 562.00 ns/op 1.3770 us/op 0.41
mainnet_e58758 - phase0 processEth1DataReset 319.00 ns/op 1.3080 us/op 0.24
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.0982 ms/op 2.0452 ms/op 0.54
mainnet_e58758 - phase0 processSlashingsReset 4.0760 us/op 7.3930 us/op 0.55
mainnet_e58758 - phase0 processRandaoMixesReset 6.3760 us/op 13.948 us/op 0.46
mainnet_e58758 - phase0 processHistoricalRootsUpdate 413.00 ns/op 1.1550 us/op 0.36
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.5730 us/op 20.240 us/op 0.18
mainnet_e58758 - phase0 afterProcessEpoch 78.206 ms/op 137.32 ms/op 0.57
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2082 ms/op 2.0470 ms/op 0.59
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.6702 ms/op 7.7077 ms/op 0.35
altair processInactivityUpdates - 250000 normalcase 21.794 ms/op 29.936 ms/op 0.73
altair processInactivityUpdates - 250000 worstcase 21.671 ms/op 34.112 ms/op 0.64
phase0 processRegistryUpdates - 250000 normalcase 8.0150 us/op 20.105 us/op 0.40
phase0 processRegistryUpdates - 250000 badcase_full_deposits 368.82 us/op 507.66 us/op 0.73
phase0 processRegistryUpdates - 250000 worstcase 0.5 128.68 ms/op 168.90 ms/op 0.76
altair processRewardsAndPenalties - 250000 normalcase 48.016 ms/op 46.280 ms/op 1.04
altair processRewardsAndPenalties - 250000 worstcase 42.888 ms/op 46.715 ms/op 0.92
phase0 getAttestationDeltas - 250000 normalcase 8.1329 ms/op 12.549 ms/op 0.65
phase0 getAttestationDeltas - 250000 worstcase 8.4655 ms/op 9.9824 ms/op 0.85
phase0 processSlashings - 250000 worstcase 121.55 us/op 110.95 us/op 1.10
altair processSyncCommitteeUpdates - 250000 116.36 ms/op 146.44 ms/op 0.79
BeaconState.hashTreeRoot - No change 266.00 ns/op 358.00 ns/op 0.74
BeaconState.hashTreeRoot - 1 full validator 114.02 us/op 129.24 us/op 0.88
BeaconState.hashTreeRoot - 32 full validator 1.1610 ms/op 1.7672 ms/op 0.66
BeaconState.hashTreeRoot - 512 full validator 13.625 ms/op 15.259 ms/op 0.89
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 155.85 us/op 189.02 us/op 0.82
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 2.2020 ms/op 2.7816 ms/op 0.79
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 30.107 ms/op 34.706 ms/op 0.87
BeaconState.hashTreeRoot - 1 balances 127.06 us/op 140.65 us/op 0.90
BeaconState.hashTreeRoot - 32 balances 1.1569 ms/op 1.5437 ms/op 0.75
BeaconState.hashTreeRoot - 512 balances 10.029 ms/op 10.871 ms/op 0.92
BeaconState.hashTreeRoot - 250000 balances 173.14 ms/op 164.90 ms/op 1.05
aggregationBits - 2048 els - zipIndexesInBitList 23.682 us/op 28.413 us/op 0.83
byteArrayEquals 32 52.092 ns/op 56.290 ns/op 0.93
Buffer.compare 32 19.618 ns/op 17.995 ns/op 1.09
byteArrayEquals 1024 1.5541 us/op 1.6530 us/op 0.94
Buffer.compare 1024 27.173 ns/op 26.526 ns/op 1.02
byteArrayEquals 16384 24.675 us/op 26.347 us/op 0.94
Buffer.compare 16384 191.67 ns/op 189.44 ns/op 1.01
byteArrayEquals 123687377 184.12 ms/op 212.68 ms/op 0.87
Buffer.compare 123687377 6.5002 ms/op 12.212 ms/op 0.53
byteArrayEquals 32 - diff last byte 51.920 ns/op 59.934 ns/op 0.87
Buffer.compare 32 - diff last byte 17.071 ns/op 19.879 ns/op 0.86
byteArrayEquals 1024 - diff last byte 1.5742 us/op 1.8055 us/op 0.87
Buffer.compare 1024 - diff last byte 25.138 ns/op 29.804 ns/op 0.84
byteArrayEquals 16384 - diff last byte 25.026 us/op 28.535 us/op 0.88
Buffer.compare 16384 - diff last byte 197.98 ns/op 227.59 ns/op 0.87
byteArrayEquals 123687377 - diff last byte 188.18 ms/op 221.27 ms/op 0.85
Buffer.compare 123687377 - diff last byte 6.0954 ms/op 11.603 ms/op 0.53
byteArrayEquals 32 - random bytes 4.9150 ns/op 5.9520 ns/op 0.83
Buffer.compare 32 - random bytes 16.499 ns/op 20.094 ns/op 0.82
byteArrayEquals 1024 - random bytes 4.9420 ns/op 5.8630 ns/op 0.84
Buffer.compare 1024 - random bytes 16.450 ns/op 22.471 ns/op 0.73
byteArrayEquals 16384 - random bytes 4.9140 ns/op 6.1990 ns/op 0.79
Buffer.compare 16384 - random bytes 16.513 ns/op 19.813 ns/op 0.83
byteArrayEquals 123687377 - random bytes 6.0600 ns/op 7.8500 ns/op 0.77
Buffer.compare 123687377 - random bytes 17.640 ns/op 22.730 ns/op 0.78
regular array get 100000 times 31.545 us/op 41.934 us/op 0.75
wrappedArray get 100000 times 31.370 us/op 41.472 us/op 0.76
arrayWithProxy get 100000 times 13.210 ms/op 19.437 ms/op 0.68
ssz.Root.equals 45.581 ns/op 55.656 ns/op 0.82
byteArrayEquals 44.956 ns/op 54.337 ns/op 0.83
Buffer.compare 10.250 ns/op 12.061 ns/op 0.85
shuffle list - 16384 els 6.2957 ms/op 7.3299 ms/op 0.86
shuffle list - 250000 els 92.077 ms/op 112.99 ms/op 0.81
processSlot - 1 slots 14.254 us/op 22.444 us/op 0.64
processSlot - 32 slots 3.2231 ms/op 4.3905 ms/op 0.73
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 42.216 ms/op 54.205 ms/op 0.78
getCommitteeAssignments - req 1 vs - 250000 vc 2.1382 ms/op 2.5613 ms/op 0.83
getCommitteeAssignments - req 100 vs - 250000 vc 4.1095 ms/op 5.3312 ms/op 0.77
getCommitteeAssignments - req 1000 vs - 250000 vc 4.3760 ms/op 5.2454 ms/op 0.83
findModifiedValidators - 10000 modified validators 245.48 ms/op 625.53 ms/op 0.39
findModifiedValidators - 1000 modified validators 160.38 ms/op 450.90 ms/op 0.36
findModifiedValidators - 100 modified validators 159.12 ms/op 345.77 ms/op 0.46
findModifiedValidators - 10 modified validators 134.45 ms/op 286.71 ms/op 0.47
findModifiedValidators - 1 modified validators 141.64 ms/op 303.21 ms/op 0.47
findModifiedValidators - no difference 146.10 ms/op 324.18 ms/op 0.45
compare ViewDUs 3.1927 s/op 4.3672 s/op 0.73
compare each validator Uint8Array 1.5798 s/op 2.2950 s/op 0.69
compare ViewDU to Uint8Array 948.44 ms/op 1.6281 s/op 0.58
migrate state 1000000 validators, 24 modified, 0 new 553.89 ms/op 683.43 ms/op 0.81
migrate state 1000000 validators, 1700 modified, 1000 new 894.03 ms/op 1.0534 s/op 0.85
migrate state 1000000 validators, 3400 modified, 2000 new 1.1205 s/op 1.2155 s/op 0.92
migrate state 1500000 validators, 24 modified, 0 new 659.84 ms/op 762.66 ms/op 0.87
migrate state 1500000 validators, 1700 modified, 1000 new 864.49 ms/op 832.10 ms/op 1.04
migrate state 1500000 validators, 3400 modified, 2000 new 1.0142 s/op 1.0789 s/op 0.94
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.8400 ns/op 4.7300 ns/op 1.02
state getBlockRootAtSlot - 250000 vs - 7PWei 932.36 ns/op 949.94 ns/op 0.98
computeProposers - vc 250000 9.2434 ms/op 7.6173 ms/op 1.21
computeEpochShuffling - vc 250000 98.265 ms/op 96.464 ms/op 1.02
getNextSyncCommittee - vc 250000 143.67 ms/op 123.15 ms/op 1.17
computeSigningRoot for AttestationData 21.426 us/op 24.217 us/op 0.88
hash AttestationData serialized data then Buffer.toString(base64) 1.6382 us/op 1.5446 us/op 1.06
toHexString serialized data 1.0419 us/op 927.19 ns/op 1.12
Buffer.toString(base64) 220.96 ns/op 194.50 ns/op 1.14

by benchmarkbot/action

@wemeetagain
Copy link
Member Author

Running into issues opening the db in the worker thread.

It seems the leveldb way of allowing concurrency is have the consumer pass the pointer to the db handle to another thread. That's not currently possible with the leveldb package as it currently exists.

@wemeetagain
Copy link
Member Author

Converting to draft until the multithreaded database access issue is resolved

@wemeetagain wemeetagain marked this pull request as draft October 17, 2023 14:51
@nflaig nflaig mentioned this pull request Oct 31, 2023
Closed
@matthewkeil matthewkeil mentioned this pull request Nov 5, 2023
Closed
@philknows philknows mentioned this pull request Nov 7, 2023
Closed
@matthewkeil matthewkeil mentioned this pull request Dec 6, 2023
Merged
@philknows
Copy link
Member

Any update on the status of this PR? Are we blocked on it somewhere externally?

@wemeetagain
Copy link
Member Author

Any update on the status of this PR?

waiting on me to fix this in response to @matthewkeil 's review

@wemeetagain wemeetagain added this to the v1.20.0 milestone Jun 4, 2024
@philknows philknows modified the milestones: v1.20.0, v1.21.0 Jul 10, 2024
@wemeetagain wemeetagain mentioned this pull request Jul 17, 2024
Merged
@wemeetagain
Copy link
Member Author

Ready for another round of review

@nflaig nflaig self-requested a review July 23, 2024 15:03
@twoeths
Copy link
Contributor

twoeths commented Jul 24, 2024 •
edited
Loading

this PR implements the fundamental work for historical state regen on worker thread, however I'm not sure how likely it could be fully functional in production, a request could take so long given archiveStateEpochFrequency = 1024, it could be up to 1024 * 10s ~= 2.8h given ~10s to process 32 blocks of an epoch like what I saw in n-historical state testing

we could explore storing incremental state diff with 2 purposes:

  • save disc space
  • faster time to restart
  • this historical state regen: apply some diffs until the checkpoint state, and reprocess blocks like in this PR

cc @nazarhussain @philknows @wemeetagain

@nazarhussain
Copy link
Contributor

we could explore storing incremental state diff with 2 purposes:

That is already in progress and will be complete very soon.

matthewkeil
matthewkeil approved these changes Jul 30, 2024
View reviewed changes
Copy link
Member

@matthewkeil matthewkeil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I think all the important stuff was addressed 🎸

@twoeths twoeths merged commit c23d70c into unstable Jul 30, 2024
20 checks passed
@twoeths twoeths deleted the cayman/historical-state-regen branch July 30, 2024 06:12
nflaig
nflaig reviewed Jul 30, 2024
View reviewed changes
Copy link
Member

@nflaig nflaig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - just a few minor comments / questions

packages/beacon-node/src/chain/chain.ts
}

return {state: stateSerialized, executionOptimistic: false, finalized: true};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have access to the block here, shouldn't this rather be executionOptimistic: isOptimisticBlock(finalizedBlock), afaik a finalized block can still be optimistic

packages/beacon-node/src/chain/historicalState/worker.ts
Comment on lines +28 to +29
// TODO: Pass options from main thread for logging
// TODO: Logging won't be visible in file loggers
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aren't both of these todos stale? file logging works and opts are passed from main thread via workerData

packages/beacon-node/src/chain/historicalState/worker.ts
regenTime: metricsRegister.histogram({
name: "lodestar_historical_state_regen_time_seconds",
help: "Time to regenerate a historical state in seconds",
// Historical state regen can take up to 3h as of Aug 2024
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We really need to tell people in the release notes that they should reduce archive state frequency if they want access to historical data, do we have a recommended value which keep state regen times below 30s while not blowing up storage requirements?

Also it might be a good time to unhide the flag as per #6908

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in my opinion we should not inform, we need to test this work carefully first
I really want to make sure we have a great performance before we inform
see #6033 (comment)

@twoeths twoeths mentioned this pull request Aug 4, 2024
Open
@wemeetagain
Copy link
Member Author

🎉 This PR is included in v1.21.0 🎉

@nflaig nflaig mentioned this pull request Oct 7, 2024
Merged
1 task
@wemeetagain wemeetagain mentioned this pull request Oct 15, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nflaig nflaig nflaig left review comments

@dapplion dapplion dapplion left review comments

@twoeths twoeths twoeths approved these changes

@matthewkeil matthewkeil matthewkeil approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v1.21.0
Development

Successfully merging this pull request may close these issues.
7 participants
@wemeetagain @philknows @twoeths @nazarhussain @matthewkeil @dapplion @nflaig