feat: asyncAggregateWithRandomness #7204
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## unstable #7204 +/- ##
===========================================
Coverage ? 49.22%
===========================================
Files ? 598
Lines ? 39789
Branches ? 2103
===========================================
Hits ? 19588
Misses ? 20161
Partials ? 40 |
Performance Report✔️ no performance regression detected Full benchmark results
|
twoeths
reviewed
Oct 29, 2024
twoeths
reviewed
Oct 29, 2024
|
there are big improvement of this PR, missed attestations + target were reduced from ~10% to 3-4% on a 
|
wemeetagain
previously approved these changes
Oct 29, 2024
matthewkeil
added
the
status-do-not-merge
Great news! Can we let the metrics soak a bit more before we merge? Just incase anything wonky shows up |
|
Oky, will wait until later today to get more data. I'm hoping to cut a new RC by my end of day today and want to include this. |
matthewkeil
removed
the
status-do-not-merge
|
Everything across the board looks healthier from attestations to scrape duration. The one thing that did stand out though is mesh peers are not as good. Not sure if that is just a function of the node still being relatively fresh and that will stabilize over the next several hours or not... CPU utilization is up a bit but i suppose its just working more efficiently with the multithreading. Wondering if we can tune that even further by changing the number of libuv threads. Can play with that in another PR though. |
I don't think that is an apples-to-apples comparison. your test is using subscribe-all-subnets and other comparable nodes aren't. |
wemeetagain
approved these changes
Oct 29, 2024
matthewkeil
added a commit
that referenced
this pull request
Nov 4, 2024
This reverts commit e31d535.
|
attestation job time decreased significantly since this PR, this is the unstable mainnet node 
|
|
same to "main to worker" latency 
|
|
"worker to main" latency on unstable mainnet node was significantly improved as well 
|
matthewkeil
added a commit
that referenced
this pull request
Nov 6, 2024
This reverts commit e31d535.
matthewkeil
added a commit
that referenced
this pull request
Nov 6, 2024
This reverts commit e31d535.
matthewkeil
added a commit
that referenced
this pull request
Nov 7, 2024
matthewkeil
added a commit
that referenced
this pull request
Nov 7, 2024
|
🎉 This PR is included in v1.23.0 🎉 |
wemeetagain
pushed a commit
that referenced
this pull request
Dec 3, 2024
* chore: update blst to 2.2.0 * feat: make aggregateWithRandomness async * feat: update metrics and remove stale comment * fix: metric collection value * feat: remove duplicate/unused metrics from asyncAggregateWithRandomness
wemeetagain
added a commit
that referenced
this pull request
Dec 20, 2024
* feat: add keymanager endpoint to retrieve proposer config (#7210) * feat: add keymanager endpoint to retrieve proposer config * Do not return empty builder config * Check all builder proposer config values * Fix settings builder config if undefined * Fix builder config parsing * Use ssz type to handle json serialization Default parsing can't handle BigInt * Revert "Use ssz type to handle json serialization" This reverts commit 01fcea7. * Fix boost factor json serialization * Remove unused import * Update test data * Update proposer config test * feat: add mekong network option (#7212) * chore: fix import order with biome syntax (#7211) Fix import order * fix: consistently validate pubkey and throw 404 if not found (#7214) * Throw error if pubkey is unknown when getting graffiti * Consistently validate pubkey and throw 404 if not found * fix: only return local keys from /eth/v1/keystores (#7215) * fix: only return local keys from /eth/v1/keystores * Fix fetching remote keys in node assertion * feat: add and use getBlobsV1 to expedite gossip import (#7134) * hookup the getblobs api to get bob and proof data from el remove unused fix import metrics overhault, test, debugging testing, some feeback fix add nethermind bug dicussion link fix resolve conflicts * deblobs timeout * fix metric * chore: revert async aggregate with randomness (#7218) Revert "feat: asyncAggregateWithRandomness (#7204)" This reverts commit e31d535. * fix: update config for relaunched mekong network (#7220) * fix: light client generating `LightClientUpdate` with wrong length of branches (#7187) * initial commit * Rewrite SyncCommitteeWitnessRepository * Fix finality branch * Update unit test * fix e2e * Review PR --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com> * fix: archive finalized state when shutting down beacon node (#7221) * Fix typo * feat: remove unfinalized pubkey cache (#7230) * Remove unfinalized pubkey cache * lint * Fix unit test * chore: skip web3_provider unit tests (#7252) * fix: prune checkpoint states at syncing time (#7241) * fix: prune checkpoint states at syncing time * fix: lint * fix: check-types in test * fix: sync cached isCompoundingValidatorArr at epoch transition (#7247) * fix: handle outOfRangeData when range sync Deneb (#7249) * fix: handle outOfRangeData for beaconBlocksMaybeBlobsByRange() * fix: lint * fix: archiveBlocks - handle deneb outOfRangeData block * fix: sync cached balance when adding new validator to registry (#7255) * fix: sync cached balance when adding new validator to registry * chore: add more comments * fix: remove persisted checkpoint states from the previous run at startup * fix: do not throw error when trying to prune missing directory (#7257) * docs: update documentation Oct 2024 (#7178) * docs update oct 2024 init * Reconfig quickstart nav and minor fixes * fix lint * spelling fixes * minor fixes and add to wordlist * prettier fix * add to wordlist * sort wordlist * modify dominance to include lighthouse * fix typescript casing and add recommendation * add selection and boost_factor with keymanager notice * update wordlist * remove builder enabled and add keymanager api * spelling --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com> * chore(deps): bump cross-spawn from 7.0.3 to 7.0.6 in /docs (#7268) Bumps [cross-spawn](https://github.com/moxystudio/node-cross-spawn) from 7.0.3 to 7.0.6. - [Changelog](https://github.com/moxystudio/node-cross-spawn/blob/master/CHANGELOG.md) - [Commits](moxystudio/node-cross-spawn@v7.0.3...v7.0.6) --- updated-dependencies: - dependency-name: cross-spawn dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: add error log to notifier if execution client auth failed (#7239) * feat: add error log to notifier if execution client auth failed * Update packages/beacon-node/src/node/notifier.ts --------- Co-authored-by: NC <17676176+ensi321@users.noreply.github.com> * docs: display rcConfig flag on CLI reference page (#7270) * docs: display rcConfig flag on CLI reference page * Update word list * chore: remove prettier as default formatter for all file types (#7275) * chore: unhide flags relevant for devnets / testing (#7271) * feat: debug too many shuffling promises (#7251) * feat: add asyncShufflingCalculation to StateTransitionOpts * feat: add asyncShufflingCalculation to all regen / processSlots consumers * fix: default to false for async shuffling and remove unnecessary props * fix: remove unnecessary flags from stateTransition * feat: implement conditional build of shuffling for prepareNextSlot * fix: spec test bug where shufflingCache is present from BeaconChain constructor * feat: sync build next shuffling if not queued async * fix: use getSync to pull next shuffling correctly * docs: add comment to prepareNextSlot * refactor: rename StateCloneOpts to StateRegenerationOpts * feat: pass asyncShufflingCalculation through to afterProcessEpoch and refactor conditional to run purely sync * docs: add issue number to comment * chore: lint * chore: unpin nodejs version from 22.4 (#6982) * Revert "chore: pin nodejs version to 22.4 (#6964)" This reverts commit f20484b. * Don't revert formatting changes --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com> Co-authored-by: Cayman <caymannava@gmail.com> * chore: update bootnodes file url for holesky and sepolia (#7276) * feat: add `debug_getHistoricalSummaries` endpoint (#7245) * feat: add new getHistoricalSummaries endpoint to debug namespace * Add JSON response * Restructure to use stateId and add proof to response * add test scaffolding * Address feedback * Move getHistoricalSummaries to lodestar namespace * add lodestar namespace unit test * update route name to lodestar namespace * cast state object as Capella state * Lint * json properties need to be lower case * Make it v1 since it's now part of lodestar namespace * Group with other /lodestar endpoints * Simplify beacon node impl * Rename return type * Update test description * Fix variable name --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com> * chore: log sync committee signature errors as `error` (#7283) * fix: update engine_getClientVersionV1 commit encoding (#7282) * fix: check pubkey or validator index known to a state (#7284) * fix: check pubkey or validator index known to a state * chore: add more comments * feat: lodestar script setup (#7254) * feat: lodestar_setup * feat: script_updates + docs * feat: script_addition_in_docs + command_update * Remove duplicate script from docs folder * Minor script updates * Update script to prepare docs and ignore copied file * Update installation page * Wording --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com> * feat: add terminal-sized Electra giraffe banner (#7286) * Create giraffeBanners.ts * Wire in banner * Fix file name * lint * Address @nflaig's comment --------- Co-authored-by: NC <17676176+ensi321@users.noreply.github.com> * chore: pin nodejs version to 22.4 (#7291) Revert "chore: unpin nodejs version from 22.4 (#6982)" This reverts commit 69ae688. * feat: expose `DOMAIN_APPLICATION_MASK` in config/spec api (#7296) * feat: expose DOMAIN_APPLICATION_MASK in config/spec api * Lint * feat: make `MAX_REQUEST_BLOB_SIDECARS` and `MAX_BLOBS_PER_BLOCK` configurable (#7294) * Init commit * Fix check-types * Add comment on how MAX_REQUEST_BLOB_SIDECARS is calculated * Ensure proper config object is passed * Address comment --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com> * feat: use `BLOB_SIDECAR_SUBNET_COUNT` to configure blob subnets (#7297) feat: use BLOB_SIDECAR_SUBNET_COUNT to configure blob subnets * chore: log sync aggregate participants when producing beacon block body (#7300) * chore: log sync aggregate participants when producing beacon block body * Use isForkLightClient instead of ForkSeq * Fix produce block unit tests * chore: print graffiti when producing beacon block body (#7303) * fix: warn if engine / builder failed to produce block within cutoff time (#7305) * feat: add kzg commitment length check when validating gossip blocks (#7302) * feat: add blob sidecar index check (#7313) Validate blobSidecar index * fix: fix blob sidecar index check (#7315) Fix index check * chore: fix format of printed graffiti from hex to utf-8 (#7306) * chore: fix format of printed graffiti from hex to utf-8 * Use Buffer.from no copy with offset * docs: batch commit typos and update contributor readme (#7312) * batch commit typos and update contributor readme * update donation text Co-authored-by: Nico Flaig <nflaig@protonmail.com> * correct spelling Co-authored-by: Nico Flaig <nflaig@protonmail.com> --------- Co-authored-by: Nico Flaig <nflaig@protonmail.com> * chore: remove trailing null bytes from printed graffiti (#7320) * chore: remove trailing null bytes from printed graffiti * Use replaceAll instead of regex * chore: unpin nodejs version from 22.4 (#7324) Revert "chore: pin nodejs version to 22.4 (#7291)" This reverts commit 99794d3. * chore: bump package versions to 1.24.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Nico Flaig <nflaig@protonmail.com> Co-authored-by: Nazar Hussain <nazarhussain@gmail.com> Co-authored-by: g11tech <develop@g11tech.io> Co-authored-by: Matthew Keil <me@matthewkeil.com> Co-authored-by: NC <17676176+ensi321@users.noreply.github.com> Co-authored-by: twoeths <tuyen@chainsafe.io> Co-authored-by: Phil Ngo <58080811+philknows@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: acolytec3 <17355484+acolytec3@users.noreply.github.com> Co-authored-by: Varun Guleria <152203177+varunguleriaCodes@users.noreply.github.com> Co-authored-by: ClockworkYuzu <yvetteheh@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Resolves #7197
Moves signature checks and aggregation for unaggregated attestations off main thread. Will help clear up thread congestion when subscribing to all subnets.