feat(p2p/discovery)!: Implement archival discovery + syncing of historic blocks and blobs
#3188
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
renaynay
force-pushed
the
shrex-getter-routing
branch
2 times, most recently
from
50dfce3 to
6e201c8
Compare
renaynay
force-pushed
the
shrex-getter-routing
branch
2 times, most recently
from
15a2d9f to
7acc848
Compare
renaynay
changed the title
archival discovery + syncing of historic blocks and blobs
renaynay
force-pushed
the
shrex-getter-routing
branch
from
ffab4a1 to
b416f83
Compare
renaynay
force-pushed
the
shrex-getter-routing
branch
from
b416f83 to
dccf56d
Compare
renaynay
added a commit
that referenced
this pull request
Apr 18, 2024
Subscription was never cancelled on Stop, found while working on swamp tests for #3188
Wondertan
pushed a commit
that referenced
this pull request
Apr 21, 2024
Subscription was never cancelled on Stop, found while working on swamp tests for #3188
renaynay
force-pushed
the
shrex-getter-routing
branch
2 times, most recently
from
3764930 to
d79a9c7
Compare
renaynay
force-pushed
the
shrex-getter-routing
branch
from
d79a9c7 to
586f97e
Compare
Wondertan
reviewed
May 7, 2024
renaynay
force-pushed
the
shrex-getter-routing
branch
2 times, most recently
from
1684a62 to
e276af8
Compare
renaynay
commented
May 8, 2024
renaynay
changed the title
archival discovery + syncing of historic blocks and blobsarchival discovery + syncing of historic blocks and blobs
walldiss
reviewed
May 16, 2024
…chival topic since default is set to true now
Wondertan
reviewed
May 17, 2024
walldiss
reviewed
May 17, 2024
cristaloleg
reviewed
May 21, 2024
chore(share/getters): remove optionality for archival peer man - require it
…d put in private opts
walldiss
reviewed
May 23, 2024
…nt of both peermans to shrex and remove from nodebuilder, require availWindow as arg for shrex ctor
Wondertan
reviewed
May 24, 2024
Wondertan
previously approved these changes
May 27, 2024
walldiss
approved these changes
May 27, 2024
There was a problem hiding this comment.
Great effort on the nodebuilder in this PR! However, it's still pretty tough to tell if everything's correct. My biggest concern is that the feature needs thorough testing to ensure all setups work as intended. Given its complexity, future changes might get tricky, so let's prioritize refactoring as soon as we have the resources.
Wondertan
approved these changes
May 28, 2024
ramin
pushed a commit
that referenced
this pull request
Jun 6, 2024
…toric blocks and blobs (#3188) This PR introduces the discovery of nodes advertising on the `archival` topic for the purpose of being able to request blobs and blocks that are older than the sampling window. In order to do so, there are now 2 instances of discovery (one for `full` nodes -- _all nodes running full block sync within the sampling window_, and one for `archival` nodes -- _nodes that sync, store and serve all historic data_) and 2 instances of peer managers (one for `full` and one for `archival`). **This PR is _breaking_ due to changes to metrics names.** Please also note that this PR introduces the ability to perform archival sync of blobs + blocks for **LIGHT AND FULL** nodes only regardless of whether the full nodes have pruning enabled or not. **Bridge nodes that have previously had pruning enabled will NOT be able to request historic data from the consensus core node.** The bridge node must run in archival mode (without `--experimental-pruning` enabled) in order to access historic data. **Bridge and Full nodes in archival mode (without `--experimental-pruning`)** ``` 2024-05-10T15:21:44.490+0200 INFO share/discovery discovery/manager.go:30 starting discovery {"topic": "full"} 2024-05-10T15:21:44.491+0200 INFO share/discovery discovery/manager.go:33 advertising to topic {"topic": "full"} 2024-05-10T15:21:44.491+0200 INFO share/discovery discovery/manager.go:30 starting discovery {"topic": "archival"} 2024-05-10T15:21:44.491+0200 INFO share/discovery discovery/manager.go:33 advertising to topic {"topic": "archival"} ``` **Bridge and Full nodes running `--experimental-pruning`** ``` 2024-05-10T15:19:20.698+0200 INFO share/discovery discovery/manager.go:30 starting discovery {"topic": "full"} 2024-05-10T15:19:20.698+0200 INFO share/discovery discovery/manager.go:33 advertising to topic {"topic": "full"} 2024-05-10T15:19:20.698+0200 INFO share/discovery discovery/manager.go:30 starting discovery {"topic": "archival"} ``` **Light nodes** ``` 2024-05-10T15:20:42.440+0200 INFO share/discovery discovery/manager.go:30 starting discovery {"topic": "full"} 2024-05-10T15:20:42.440+0200 INFO share/discovery discovery/manager.go:30 starting discovery {"topic": "archival"} ``` ----------------------------------------------------------- **TODO** - [x] test archival request routing per node type - [x] renaming some things inside discovery away from being `full`-node-centric - [x] rebase - [x] swamp test For follow-up PR: - [x] clarify / make disc metrics generic as there will now be 2 instances of discovery happening Nice to have; - [ ] benchmarks for LN
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces the discovery of nodes advertising on the
archivaltopic for the purpose of being able to request blobs and blocks that are older than the sampling window.In order to do so, there are now 2 instances of discovery (one for
fullnodes -- all nodes running full block sync within the sampling window, and one forarchivalnodes -- nodes that sync, store and serve all historic data) and 2 instances of peer managers (one forfulland one forarchival).This PR is breaking due to changes to metrics names.
Please also note that this PR introduces the ability to perform archival sync of blobs + blocks for LIGHT AND FULL nodes only regardless of whether the full nodes have pruning enabled or not. Bridge nodes that have previously had pruning enabled will NOT be able to request historic data from the consensus core node. The bridge node must run in archival mode (without
--experimental-pruningenabled) in order to access historic data.Bridge and Full nodes in archival mode (without
--experimental-pruning)Bridge and Full nodes running
--experimental-pruningLight nodes
TODO
full-node-centricFor follow-up PR:
Nice to have;