Embed the topology in the NetworkBehaviour #889

tomaka · 2019-01-25T10:46:06Z

Close #854
See the issue for description.
Also cc'ing #843

Changes:

Entirely removes the topology module and its content. You now have to pass a PeerId (id of the local node) instead of a Topology when creating the Swarm.
The user now has to manually pass to Floodsub the list of peer IDs that should be considered for gossiping. See the changes in chat.rs. While this seems annoying, we should probably add network behaviours that automatically do that in the future.
Kademlia and mDNS are now purely discovery protocols and now report the list of nodes that they discover.
The implementation of k-buckets has changed. Normally in the Kademlia paper you're supposed to ping the oldest node when you discard it. In libp2p we need to build a connection in order to ping a node, therefore instead of "pinging the node" we just try to connect to it instead. If we are already connected to this node, we assume that it is responsive. Consequently, if a bucket is full of nodes we are already connected to, we refuse any new insertion.

twittner · 2019-01-25T15:29:33Z

core/src/swarm.rs

-            supported_protocols,
-            listened_addrs: SmallVec::new(),
-        }
+    pub fn new(transport: TTransport, behaviour: TBehaviour, local_peer_id: PeerId) -> Self {


With #881 merged we should make Swarm generic over PeerId as well here.

twittner · 2019-01-25T15:30:38Z

core/src/swarm.rs

@@ -355,6 +317,10 @@ pub trait NetworkBehaviour<TTopology> {
    /// Builds a new `ProtocolsHandler`.
    fn new_handler(&mut self) -> Self::ProtocolsHandler;

+    /// Addresses that this behaviour is aware of for this specific peer, and that may allow
+    /// reaching the peer.
+    fn addresses_of_peer(&self, peer_id: &PeerId) -> Vec<Multiaddr>;


This should likewise be generic over peer IDs.

It's not as easy, because then we have to make the NetworkBehaviour trait generic over peer IDs as well.

Yes, I know.

@twittner

* Create gossipsub crate - Basic template, borrowed from floodsub * Add a GossipsubConfig struct and set up basic structures in the Gossipsub struct * Begin implementation of join. Adds get_random_peers helper function and adds tests * Implements gossipsub leave() * Update publishMany to incorporate gossipsub mesh and fanout logic * Use the gossipsub mesh for determining peer subscription * Remove subscribed_topics field from the Gossipsub struct * Rename gossipsubconfig to ProtocolConfig * Implement the gossipsub control messages into the Codec's Encode/Decode and modifies GossipsubRpc * Modify GossipsubActions to enums for succinctness. * Modify the memcache to store Gossipsub messages * Implement control message handling. * Update control message handling to handle multiple messages. * Handle received gossipsub messages using pre-built handlers. * Remove excess connected peer hashmap * Add extra peer mapping and consistent topic naming. * Implement heartbeat, emit_gossip and send_graft_prune. * Group logic in forwarding messages. Add messages to memcache. * Add heartbeat timer and move location of helper function. * Add gossipsub the libp2p workspace, makes layer structs public * Add logging to gossipsub - Adds the log crate and implements logging macros - Specifies versions for external crates * Add example chat for debugging purposes * Implement #868 for gossipsub. * Add rust documentation to gossipsub crate. - Adds basic documentation, overview and examples to the gossipsub crate. * Re-introduce the initial heartbeat time config. This commit also adds the inject_connected test. * Add subscribe tests. - Modifies `handle_received_subscriptions` to take a reference of subscriptions - Adds `test_subscribe` - Adds `test_handle_received_subscriptions` - Adds tests for the filter in `get_random_peers` * Add Bug fixes and further testing for gossipsub. - Corrects the tuple use of topic_hashes - Corrects JOIN logic around fanout and adding peers to the mesh - Adds test_unsubscribe - Adds test_join * Rename GossipsubMessage::msg_id -> id * Add bug fix for handling disconnected peers. * Implements (partially) #889 for Gossipsub. * handle_iwant event count tests * handle_ihave event count tests * Move layer.rs tests into separate file. * Implement clippy suggestions for gossipsub. * Modify control message tests for specific types. * Implement builder pattern for GossipsubConfig. As suggested by @twittner - The builder pattern for building GossipsubConfig struct is implemented. * Package version updates as suggested by @twittner. * Correct line lengths in gossipsub. * Correct braces in found by @twittner. * Implement @twittner's suggestions. - Uses `HashSet` where applicable - Update `FnvHashMap` to standard `HashMap` - Uses `min` function in code simplification. * Add NodeList struct to clarify topic_peers. * Cleaner handling of messagelist Co-Authored-By: AgeManning <Age@AgeManning.com> * Cleaner handling of added peers. Co-Authored-By: AgeManning <Age@AgeManning.com> * handle_prune peer removed test * basic grafting tests * multiple topic grafting test * Convert &vec to slice. Co-Authored-By: AgeManning <Age@AgeManning.com> * Convert to lazy insert. Co-Authored-By: AgeManning <Age@AgeManning.com> * Cleaner topic handling. Co-Authored-By: AgeManning <Age@AgeManning.com> * control pool piggybacking using HashMap.drain() in control_pool_flush going to squash this * Add Debug derives to gossipsub and correct tests. * changes from PR squash this all tests passing, but still some that need to be reconsidered test reform * Implements Arc for GossipsubRpc events * Remove support for floodsub nodes * Reconnected to disconnected peers, to mitigate timeout * Use ReadOne WriteOne with configurable max gossip sizes * Remove length delimination from RPC encoding * Prevent peer duplication in mesh * Allow oneshot handler's inactivity_timeout to be configurable * Correct peer duplication in mesh bug * Remove auto-reconnect to allow for user-level disconnects * Single long-lived inbound/outbound streams to match go implementation * Allow gossipsub topics to be optionally hashable * Improves gossipsub stream handling - Corrects the handler's keep alive. - Correct the chat example. - Instantly add peers to the mesh on subscription if the mesh is low. * Allows message validation in gossipsub * Replaces Cuckoofilter with LRUCache The false positive rate was unacceptable for rejecting messages. * Renames configuration parameter and corrects logic * Removes peer from fanout on disconnection * Add publish and fanout tests * Apply @mxinden suggestions * Resend message if outbound stream negotiated - Downgrades log warnings * Implement further reviewer suggestions - Created associated functions to avoid unnecessary cloning - Messages are rejected if their sequence numbers are not u64 - `GossipsbuConfigBuilder` has the same defaults as `GossipsubConfig` - Miscellaneous typos * Add MessageId type and remove unnecessary comments * Add a return value to propagate_message function * Adds user-customised gossipsub message ids * Adds the message id to GossipsubEvent * Implement Debug for GossipsubConfig * protocols/gossipsub: Add basic smoke test Implement a basic smoke test that: 1. Builds a fully connected graph of size N. 2. Subscribes each node to the same topic. 3. Publishes a single message. 4. Waits for all nodes to receive the above message. N and the structure of the graph are reproducibly randomized via Quickcheck. * Corrections pointed out by @mxinden * Add option to remove source id publishing * protocols/gossipsub/tests/smoke: Remove unused variable * Merge latest master * protocols/gossipsub: Move to stable futures * examples/gossipsub-chat.rs: Move to stable futures * protocols/gossipsub/src/behaviour/tests: Update to stable futures * protocols/gossipsub/tests: Update to stable futures * protocols/gossipsub: Log substream errors * protocols/gossipsub: Log outbound substream errors * Remove rust-fmt formatting * Shift to prost for protobuf compiling * Use wasm_timer for wasm compatibility Co-authored-by: Grant Wuerker <gwuerker@gmail.com> Co-authored-by: Toralf Wittner <tw@dtex.org> Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com> Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

This logic seems to be a leftover of libp2p#889 and unused today.

This logic seems to be a leftover of #889 and unused today.

commit 6d7c73a Merge: 2c9f0d3 25004c4 Author: Max Inden <mail@max-inden.de> Date: Mon Aug 23 15:04:51 2021 +0200 Merge branch 'libp2p/master' into handler commit 2c9f0d3 Author: Max Inden <mail@max-inden.de> Date: Mon Aug 23 14:53:42 2021 +0200 protocols/: Remove unused imports commit aa02e5f Author: Max Inden <mail@max-inden.de> Date: Fri Aug 20 14:57:22 2021 +0200 swarm/src/toggle: Fix TODOs commit b73139a Author: Max Inden <mail@max-inden.de> Date: Fri Aug 20 14:37:36 2021 +0200 core/src/connection: Remove outdated doc comment commit d4960b7 Author: Max Inden <mail@max-inden.de> Date: Fri Aug 20 14:30:40 2021 +0200 *: Format with rustfmt commit a78de13 Author: Max Inden <mail@max-inden.de> Date: Fri Aug 20 14:24:32 2021 +0200 core/: Fix ConectionClose and PendingAborted reporting commit 174693a Author: Max Inden <mail@max-inden.de> Date: Fri Aug 20 13:44:07 2021 +0200 swarm-derive: Adjust to changes commit 62c5e13 Author: Max Inden <mail@max-inden.de> Date: Thu Aug 19 21:46:58 2021 +0200 protocols/*: Update commit 9262c03 Merge: 682f6be 1e9fcf9 Author: Max Inden <mail@max-inden.de> Date: Thu Aug 19 20:21:35 2021 +0200 Merge branch 'libp2p/master' into handler commit 682f6be Author: Max Inden <mail@max-inden.de> Date: Thu Aug 19 19:41:10 2021 +0200 protocols/*: Update commit 7d9285f Author: Max Inden <mail@max-inden.de> Date: Thu Aug 19 19:08:13 2021 +0200 core/CHANGELOG: Add entry for ConnectionLimit change commit 5ff6397 Author: Max Inden <mail@max-inden.de> Date: Thu Aug 19 19:01:56 2021 +0200 core/tests: Adjust to type changes commit 425e777 Author: Max Inden <mail@max-inden.de> Date: Thu Aug 19 18:50:16 2021 +0200 core/: Emit Event::Failed on aborted pending connection commit ce0d278 Author: Max Inden <mail@max-inden.de> Date: Wed Aug 18 18:03:35 2021 +0200 core/src/connection: Report ConnectionLimit through task commit 5c2aef6 Author: Max Inden <mail@max-inden.de> Date: Wed Aug 18 17:31:13 2021 +0200 core/: Remove DisconnectedPeer::set_connected and Pool::add This logic seems to be a leftover of libp2p#889 and unused today. commit d744b4a Author: Max Inden <mail@max-inden.de> Date: Wed Aug 18 17:13:19 2021 +0200 core/src/connection/manager: Fully close a task on disconnect commit 705842f Author: Max Inden <mail@max-inden.de> Date: Wed Aug 18 16:49:26 2021 +0200 swarm/src/behaviour: Add default trait para on NetworkBehaviourAction commit 595623f Merge: d756be1 f2905c0 Author: Max Inden <mail@max-inden.de> Date: Wed Aug 18 15:09:08 2021 +0200 Merge branch 'libp2p/master' into handler commit d756be1 Author: Max Inden <mail@max-inden.de> Date: Thu Aug 12 17:16:20 2021 +0200 swarm/src/behaviour: Provide handler with Dial and DialAddr commit 46904a6 Author: Max Inden <mail@max-inden.de> Date: Thu Aug 12 16:51:13 2021 +0200 swarm/: Inject handler on connection error and closed commit a332591 Author: Max Inden <mail@max-inden.de> Date: Thu Aug 12 16:17:37 2021 +0200 core/: Return handler on connection error and closed

* protocols/gossipsub: Fix inconsistency in mesh peer tracking (libp2p#2189) Co-authored-by: Age Manning <Age@AgeManning.com> * misc/metrics: Add auxiliary crate to record events as OpenMetrics (libp2p#2063) This commit adds an auxiliary crate recording protocol and Swarm events and exposing them as metrics in the OpenMetrics format. * README: Mention security@ipfs.io * examples/: Add file sharing example (libp2p#2186) Basic file sharing application with peers either providing or locating and getting files by name. While obviously showcasing how to build a basic file sharing application, the actual goal of this example is **to show how to integrate rust-libp2p into a larger application**. Architectural properties - Clean clonable async/await interface ([`Client`]) to interact with the network layer. - Single task driving the network layer, no locks required. * examples/README: Give an overview over the many examples (libp2p#2194) * protocols/kad: Enable filtering of (provider) records (libp2p#2163) Introduce `KademliaStoreInserts` option, which allows to filter records. Co-authored-by: Max Inden <mail@max-inden.de> * swarm/src/protocols_handler: Impl ProtocolsHandler on either::Either (libp2p#2192) Implement ProtocolsHandler on either::Either representing either of two ProtocolsHandler implementations. Co-authored-by: Thomas Eizinger <thomas@eizinger.io> * *: Make libp2p-core default features optional (libp2p#2181) Co-authored-by: Max Inden <mail@max-inden.de> * core/: Remove DisconnectedPeer::set_connected and Pool::add (libp2p#2195) This logic seems to be a leftover of libp2p#889 and unused today. * protocols/gossipsub: Use ed25519 in tests (libp2p#2197) With f2905c0 the secp256k1 feature is disabled by default. Instead of enabling it in the dev-dependency, simply use ed25519. * build(deps): Update minicbor requirement from 0.10 to 0.11 (libp2p#2200) Updates the requirements on [minicbor](https://gitlab.com/twittner/minicbor) to permit the latest version. - [Release notes](https://gitlab.com/twittner/minicbor/tags) - [Changelog](https://gitlab.com/twittner/minicbor/blob/master/CHANGELOG.md) - [Commits](https://gitlab.com/twittner/minicbor/compare/minicbor-v0.10.0...minicbor-v0.11.0) --- updated-dependencies: - dependency-name: minicbor dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): Update salsa20 requirement from 0.8 to 0.9 (libp2p#2206) * build(deps): Update salsa20 requirement from 0.8 to 0.9 Updates the requirements on [salsa20](https://github.com/RustCrypto/stream-ciphers) to permit the latest version. - [Release notes](https://github.com/RustCrypto/stream-ciphers/releases) - [Commits](RustCrypto/stream-ciphers@ctr-v0.8.0...salsa20-v0.9.0) --- updated-dependencies: - dependency-name: salsa20 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * *: Bump pnet to v0.22 Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Max Inden <mail@max-inden.de> * *: Dial with handler and return handler on error and closed (libp2p#2191) Require `NetworkBehaviourAction::{DialPeer,DialAddress}` to contain a `ProtocolsHandler`. This allows a behaviour to attach custom state to its handler. The behaviour would no longer need to track this state separately during connection establishment, thus reducing state required in a behaviour. E.g. in the case of `libp2p-kad` the behaviour can include a `GetRecord` request in its handler, or e.g. in the case of `libp2p-request-response` the behaviour can include the first request in the handler. Return `ProtocolsHandler` on connection error and close. This allows a behaviour to extract its custom state previously included in the handler on connection failure and connection closing. E.g. in the case of `libp2p-kad` the behaviour could extract the attached `GetRecord` from the handler of the failed connection and then start another connection attempt with a new handler with the same `GetRecord` or bubble up an error to the user. Co-authored-by: Thomas Eizinger <thomas@eizinger.io> * core/: Remove deprecated read/write functions (libp2p#2213) Co-authored-by: Max Inden <mail@max-inden.de> * protocols/ping: Revise naming of symbols (libp2p#2215) Co-authored-by: Max Inden <mail@max-inden.de> * protocols/rendezvous: Implement protocol (libp2p#2107) Implement the libp2p rendezvous protocol. > A lightweight mechanism for generalized peer discovery. It can be used for bootstrap purposes, real time peer discovery, application specific routing, and so on. Co-authored-by: rishflab <rishflab@hotmail.com> Co-authored-by: Daniel Karzel <daniel@comit.network> * core/src/network/event.rs: Fix typo (libp2p#2218) * protocols/mdns: Do not fire all timers at the same time. (libp2p#2212) Co-authored-by: Max Inden <mail@max-inden.de> * misc/metrics/src/kad: Set query_duration lowest bucket to 0.1 sec (libp2p#2219) Probability for a Kademlia query to return in less than 100 milliseconds is low, thus increasing the lower bucket to improve accuracy within the higher ranges. * misc/metrics/src/swarm: Expose role on connections_closed (libp2p#2220) Expose whether closed connection was a Dialer or Listener. * .github/workflows/ci.yml: Use clang 11 (libp2p#2233) * protocols/rendezvous: Update prost (libp2p#2226) Co-authored-by: Max Inden <mail@max-inden.de> * *: Fix clippy warnings (libp2p#2227) * swarm-derive/: Make event_process = false the default (libp2p#2214) Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Age Manning <Age@AgeManning.com> Co-authored-by: Ruben De Smet <ruben.de.smet@rubdos.be> Co-authored-by: Thomas Eizinger <thomas@eizinger.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: rishflab <rishflab@hotmail.com> Co-authored-by: Daniel Karzel <daniel@comit.network> Co-authored-by: David Craven <david@craven.ch>

This logic seems to be a leftover of libp2p/rust-libp2p#889 and unused today.

tomaka added 8 commits January 22, 2019 16:16

Embed the topology in the NetworkBehaviour

006a63c

Put topologies inside of Floodsub and Kad

f12e574

Fix core tests

94e2086

Fix chat example

6372fb5

More work

5272d21

Merge remote-tracking branch 'upstream/master' into embed-topology

44716bb

Some cleanup

403abb5

Restore external addresses system

c32649c

tomaka added the changelog label Jan 25, 2019

ghost assigned tomaka Jan 25, 2019

ghost added the in progress label Jan 25, 2019

twittner reviewed Jan 25, 2019

View reviewed changes

twittner approved these changes Jan 25, 2019

View reviewed changes

Merge branch 'master' into embed-topology

bd81825

tomaka merged commit df92352 into libp2p:master Jan 26, 2019

tomaka deleted the embed-topology branch January 26, 2019 22:57

ghost removed the in progress label Jan 26, 2019

AgeManning added a commit to sigp/rust-libp2p that referenced this pull request Jan 29, 2019

Implements (partially) libp2p#889 for Gossipsub.

dea3929

tomaka mentioned this pull request Jan 31, 2019

Update libp2p to v0.3 paritytech/substrate#1634

Merged

AgeManning added a commit to sigp/rust-libp2p that referenced this pull request Feb 2, 2019

Implements (partially) libp2p#889 for Gossipsub.

abfa4da

tomaka mentioned this pull request Mar 22, 2019

Should the nodes floodsub sends messages to be better defined? #713

Closed

mxinden added a commit to mxinden/rust-libp2p that referenced this pull request Aug 18, 2021

core/: Remove DisconnectedPeer::set_connected and Pool::add

f838ad2

This logic seems to be a leftover of libp2p#889 and unused today.

mxinden mentioned this pull request Aug 18, 2021

core/: Remove DisconnectedPeer::set_connected and Pool::add #2195

Merged

mxinden added a commit to mxinden/rust-libp2p that referenced this pull request Aug 18, 2021

core/: Remove DisconnectedPeer::set_connected and Pool::add

5c2aef6

This logic seems to be a leftover of libp2p#889 and unused today.

mxinden added a commit that referenced this pull request Aug 19, 2021

core/: Remove DisconnectedPeer::set_connected and Pool::add (#2195)

1e9fcf9

This logic seems to be a leftover of #889 and unused today.

santos227 pushed a commit to santos227/rustlib that referenced this pull request Jun 20, 2022

core/: Remove DisconnectedPeer::set_connected and Pool::add (#2195)

cca1d09

This logic seems to be a leftover of libp2p/rust-libp2p#889 and unused today.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Embed the topology in the NetworkBehaviour #889

Embed the topology in the NetworkBehaviour #889

tomaka commented Jan 25, 2019 •

edited

Loading

twittner Jan 25, 2019

twittner Jan 25, 2019

tomaka Jan 25, 2019

twittner Jan 25, 2019

Embed the topology in the NetworkBehaviour #889

Embed the topology in the NetworkBehaviour #889

Conversation

tomaka commented Jan 25, 2019 • edited Loading

twittner Jan 25, 2019

Choose a reason for hiding this comment

twittner Jan 25, 2019

Choose a reason for hiding this comment

tomaka Jan 25, 2019

Choose a reason for hiding this comment

twittner Jan 25, 2019

Choose a reason for hiding this comment

tomaka commented Jan 25, 2019 •

edited

Loading