Ambient peer discovery protocol

[thomaseizinger]

While doing some work on the universal-connectivity app, a discussion came up around supporting "ambient peer discovery". See libp2p/universal-connectivity#96.

I'd like to discuss the feasibility of an "ambient peer discovery" protocol.

Usecase

• Node A only has a connection to node B and would like to connect to more nodes.
• We assume kademlia is not an option because it is either not used on the network or node B is a in kademlia client-mode.
• Node B however is much better connected in the network.

Goal

Develop a protocol that allows node B to share its ambient peers with node A.

Proposal

Protocol name: /libp2p/ambient-peers
Kind: request-response

1. Node A opens a stream to node B with the above protocol identifier.
2. Node B returns a list of at most 10 peer records of nodes that it is connected to.

Interactions with other libp2p protocols

The protocol should be applicable in as many situations as possible and not unnecessarily tied to or assume the presence of other protocols like gossipsub, rendezvous, kademlia etc. We assume that simply connecting to some more nodes is good enough. The ambient-peers protocol can be repeated on peers found through it.

Kademlia

It is likely that through several hops of the ambient peer protocol, we eventually connect to a server that allows us to perform a kademlia random walk.

WebRTC

If node A is a browser node and connects to server node B via webrtc-direct, it can immediately detect not only other server nodes but also other browsers that are connected to the same server node. Those two browsers can then attempt to establish a direct connection, assuming node B also supports the relay protocol.

Security

1. Is there an amplification attack vector?

The proposed protocol allows a malicious actor to fake peer records and distribute them in a network and thus implicitly target a certain machine / IP address. However, I don't think it is an issue because:

a. The attacker cannot be sure that the other peer will actually dial any of the addresses.
b. The attacker cannot actively distribute the records, it has to wait for other nodes to request them.

Open questions

1. Should we allow node A to supply a list of transport technologies and have node B filter records based on them?
2. Should node B prefer peer records that it verified contain dial-able addresses?
3. Should we use peer records at all? They would probably be exchanged via identify. One peer record like contains more than a single address however. Without peer records, we would be able to directly share the addresses that we connected to. Additionally, it would be usable in libp2p-deployments that don't use identify or peer records¹.
4. Should we specify anything in regards to when this protocol should be used? Should it run periodically?

Prior art

1. Bitcoin has an addr message that allows peers to exchange IP addresses: https://en.bitcoin.it/wiki/Protocol_documentation#addr
2. "Peer exchange protocol" which never went anywhere: Simple, minimal peer exchange protocol #222
3. WAKU's ambient peer discovery: https://github.com/vacp2p/rfc/blob/master/content/docs/rfcs/34/README.md
4. BitTorrent's PEX: https://en.wikipedia.org/wiki/Peer_exchange
5. Local Peer Exchange Protocol notes#3
6. Peer Exchange Protocol notes#7

Footnotes

1. There is an argument that if we want to call ourselves a "modular networking stack" then we should be modular at the specification level and not have too many dependencies between them. ↩

[marten-seemann]

• Should we allow node A to supply a list of transport technologies and have node B filter records based on them?

That's pretty much a requirement for this to be useful for browsers.

A filter / preference for supported protocols would likely be helpful for bootstrapping networks as well.

[thomaseizinger]

A filter / preference for supported protocols would likely be helpful for bootstrapping networks as well.

I am not sure we should go this far. I'd expect that in a given network, a node will connect to a bootnode that only keeps connections to nodes of the same network, like IPFS or Ethereum.

It also makes this protocol more complicated to implement and run because now we have to filter through much more parameters on demand.

[aschmahmann]

This isn't really true. Sure, some protocols are uniform and happy to communicate with anyone who speaks the same protocols. For example, blockchain systems may want every peer to share the same state and therefore it doesn't really matter what nodes a given one is communicating with as long as all of the information is eventually disseminated. For other systems (e.g. most IPFS implementations, BitTorrent, p2p games, etc.) it would be quite bad if every peer had to bumble into every other peer and sync with them since they tend to operate by trying to efficiently connect the parties that WANT to be connected to each other. This is the reason that some systems use provider records in the Amino DHT as bootstrapping mechanisms, it allows them to narrow discovery by a specific topic.

The demand for this increases as libp2p-based systems increase interoperability. For example, if the same node that participates in validating some blockchain decides to serve the blocks and messages from that system over some common IPFS protocols to users wishing to examine blockchain state.

[thomaseizinger]

That makes sense but I am not sure if we can address this easily with this protocol. We assume that ambient peers are somewhat useful, especially if the peer we are asking has been provided by the user. I see it as a stepping stone that - in combination with other protocols - allows a node to somewhat autonomously find more interesting peers.

[Menduist]

FWIW, I'm still slowly pushing for rendezvous to become that protocol
The wire format enables whatever we want to do for discovery, it's just a question of implementing it in a way that is decentralized
(and adding some namespace "format")

Some consideration though

Node B returns a list of at most 10 peer records of nodes that it is connected to.

Returning a list of peers you are currently connected to is a big no-no in security-concerned networks, as it opens a list of nasty attacks. I would instead advise to return a list of peers you were connected to, which is still not ideal, but also the best we can do :)

[Menduist]

Also, would need to think a bit more about the privacy implications, but as long as you also return disconnected peers, it might be safe to also send connected peers

Though if all the disconnected peers are offline, it becomes trivial to find which are the current peers

[achingbrain]

Would it be bad to let nodes opt-in to sending only connected peers?

I'm primarily thinking of the public bootstrap nodes - they only exist to get peers connected to other peers and aren't doing any sensitive work so perhaps it would be ok in this special case?

Non-bootstrap nodes would then be able to send a mixed list of online/offline peers as suggested.

[Menduist]

As long as it's called
iKnowThatIAmLeakingMyListOfPeersAndMayGetSurprisingAttacksBecauseOfItAndImNotParticipatingInAnySensitiveProtocol: true
I don't mind, but I'd rather design a protocol which works in all cases:tm: without magic flags

[Menduist]

ie, I'd rather start without this flag, and only add it as a "last resort"

[achingbrain]

I think this is more of an implementation detail rather than something that would be part of the protocol, but we might want to discuss it in a "Security Considerations" addendum.

I like the flag name, it does exactly what it says on the tin.

[achingbrain]

Should we allow node A to supply a list of transport technologies and have node B filter records based on them?

That's pretty much a requirement for this to be useful for browsers.

Strongly agree here. If a browser node is sent 10x TCP/QUIC-only peers it kind of defeats the purpose.

Should we use peer records at all?

I would say yes. Gossipsub uses peer records to exchange peer information so there's precedent.

As an attacker you could create a bunch of peer ids and make up garbage addresses to return but having to sign the records makes it a little more expensive?

Should we specify anything in regards to when this protocol should be used?

Probably not? It sounds like it's going to be quite lightweight so it would probably end up getting run on every connection after Identify.

Returning a list of peers you are currently connected to is a big no-no in security-concerned networks

If it's a risk for a given deployment I would encourage them to opt out of this (or any tbh) protocol.

For example I would not expect any networks that have homogenous peers to have it turned on. By which I mean if you can guarantee the transports peers on the network have by spec (e.g. lodestar/ethereum) you should be able to use regular means to discover peers.

The value I see here is for nodes that cannot realistically use the DHT (e.g. browser nodes) because the majority of discovered peers do not speak compatible transports so your queries usually fail.

[thomaseizinger]

Should we allow node A to supply a list of transport technologies and have node B filter records based on them?

That's pretty much a requirement for this to be useful for browsers.

Strongly agree here. If a browser node is sent 10x TCP/QUIC-only peers it kind of defeats the purpose.

Right. Instead of arbitrary filtering, what if we say that node B SHOULD only return peer records that contain at least 1 address of the same kind as the connection that node A used the ambient peer protocol over? We know for sure that the node speaks that protocol and it reduces the surface a bit for nodes to arbitrarily probe for nodes around node B.

[thomaseizinger]

I think this could work quite well: Because we are sharing peer-records and not just addresses, chances are that if a browser connects to a bootnode via /webrtc-direct and the bootnode shares a peer record that contains a /webrtc-direct address that the same peer record will contain addresses that are interesting for node A, like /webrtc. Node A can then use that to establish a browser-to-browser WebRTC connection.

[maceip]

If, by default, all nodes run circuitRelayV2 & listen on ws/webtransport, then does this become obsolete?

[vyzo]

Sure

[maceip]

cannot "force" a given libp2p deployment to run certain protocols.

That's why I said 'by default', and not 'force'.

This proposal is aimed at providing one option to do this

Would an example / doc that sets this up with functionality already in libp2p suffice? e.g.,

services: {
      identify: identifyService(),
      pubsub: gossipsub({ emitSelf: true, allowPublishToZeroPeers: true }),
      dht: kadDHT({
        clientMode: true,
        validators: { ipns: ipnsValidator },
        selectors: { ipns: ipnsSelector }
      })

[thomaseizinger]

I don't know what emitSelf in gossipsub does but assuming the following scenario:

• node A is a browser node
• node B is a browser node
• relay node R does not speak any other protocols
• node B is connected to multiple other browers already and some servers
• user pastes a (relayed) /webrtc listening address of node B into UI of node A
• we establish a WebRTC connection

How can node A learn about addresses of other nodes connected to node B?

This is pretty much what we want to solve in universal connectivity: libp2p/universal-connectivity#95. I just added "relay node does not speak any other protocols" because if we want to rely on this to work, we can't make assumptions about which address the user pasts into the UI.

[maceip]

I am suggesting what was mentioned here:

#587 (reply in thread)

Run your own dht + pub/sub to announce peers?

[thomaseizinger]

The proposal explicitly mentions that DHT is not option because we e.g. may have a connection between two lite-clients like browsers or mobile phones.

I am not aware of a libp2p spec that uses pub/sub to discover new peers. I think it could make sense to build one but I see that as orthogonal to this spec.

[2color]

@thomaseizinger How is this proposal different to the Rendezvous Protocol?

If I understand it correctly, this ambient discovery protocol doesn't rely on a "central" Rendezvous point and instead allows any peer supporting this protocol to just share other peers they are/were connected to. Is that a good way to understand it?

[thomaseizinger]

Yes, it is just drastically simpler and not opinionated about "advertising" your services before somebody else can discover you.

[SgtPooki]

The proposed protocol allows a malicious actor to fake peer records and distribute them in a network and thus implicitly target a certain machine / IP address. However, I don't think it is an issue because:

a. The attacker cannot be sure that the other peer will actually dial any of the addresses.
b. The attacker cannot actively distribute the records, it has to wait for other nodes to request them.

I'm no security expert, but I think there is still an opportunity for users to poison the network with invalid peers something akin to https://owasp.org/www-community/attacks/Cache_Poisoning

Also, for the assumption that peers won't dial them (a), this protocol is intended to expose methods for peers to find others to dial, so it's implied that the given peers will be dialed.

For (b), I think that's an important point, but if this spec is implemented and useful, one can assume that they will eventually receive requests; especially from browser nodes who are very thirsty for dialable peers.

Some sort of reputation mechanism could be implemented as part of this ambient-peer-discovery spec so that we can ignore nodes if we discover that they're giving us fake/bad peers. Do we need this, and how would we do that?

---

Should we allow node A to supply a list of transport technologies and have node B filter records based on them?

That would be great. We were just talking about a POC for "please give me dialable peers" in our recent JS colo in NY: https://pl-strflt.notion.site/libp2p-smart-dialing-ded9379cbfb449fa928b34a7555ea7ff?pvs=4

Should node B prefer peer records that it verified contain dial-able addresses?

This seems like a Yes to me, but, to be more specific: prefer them over what exactly?

Should we use peer records at all? They would probably be exchanged via identify. One peer record like contains more than a single address however. Without peer records, we would be able to directly share the addresses that we connected to. Additionally, it would be usable in libp2p-deployments that don't use identify or peer records1.

One callout here is that another attack vector could be spamming nodes with "give me peers." If we have to do filtering/sorting on node A, in order to return only valid/transportX multiaddrs, then a swarm of nodes requesting this information could cause a brown-out on node A.

We should ensure we're limiting the work nodes we request things from have to do (aka relayV2 over relayV1).

Should we specify anything in regards to when this protocol should be used? Should it run periodically?

I think periodically running the protocol makes sense, but also whenever identify is ran. Some scenarios when I think we could run it:

1. Connect to new node, discover peers it knows
2. Have connected to a "great" node for a while, ensure we are getting updates of new nodes that node is discovering
3. have re-connected to a node we connected to over X time ago, re-ask it for peers

[thomaseizinger]

The proposed protocol allows a malicious actor to fake peer records and distribute them in a network and thus implicitly target a certain machine / IP address. However, I don't think it is an issue because:
a. The attacker cannot be sure that the other peer will actually dial any of the addresses.
b. The attacker cannot actively distribute the records, it has to wait for other nodes to request them.

I'm no security expert, but I think there is still an opportunity for users to poison the network with invalid peers something akin to owasp.org/www-community/attacks/Cache_Poisoning

Also, for the assumption that peers won't dial them (a), this protocol is intended to expose methods for peers to find others to dial, so it's implied that the given peers will be dialed.

For (b), I think that's an important point, but if this spec is implemented and useful, one can assume that they will eventually receive requests; especially from browser nodes who are very thirsty for dialable peers.

I think the main point here is that the malicious node has no control over timing at all. For example, if you wanted to perform some kind of DoS attack, you need to have a lot of dials happen at the same time. Due to the passive nature of this protocol, that risk is slightly lower I guess?

Some sort of reputation mechanism could be implemented as part of this ambient-peer-discovery spec so that we can ignore nodes if we discover that they're giving us fake/bad peers. Do we need this, and how would we do that?

I would really like to avoid going down that path because it is very hard to build something on this abstraction level that is generally useful.

Should we allow node A to supply a list of transport technologies and have node B filter records based on them?

That would be great. We were just talking about a POC for "please give me dialable peers" in our recent JS colo in NY: pl-strflt.notion.site/libp2p-smart-dialing-ded9379cbfb449fa928b34a7555ea7ff?pvs=4

See #587 (reply in thread).

Should we use peer records at all? They would probably be exchanged via identify. One peer record like contains more than a single address however. Without peer records, we would be able to directly share the addresses that we connected to. Additionally, it would be usable in libp2p-deployments that don't use identify or peer records1.

One callout here is that another attack vector could be spamming nodes with "give me peers." If we have to do filtering/sorting on node A, in order to return only valid/transportX multiaddrs, then a swarm of nodes requesting this information could cause a brown-out on node A.

We should ensure we're limiting the work nodes we request things from have to do (aka relayV2 over relayV1).

Good point. Receiving multiple requests from the same node in a short period of time doesn't make much sense. Thus I think it would be fair to say that node B MAY return an empty list at any point if it wishes to not return any peers.

Should we specify anything in regards to when this protocol should be used? Should it run periodically?

I think periodically running the protocol makes sense, but also whenever identify is ran. Some scenarios when I think we could run it:

Does it make sense? I would have thought that it is more of a one-off use or if your connections drop to less than 5 or something.

[SgtPooki]

Does it make sense? I would have thought that it is more of a one-off use or if your connections drop to less than 5 or something.

I guess we could lazily determine new peers to connect to (i.e. Only when we NEED to) but proactive awareness has its benefits (i.e. my connections dropped below threshold, connect to nodes i’ve already discovered)

If we’re talking about enabling a peers awareness of other peers in the network, doing so while we CAN makes more sense to me than when we NEED to.

Both make sense, but either option optimizes for a different risk, and has its own drawbacks. So, provide a config needed to enable user choice?

[SgtPooki]

Also, +1 to

[reputation mechanism]

I would really like to avoid going down that path because it is very hard to build something on this abstraction level that is generally useful.

[thomaseizinger]

Does it make sense? I would have thought that it is more of a one-off use or if your connections drop to less than 5 or something.

I guess we could lazily determine new peers to connect to (i.e. Only when we NEED to) but proactive awareness has its benefits (i.e. my connections dropped below threshold, connect to nodes i’ve already discovered)

If we’re talking about enabling a peers awareness of other peers in the network, doing so while we CAN makes more sense to me than when we NEED to.

Both make sense, but either option optimizes for a different risk, and has its own drawbacks. So, provide a config needed to enable user choice?

I think with the latest idea of only returning peers we were once connected to, it is fine to run this several times per connection. Looking at the identify spec, it doesn't say when or how often you should run this so I think we also don't have to here.

[vyzo]

I posted this in a thread, and reposting here at top level for visibility:

This feature should be strictly opt in.

I understand where you are coming from, but there are serious security implications from revealing your peers.

[achingbrain]

We assume kademlia is not an option because it is either not used on the network or node B is a in kademlia client-mode.

Another datapoint here - the reason Kademlia is not a realistic option for browsers right now is neatly summarised by the enormous gulf on the right hand side of this graph.

Chrome supports only Secure WebSockets, WebTransport and WebRTC - out of the 32k nodes on the IPFS DHT, only about 5.5% of them listen on those transports. For Firefox it's even worse since WebTransport is unusable until serverCertificateHashes support is implemented.

So after connecting to the PL managed bootstrappers and issuing a query to find peers that are KAD-close to you, the chances of actually being able to dial those peers is very small.

Individual DHT queries also have a high failure rate/results inconsistent with QUIC/TCP peers for the same reason.

[aschmahmann]

This isn't really related. You could make a new kademlia DHT other than Amino, even reusing the same code but with protocol ID /webtransport-or-bust/1.0.0 and require that everyone in your network supports webtransport... sure you have to redeploy code everywhere but you do for this protocol too.

[achingbrain]

You could, yes & it's part of how the universal connectivity demo worked. But for practical purposes the Amino DHT is the one in use, it's the one we measure and it's the one that has barely any support for transports that work in browsers.

[aschmahmann]

Sure, but how does this proposal resolve any of that? It's just a totally different topology, it switches from a structured to an unstructured network. Not saying having unstructured networks is bad (people do this with gossipsub all the time), just that it's unrelated to why this protocol is/is not a good idea.

[achingbrain]

It doesn't, nor do I think it's trying to. It's just another peer discovery mechanism.

[aschmahmann]

Perhaps a stupid question, but given that this is:

1. a request-response protocol
2. considered for browser usage
3. considered for bootstrapping

Wouldn't it make sense for this to be an HTTP protocol that can be served over libp2p if desired? Seems like a pretty good match if you're choosing to do something new.

[achingbrain]

Request/response protocols in general are a good fit for HTTP. Does the transport need to be codified into the protocol spec?

[thomaseizinger]

Based on how we are designing HTTP+libp2p, this can easily be served over HTTP, yes! I wouldn't put it into the initial spec though. We can always update it later by saying something like:

When accessed over HTTP, the protocol is available as GET /, i.e. directly on the path that it is mounted on using the protocol-identifier.

So if the HTTP+libp2p module mounts the /libp2p/ambient-peers protocol at /ambient-peers, then a client can fetch them using GET /ambient-peers.

Authentication would likely be optional.

[Menduist]

I would also like for this to have some form of network identifier (ie like namespaces in rendezvous)
Most protocols also have testnets, or even just multiple prod networks. And if, for any reason, you connect to a peer in another network (or a peer connected to multiple networks, which can happen for bridges, aggregators, whatever), he'll start to send you peers you don't care about, and you'll only be able to filter them after connecting to them

I think namespaces are even more useful than that, since the app can even split further than the "network level" (ie, create a namespace for nodes storing all the history of the network, vs nodes that only have the recent one), but we don't have to go that far.

Both bitcoin PX (in the addr_list.net_addr.services), bittorrent PX (they only exchange peers for a specific file) and waku PX (in the ENR) have such a field

[thomaseizinger]

Could we solve that by just recommending that implementations should allow users to change the protocol name? That is what we do for kademlia to solve the same problem.

[Menduist]

How does the "server" knows which peer is in which "network" though? Looking at supported protocols in identify?

[thomaseizinger]

If you are not in the same network, you won't be supporting the same protocol so you can't exchange peers. This assumes networks have their own bootnodes and don't reuse bootnodes across networks.

[thomaseizinger]

i.e. From the PoV of libp2p, they are entirely different protocols which happen to use the same wireformat under the hood. Just semantically incompatible.

[Menduist]

Ok, seems less flexible as this won't handle multiple "namespace" per node, but if we ever want to do so, we could do the identify trick, so why not

[thomaseizinger]

I am closing this in favor of a spec draft as a PR: #590

Thanks for all the feedback!