From 3d7385294ce46df6e251fb3da0d14c3464162988 Mon Sep 17 00:00:00 2001 From: vyzo Date: Tue, 29 Jun 2021 22:06:55 +0300 Subject: [PATCH] relay/: Add Circuit Relay v2 specification (#325) * Specify the Circuit Relay v2 protocol. * Move README.md specifying the Circuit Relay v1 protocol to circuit-v1.md Co-authored-by: Max Inden Co-authored-by: raulk --- relay/README.md | 297 +----------------------------- relay/circuit-v1.md | 292 ++++++++++++++++++++++++++++++ relay/circuit-v2.md | 418 +++++++++++++++++++++++++++++++++++++++++++ relay/circuit-v2.svg | 39 ++++ 4 files changed, 754 insertions(+), 292 deletions(-) create mode 100644 relay/circuit-v1.md create mode 100644 relay/circuit-v2.md create mode 100644 relay/circuit-v2.svg diff --git a/relay/README.md b/relay/README.md index cddcf4d3a..3afadbcb8 100644 --- a/relay/README.md +++ b/relay/README.md @@ -1,294 +1,7 @@ -# Circuit Relay v0.1.0 +# p2p-circuit relay -> Circuit Switching for libp2p, also known as TURN or Relay in Networking literature. +Circuit Switching for libp2p, also known as TURN or Relay in Networking literature. -| Lifecycle Stage | Maturity | Status | Latest Revision | -|-----------------|----------------|--------|-----------------| -| 3A | Recommendation | Active | r1, 2018-06-03 | - -Authors: [@daviddias] - -Interest Group: [@lgierth], [@hsanjuan], [@jamesray1], [@vyzo], [@yusefnapora] - -[@daviddias]: https://github.com/daviddias -[@lgierth]: https://github.com/lgierth -[@hsanjuan]: https://github.com/hsanjuan -[@jamesray1]: https://github.com/jamesray1 -[@vyzo]: https://github.com/vyzo -[@yusefnapora]: https://github.com/yusefnapora - -See the [lifecycle document][lifecycle-spec] for context about maturity level -and spec status. - -[lifecycle-spec]: https://github.com/libp2p/specs/blob/master/00-framework-01-spec-lifecycle.md - -## Implementations - -- [js-libp2p-circuit](https://github.com/libp2p/js-libp2p-circuit) -- [go-libp2p-circuit](https://github.com/libp2p/go-libp2p-circuit) -- [rust-libp2p-relay](https://github.com/libp2p/rust-libp2p/tree/master/protocols/relay) - -## Table of Contents - -- [Overview](#overview) -- [Dramatization](#dramatization) -- [Addressing](#addressing) -- [Wire protocol](#wire-protocol) -- [Interfaces](#interfaces) -- [Implementation Details](#implementation-details) -- [Removing existing relay protocol](#removing-existing-relay-protocol) - -## Overview - -The circuit relay is a means to establish connectivity between libp2p nodes (e.g. IPFS nodes) that wouldn't otherwise be able to establish a direct connection to each other. - -Relay is needed in situations where nodes are behind NAT, reverse proxies, firewalls and/or simply don't support the same transports (e.g. go-ipfs vs. browser-ipfs). Even though libp2p has modules for NAT traversal ([go-libp2p-nat](https://github.com/libp2p/go-libp2p-nat)), piercing through NATs isn't always an option. The circuit relay protocol exists to overcome those scenarios. - -Unlike a transparent **tunnel**, where a libp2p peer would just proxy a communication stream to a destination (the destination being unaware of the original source), a circuit relay makes the destination aware of the original source and the circuit followed to establish communication between the two. This provides the destination side with full knowledge of the circuit which, if needed, could be rebuilt in the opposite direction. - -Apart from that, this relayed connection behaves just like a regular connection would, but over an existing swarm stream with another peer (instead of e.g. TCP). A node asks a relay node to connect to another node on its behalf. The relay node short-circuits streams between the two nodes, enabling them to reach each other. - -Relayed connections are end-to-end encrypted just like regular connections. - -The circuit relay is both a tunneled transport and a mounted swarm protocol. The transport is the means of ***establishing*** and ***accepting*** connections, and the swarm protocol is the means to ***relaying*** connections. - -``` -+-----+ /ip4/.../tcp/.../ws/p2p/QmRelay +-------+ /ip4/.../tcp/.../p2p/QmTwo +-----+ -|QmOne| <------------------------------------>|QmRelay|<----------------------------------->|QmTwo| -+-----+ (/libp2p/relay/circuit multistream) +-------+ (/libp2p/relay/circuit multistream) +-----+ - ^ +-----+ ^ - | /p2p-circuit/QmTwo | | | - +-----------------------------------------+ +-----------------------------------------+ -``` - -**Notes for the reader:** - -- In this document, we use `/p2p/Qm...` multiaddrs. Libp2p previously used `/ipfs/Qm...` for multiaddrs you'll likely see uses of this notation in the wild. `/ipfs` and `/p2p` multiaddrs are equivalent but `/ipfs` is deprecated `/p2p` should be preferred. -- You may also see `/ipfs/Qm...` used for _file paths_ in IPFS. These are _not_ multiaddrs and this confusion is one of the many motivations for switching to `/p2p/Qm...` multiaddrs. - -## Dramatization - -Cast: -- QmOne, the dialing node (browser). -- QmTwo, the listening node (go-ipfs). -- QmRelay, a node which speaks the circuit relay protocol (go-ipfs or js-ipfs). - -Scene 1: -- QmOne wants to connect to QmTwo, and through peer routing has acquired a set of addresses of QmTwo. -- QmTwo doesn't support any of the transports used by QmOne. -- Awkward silence. - -Scene 2: -- All three nodes have learned to speak the `/ipfs/relay/circuit` protocol. -- QmRelay is configured to allow relaying connections between other nodes. -- QmOne is configured to use QmRelay for relaying. -- QmOne automatically added `/p2p-circuit/p2p/QmTwo` to its set of QmTwo addresses. -- QmOne tries to connect via relaying, because it shares this transport with QmTwo. -- A lively and prolonged dialogue ensues. - -## Addressing - -`/p2p-circuit` multiaddrs don't carry any meaning of their own. They need to encapsulate a `/p2p` address, or be encapsulated in a `/p2p` address, or both. - -As with all other multiaddrs, encapsulation of different protocols determines which metaphorical tubes to connect to each other. - -A `/p2p-circuit` circuit address, is formated following: - -`[]/p2p-circuit/` - -Examples: - -- `/p2p-circuit/p2p/QmVT6GYwjeeAF5TR485Yc58S3xRF5EFsZ5YAF4VcP3URHt` - Arbitrary relay node -- `/ip4/127.0.0.1/tcp/5002/p2p/QmdPU7PfRyKehdrP5A3WqmjyD6bhVpU1mLGKppa2FjGDjZ/p2p-circuit/p2p/QmVT6GYwjeeAF5TR485Yc58S3xRF5EFsZ5YAF4VcP3URHt` - Specific relay node - -This opens the room for multiple hop relay, where the second relay is encapsulated in the first relay multiaddr, such as: - -`<1st relay>/p2p-circuit/<2nd relay>/p2p-circuit/` - -A few examples: - -Using any relay available: - -- `/p2p-circuit/p2p/QmTwo` - - Dial QmTwo, through any available relay node (or find one node that can relay). - - The relay node will use peer routing to find an address for QmTwo if it doesn't have a direct connection. -- `/p2p-circuit/ip4/../tcp/../p2p/QmTwo` - - Dial QmTwo, through any available relay node, but force the relay node to use the encapsulated `/ip4` multiaddr for connecting to QmTwo. - -Specify a relay: - -- `/p2p/QmRelay/p2p-circuit/p2p/QmTwo` - - Dial QmTwo, through QmRelay. - - Use peer routing to find an address for QmRelay. - - The relay node will also use peer routing, to find an address for QmTwo. -- `/ip4/../tcp/../p2p/QmRelay/p2p-circuit/p2p/QmTwo` - - Dial QmTwo, through QmRelay. - - Includes info for connecting to QmRelay. - - The relay node will use peer routing to find an address for QmTwo. - -Double relay: - -- `/p2p-circuit/p2p/QmTwo/p2p-circuit/p2p/QmThree` - - Dial QmThree, through a relayed connection to QmTwo. - - The relay nodes will use peer routing to find an address for QmTwo and QmThree. - - We'll **not support nested relayed connections for now**, see [Future Work](#future-work) section. - -## Wire format - -We start the description of the Wire format by illustrating a possible flow scenario and then describing them in detail by phases. - -### Relay Message - -Every message in the relay protocol uses the following protobuf: - -``` -message CircuitRelay { - - enum Status { - SUCCESS = 100; - HOP_SRC_ADDR_TOO_LONG = 220; - HOP_DST_ADDR_TOO_LONG = 221; - HOP_SRC_MULTIADDR_INVALID = 250; - HOP_DST_MULTIADDR_INVALID = 251; - HOP_NO_CONN_TO_DST = 260; - HOP_CANT_DIAL_DST = 261; - HOP_CANT_OPEN_DST_STREAM = 262; - HOP_CANT_SPEAK_RELAY = 270; - HOP_CANT_RELAY_TO_SELF = 280; - HOP_BACKOFF = 290; - STOP_SRC_ADDR_TOO_LONG = 320; - STOP_DST_ADDR_TOO_LONG = 321; - STOP_SRC_MULTIADDR_INVALID = 350; - STOP_DST_MULTIADDR_INVALID = 351; - STOP_RELAY_REFUSED = 390; - MALFORMED_MESSAGE = 400; - } - - enum Type { // RPC identifier, either HOP, STOP or STATUS - HOP = 1; - STOP = 2; - STATUS = 3; - CAN_HOP = 4; // is peer a relay? - } - - message Peer { - required bytes id = 1; // peer id - repeated bytes addrs = 2; // peer's known addresses - } - - optional Type type = 1; // Type of the message - - optional Peer srcPeer = 2; // srcPeer and dstPeer are used when Type is HOP or STOP - optional Peer dstPeer = 3; - - optional Status code = 4; // Status code, used when Type is STATUS -} -``` - -### High level overview of establishing a relayed connection - -**Setup:** -- Peers involved, A, B, R -- A wants to connect to B, but needs to relay through R - -**Assumptions:** -- A has connection to R, R has connection to B - -**Events:** -- phase I: Open a request for a relayed stream (A to R). - - A dials a new stream `sAR` to R using protocol `/libp2p/circuit/relay/0.1.0`. - - A sends a CircuitRelay message with `{ type: 'HOP', srcPeer: '/p2p/QmA', dstPeer: '/p2p/QmB' }` to R through `sAR`. - - R receives stream `sAR` and reads the message from it. -- phase II: Open a stream to be relayed (R to B). - - R opens a new stream `sRB` to B using protocol `/libp2p/circuit/relay/0.1.0`. - - R sends a CircuitRelay message with `{ type: 'STOP', srcPeer: '/p2p/QmA', dstPeer: '/p2p/QmB' }` on `sRB`. - - R sends a CircuitRelay message with `{ type: 'STATUS', code: 'SUCCESS' }` on `sAR`. -- phase III: Streams are piped together, establishing a circuit - - B receives stream `sRB` and reads the message from it - - B sends a CircuitRelay message with `{ type: 'STATUS', code: 'SUCCESS' }` on `sRB`. - - B passes stream to `NewConnHandler` to be handled like any other new incoming connection. - -### Under the microscope - -- We've defined a max length for the multiaddrs of arbitrarily 1024 bytes -- Multiaddrs are transfered on its binary packed format -- Peer Ids are transfered on its non base encoded format (aka byte array containing the multihash of the Public Key). - - -### Status codes table - -This is a table of status codes and sample messages that may occur during a relay setup. Codes in the 200 range are returned by the relay node. Codes in the 300 range are returned by the destination node. - - -| Code | Message | Meaning | -| ----- |:--------------------------------------------------|:----------:| -| 100 | "success" | Relay was setup correctly | -| 220 | "src address too long" | | -| 221 | "dst address too long" | | -| 250 | "failed to parse src addr: no such protocol ipfs" | The `` multiaddr in the header was invalid | -| 251 | "failed to parse dst addr: no such protocol ipfs" | The `` multiaddr in the header was invalid | -| 260 | "passive relay has no connection to dst" | | -| 261 | "active relay couldn't dial to dst: conn refused" | relay could not form new connection to target peer | -| 262 | "couldn't' dial to dst" | relay has conn to dst, but failed to open a stream | -| 270 | "dst does not support relay" | | -| 280 | "can't relay to itself" | The relay got its own address as destination | -| 290 | "temporary backoff" | The relay wants us to backoff and try again later | -| 320 | "src address too long" | | -| 321 | "dst address too long" | | -| 350 | "failed to parse src addr" | src multiaddr in the header was invalid | -| 351 | "failed to parse dst addr" | dst multiaddr in the header was invalid | -| 390 | "connection refused by stop endpoint" | The stop endpoint couldn't accept the connection | -| 400 | "malformed message" | A malformed or too long message was received | - -## Implementation details - -### Interfaces - -> These are go-ipfs specific - -As explained above, the relay is both a transport (`tpt.Transport`) and a mounted stream protocol (`p2pnet.StreamHandler`). In addition it provides a means of specifying relay nodes to listen/dial through. - -Note: the usage of p2pnet.StreamHandler is a little bit off, but it gets the point across. - -```go -import ( - tpt "github.com/libp2p/go-libp2p-transport" - p2phost "github.com/libp2p/go-libp2p-host" - p2pnet "github.com/libp2p/go-libp2p-net" - p2proto "github.com/libp2p/go-libp2p-protocol" -) - -const ID p2proto.ID = "/libp2p/circuit/relay/0.1.0" - -type CircuitRelay interface { - tpt.Transport - p2pnet.StreamHandler - - EnableRelaying(enabled bool) -} - -fund NewCircuitRelay(h p2phost.Host) -``` - -### Removing existing relay protocol in Go - -Note that there is an existing swarm protocol colloqiually called relay. It lives in the go-libp2p package and is named `/ipfs/relay/line/0.1.0`. - -- Introduced in ipfs/go-ipfs#478 (28-Dec-2014). -- No changes except for ipfs/go-ipfs@de50b2156299829c000b8d2df493b4c46e3f24e9. - - Changed to use multistream muxer. -- Shortcomings - - No end-to-end encryption. - - No rate limiting (DoS by resource exhaustion). - - Doesn't verify src id in ReadHeader(), easy to fix. -- Capable of *accepting* connections, and *relaying* connections. -- Not capable of *connecting* via relaying. - -Since the existing protocol is incomplete, insecure, and certainly not used, we can safely remove it. - -## Future work - -We have considered more features but won't be adding them on the first iteration of Circuit Relay, the features are: - -- Multihop relay - With this specification, we are only enabling single hop relays to exist. Multihop relay will come at a later stage as Packet Switching. -- Relay discovery mechanism - At the moment we're not including a mechanism for discovering relay nodes. For the time being, they should be configured statically. +Specifications: +- [p2p-circuit v1](circuit-v1.md) +- [p2p-cricuit v2](circuit-v2.md) diff --git a/relay/circuit-v1.md b/relay/circuit-v1.md new file mode 100644 index 000000000..9d65a7a16 --- /dev/null +++ b/relay/circuit-v1.md @@ -0,0 +1,292 @@ +# Circuit Relay v0.1.0 + +| Lifecycle Stage | Maturity | Status | Latest Revision | +|-----------------|----------------|--------|-----------------| +| 3A | Recommendation | Active | r1, 2018-06-03 | + +Authors: [@daviddias] + +Interest Group: [@lgierth], [@hsanjuan], [@jamesray1], [@vyzo], [@yusefnapora] + +[@daviddias]: https://github.com/daviddias +[@lgierth]: https://github.com/lgierth +[@hsanjuan]: https://github.com/hsanjuan +[@jamesray1]: https://github.com/jamesray1 +[@vyzo]: https://github.com/vyzo +[@yusefnapora]: https://github.com/yusefnapora + +See the [lifecycle document][lifecycle-spec] for context about maturity level +and spec status. + +[lifecycle-spec]: https://github.com/libp2p/specs/blob/master/00-framework-01-spec-lifecycle.md + +## Implementations + +- [js-libp2p-circuit](https://github.com/libp2p/js-libp2p-circuit) +- [go-libp2p-circuit](https://github.com/libp2p/go-libp2p-circuit) +- [rust-libp2p-relay](https://github.com/libp2p/rust-libp2p/tree/master/protocols/relay) + +## Table of Contents + +- [Overview](#overview) +- [Dramatization](#dramatization) +- [Addressing](#addressing) +- [Wire protocol](#wire-protocol) +- [Interfaces](#interfaces) +- [Implementation Details](#implementation-details) +- [Removing existing relay protocol](#removing-existing-relay-protocol) + +## Overview + +The circuit relay is a means to establish connectivity between libp2p nodes (e.g. IPFS nodes) that wouldn't otherwise be able to establish a direct connection to each other. + +Relay is needed in situations where nodes are behind NAT, reverse proxies, firewalls and/or simply don't support the same transports (e.g. go-ipfs vs. browser-ipfs). Even though libp2p has modules for NAT port mapping ([go-libp2p-nat](https://github.com/libp2p/go-libp2p-nat)), this isn't always an option, nor does it always work (e.g. non-residential routers, hotspots, etc.). The circuit relay protocol exists to overcome those scenarios. + +Unlike a transparent **tunnel**, where a libp2p peer would just proxy a communication stream to a destination (the destination being unaware of the original source), a circuit relay makes the destination aware of the original source and the circuit followed to establish communication between the two. This provides the destination side with full knowledge of the circuit which, if needed, could be rebuilt in the opposite direction. As a word of caution, dialing a peer back on its source addr:port usually won't work. However, most libp2p implementations (e.g. go-libp2p) enable `SO_REUSEPORT` and `SO_REUSEADDR`, and use the listening address as the local address when dialing, to facilitate this connection reversability. + +Apart from that, this relayed connection behaves just like a regular connection would, but over an existing fully formed libp2p stream with another peer (instead of e.g. a raw TCP connection). Think of this as a "virtualized connection". This enables further resource efficiency and maximizes the utility of the underlying connection, as once a NAT'ted peer A has established a connection to a relay R, many peers (B1...Bn) can establish relayed connections to A over that single physical connection. The relay node acts like a circuit switcher over streams between the two nodes, enabling them to reach each other. + +Relayed connections are end-to-end encrypted just like regular connections. + +The circuit relay consists of both a (tunneled) libp2p transport and a libp2p protocol, mounted on the host. The libp2p transport is the means of ***establishing*** and ***accepting*** connections, and the libp2p protocol is the means to ***relaying*** connections. + +``` ++-----+ /ip4/.../tcp/.../ws/p2p/QmRelay +-------+ /ip4/.../tcp/.../p2p/QmTwo +-----+ +|QmOne| <------------------------------------>|QmRelay|<----------------------------------->|QmTwo| ++-----+ (/libp2p/relay/circuit multistream) +-------+ (/libp2p/relay/circuit multistream) +-----+ + ^ +-----+ ^ + | /p2p-circuit/QmTwo | | | + +-----------------------------------------+ +-----------------------------------------+ +``` + +**Notes for the reader:** + +- In this document, we use `/p2p/Qm...` multiaddrs. libp2p previously used `/ipfs/Qm...` for multiaddrs, and you'll likely see uses of this notation in the wild. `/ipfs` and `/p2p` multiaddrs are equivalent but `/ipfs` is deprecated `/p2p` should be preferred. +- You may also see `/ipfs/Qm...` used for _content-addressed pathing_ in IPFS. These are _not_ multiaddrs and this confusion is one of the many motivations for switching to `/p2p/Qm...` multiaddrs. + +## Dramatization + +Cast: +- QmOne, the dialing node (browser). +- QmTwo, the listening node (go-ipfs). +- QmRelay, a node which speaks the circuit relay protocol (go-ipfs or js-ipfs). + +Scene 1: +- QmOne wants to connect to QmTwo, and through peer routing has acquired a set of addresses of QmTwo. +- QmTwo doesn't support any of the transports used by QmOne. +- Awkward silence. + +Scene 2: +- All three nodes have learned to speak the `/ipfs/relay/circuit` protocol. +- QmRelay is configured to allow relaying connections between other nodes. +- QmOne is configured to use QmRelay for relaying. +- QmOne automatically added `/p2p-circuit/p2p/QmTwo` to its set of QmTwo addresses. +- QmOne tries to connect via relaying, because it shares this transport with QmTwo. +- A lively and prolonged dialogue ensues. + +## Addressing + +`/p2p-circuit` multiaddrs don't carry any meaning of their own. They need to encapsulate a `/p2p` address, or be encapsulated in a `/p2p` address, or both. + +As with all other multiaddrs, encapsulation of different protocols determines which metaphorical tubes to connect to each other. + +A `/p2p-circuit` circuit address, is formatted as following: + +`[]/p2p-circuit/` + +Examples: + +- `/p2p-circuit/p2p/QmVT6GYwjeeAF5TR485Yc58S3xRF5EFsZ5YAF4VcP3URHt` - Arbitrary relay node that can relay to `QmVT6GYwjeeAF5TR485Yc58S3xRF5EFsZ5YAF4VcP3URHt` (target) +- `/ip4/127.0.0.1/tcp/5002/p2p/QmdPU7PfRyKehdrP5A3WqmjyD6bhVpU1mLGKppa2FjGDjZ/p2p-circuit/p2p/QmVT6GYwjeeAF5TR485Yc58S3xRF5EFsZ5YAF4VcP3URHt` - Specific relay node to relay to `QmVT6GYwjeeAF5TR485Yc58S3xRF5EFsZ5YAF4VcP3URHt` (target) + +This opens the room for multiple hop relay, where the second relay is encapsulated in the first relay multiaddr, such that one relay relays to the next relay, in a daisy-chain fashion. Example: + +`<1st relay>/p2p-circuit/<2nd relay>/p2p-circuit/` + +A few examples: + +Using any relay available: + +- `/p2p-circuit/p2p/QmTwo` + - Dial QmTwo, through any available relay node (or find one node that can relay). + - The relay node will use peer routing to find an address for QmTwo if it doesn't have a direct connection. +- `/p2p-circuit/ip4/../tcp/../p2p/QmTwo` + - Dial QmTwo, through any available relay node, but force the relay node to use the encapsulated `/ip4` multiaddr for connecting to QmTwo. + +Specify a relay: + +- `/p2p/QmRelay/p2p-circuit/p2p/QmTwo` + - Dial QmTwo, through QmRelay. + - Use peer routing to find an address for QmRelay. + - The relay node will also use peer routing, to find an address for QmTwo. +- `/ip4/../tcp/../p2p/QmRelay/p2p-circuit/p2p/QmTwo` + - Dial QmTwo, through QmRelay. + - Includes info for connecting to QmRelay. + - The relay node will use peer routing to find an address for QmTwo, if not already connected. + +Double relay: + +- `/p2p-circuit/p2p/QmTwo/p2p-circuit/p2p/QmThree` + - Dial QmThree, through a relayed connection to QmTwo. + - The relay nodes will use peer routing to find an address for QmTwo and QmThree. + - go-libp2p (reference implementation) **does not support nested relayed connections for now**, see [Future Work](#future-work) section. + +## Wire format + +We start the description of the Wire format by illustrating a possible flow scenario and then describing them in detail by phases. + +### Relay Message + +Every message in the relay protocol uses the following protobuf: + +``` +message CircuitRelay { + + enum Status { + SUCCESS = 100; + HOP_SRC_ADDR_TOO_LONG = 220; + HOP_DST_ADDR_TOO_LONG = 221; + HOP_SRC_MULTIADDR_INVALID = 250; + HOP_DST_MULTIADDR_INVALID = 251; + HOP_NO_CONN_TO_DST = 260; + HOP_CANT_DIAL_DST = 261; + HOP_CANT_OPEN_DST_STREAM = 262; + HOP_CANT_SPEAK_RELAY = 270; + HOP_CANT_RELAY_TO_SELF = 280; + HOP_BACKOFF = 290; + STOP_SRC_ADDR_TOO_LONG = 320; + STOP_DST_ADDR_TOO_LONG = 321; + STOP_SRC_MULTIADDR_INVALID = 350; + STOP_DST_MULTIADDR_INVALID = 351; + STOP_RELAY_REFUSED = 390; + MALFORMED_MESSAGE = 400; + } + + enum Type { // RPC identifier, either HOP, STOP or STATUS + HOP = 1; + STOP = 2; + STATUS = 3; + CAN_HOP = 4; // is peer a relay? + } + + message Peer { + required bytes id = 1; // peer id + repeated bytes addrs = 2; // peer's known addresses + } + + optional Type type = 1; // Type of the message + + optional Peer srcPeer = 2; // srcPeer and dstPeer are used when Type is HOP or STOP + optional Peer dstPeer = 3; + + optional Status code = 4; // Status code, used when Type is STATUS +} +``` + +### High level overview of establishing a relayed connection + +**Setup:** +- Peers involved, A, B, R +- A wants to connect to B, but needs to relay through R + +**Assumptions:** +- A has connection to R, R has connection to B + +**Events:** +- phase I: Open a request for a relayed stream (A to R). + - A dials a new stream `sAR` to R using protocol `/libp2p/circuit/relay/0.1.0`. + - A sends a CircuitRelay message with `{ type: 'HOP', srcPeer: '/p2p/QmA', dstPeer: '/p2p/QmB' }` to R through `sAR`. + - R receives stream `sAR` and reads the message from it. +- phase II: Open a stream to be relayed (R to B). + - R opens a new stream `sRB` to B using protocol `/libp2p/circuit/relay/0.1.0`. + - R sends a CircuitRelay message with `{ type: 'STOP', srcPeer: '/p2p/QmA', dstPeer: '/p2p/QmB' }` on `sRB`. + - R sends a CircuitRelay message with `{ type: 'STATUS', code: 'SUCCESS' }` on `sAR`. +- phase III: Streams are piped together, establishing a circuit + - B receives stream `sRB` and reads the message from it + - B sends a CircuitRelay message with `{ type: 'STATUS', code: 'SUCCESS' }` on `sRB`. + - B passes stream to `NewConnHandler` to be handled like any other new incoming connection. + +### Under the microscope + +- We've defined a max length for the multiaddrs of arbitrarily 1024 bytes +- Multiaddrs are transfered on its binary packed format +- Peer Ids are transfered on its non base encoded format (aka byte array containing the multihash of the Public Key). + + +### Status codes table + +This is a table of status codes and sample messages that may occur during a relay setup. Codes in the 200 range are returned by the relay node. Codes in the 300 range are returned by the destination node. + + +| Code | Message | Meaning | +| ----- |:--------------------------------------------------|:----------:| +| 100 | "success" | Relay was setup correctly | +| 220 | "src address too long" | | +| 221 | "dst address too long" | | +| 250 | "failed to parse src addr: no such protocol ipfs" | The `` multiaddr in the header was invalid | +| 251 | "failed to parse dst addr: no such protocol ipfs" | The `` multiaddr in the header was invalid | +| 260 | "passive relay has no connection to dst" | | +| 261 | "active relay couldn't dial to dst: conn refused" | relay could not form new connection to target peer | +| 262 | "couldn't' dial to dst" | relay has conn to dst, but failed to open a stream | +| 270 | "dst does not support relay" | | +| 280 | "can't relay to itself" | The relay got its own address as destination | +| 290 | "temporary backoff" | The relay wants us to backoff and try again later | +| 320 | "src address too long" | | +| 321 | "dst address too long" | | +| 350 | "failed to parse src addr" | src multiaddr in the header was invalid | +| 351 | "failed to parse dst addr" | dst multiaddr in the header was invalid | +| 390 | "connection refused by stop endpoint" | The stop endpoint couldn't accept the connection | +| 400 | "malformed message" | A malformed or too long message was received | + +## Implementation details + +### Interfaces + +> These are go-ipfs specific + +As explained above, the relay is both a transport (`tpt.Transport`) and a mounted stream protocol (`p2pnet.StreamHandler`). In addition it provides a means of specifying relay nodes to listen/dial through. + +Note: the usage of `p2pnet.StreamHandler` is a little bit off herein, but it gets the point across. + +```go +import ( + tpt "github.com/libp2p/go-libp2p-transport" + p2phost "github.com/libp2p/go-libp2p-host" + p2pnet "github.com/libp2p/go-libp2p-net" + p2proto "github.com/libp2p/go-libp2p-protocol" +) + +const ID p2proto.ID = "/libp2p/circuit/relay/0.1.0" + +type CircuitRelay interface { + tpt.Transport + p2pnet.StreamHandler + + EnableRelaying(enabled bool) +} + +fund NewCircuitRelay(h p2phost.Host) +``` + +### Removing existing relay protocol in Go + +Note that there is an existing swarm protocol colloquially called relay. It lives in the go-libp2p package and is named `/ipfs/relay/line/0.1.0`. + +- Introduced in ipfs/go-ipfs#478 (28-Dec-2014). +- No changes except for ipfs/go-ipfs@de50b2156299829c000b8d2df493b4c46e3f24e9. + - Changed to use multistream muxer. +- Shortcomings + - No end-to-end encryption. + - No rate limiting (DoS by resource exhaustion). + - Doesn't verify src id in ReadHeader(), easy to fix. +- Capable of *accepting* connections, and *relaying* connections. +- Not capable of *connecting* via relaying. + +Since the existing protocol is incomplete, insecure, and certainly not used, we can safely remove it. + +## Future work + +We have considered more features but won't be adding them on the first iteration of Circuit Relay, the features are: + +- Multihop relay - With this specification, we are only enabling single hop relays to exist. Multihop relay will come at a later stage as Packet Switching. +- Relay discovery mechanism - At the moment we're not including a mechanism for discovering relay nodes. For the time being, they should be configured statically. diff --git a/relay/circuit-v2.md b/relay/circuit-v2.md new file mode 100644 index 000000000..90c7f8510 --- /dev/null +++ b/relay/circuit-v2.md @@ -0,0 +1,418 @@ +# Circuit Relay v2 + +This is the version 2 of the libp2p Circuit Relay protocol. + +| Lifecycle Stage | Maturity | Status | Latest Revision | +|-----------------|----------------|--------|-----------------| +| 1A | DRAFT | Active | r0, 2021-05-05 | + +Authors: [@vyzo] + +Interest Group: [@mxinden], [@stebalien], [@raulk] + +[@vyzo]: https://github.com/vyzo +[@mxinden]: https://github.com/mxinden +[@stebalien]: https://github.com/stebalien +[@raulk]: https://github.com/raulk + +See the [lifecycle document][lifecycle-spec] for context about maturity level +and spec status. + +[lifecycle-spec]: https://github.com/libp2p/specs/blob/master/00-framework-01-spec-lifecycle.md + +## Table of Contents + +- [Introduction](#introduction) + - [Rationale](#rationale) +- [The Protocol](#the-protocol) + - [Interaction](#interaction) + - [Hop Protocol](#hop-protocol) + - [Stop Protocol](#stop-protocol) + - [Reservation Vouchers](#reservation-vouchers) +- [Protobuf](#protobuf) + +## Introduction + +This is the specification of v2 of the p2p-circuit relay protocol. + +Compared to the first version of the protocol, there are some significant departures: +- The protocol has been split into two subprotocols, `hop` and `stop` + - The `hop` protocol is client-initiated, and is used when clients send commands + to relays; it is used for + reserving resources in the relay and opening a switched connection + to a peer through the relay. + - The `stop` protocol governs the endpoints of circuit switched + connections. +- The concept of resource reservation has been introduced, whereby + peers wishing to use a relay explicitly reserve resources and obtain + _reservation vouchers_ which can be distributed to their peers for + routing purposes. +- The concept of limited relaying has been introduced, whereby relays + provide switched connectivity with a limited duration and data cap. + +### Rationale + +The evolution of the protocol towards v2 has been influenced by our +experience in operating open relays in the wild. The original +protocol, while very flexible, has some limitations when it comes to +the practicalities of relaying connections. + +The main problem is that v1 has no mechanism to reserve resources in the +relay, which leads to continuoues oversubscription of relays and the +necessity of (often ineffective) heuristics for balancing resources. +In practice, running a relay proved to be an expensive proposition +requiring dedicated hosts with significant hardware and bandwidth +costs. In addition, there is ongoing work in Hole Punching +coordination for direct connection upgrade through relays, which +doesn't require an unlimited relay connection. + +In order to address the situation and seamlessly support pervasive +hole punching, we have introduced limited relays and slot +reservations. This allows relays to effectively manage their +resources and provide service at a small scale, thus enabling the +deployment of an army of relays for extreme horizontal scaling without +excessive bandwidth costs and dedicated hosts. + +Furthermore, the original decision to conflate circuit initiation and +termination in the same protocol has made it very hard to provide +relay service on demand, decoupled with whether _client_ functionality +is supported by the host. + +In order to address this problem, we have split the protocol into the +`hop` and `stop` subprotocols. This allows us to always enable the +client-side functionality in a host, while providing the option to +later mount the relay service in public hosts, _after_ the +reachability of the host has been determined through [AutoNAT]. + +[AutoNAT]: https://github.com/libp2p/specs/issues/180 + +## The Protocol + +### Interaction + +The following diagram illustrates the interaction between three peers, +_A_, _B_, and _R_, in the course of establishing a relayed connection. +Peer _A_ is a private peer, which is not publicly reachable; it +utilizes the services of peer _R_ as the relay. Peer _B_ is another +peer who wishes to connect to peer _A_ through _R_. + +![Circuit v2 Protocol Interaction](circuit-v2.svg) + +
+ Instructions to reproduce diagram + +Use https://plantuml.com/ and the specification below to reproduce the diagram. + +``` +@startuml +participant A +participant R +participant B + +skinparam sequenceMessageAlign center + +== Reservation == + +A -> R: [hop] RESERVE +R -> A: [hop] STATUS:OK + +hnote over A: Reservation timeout approaching. +hnote over A: Refresh. + +A -> R: [hop] RESERVE +R -> A: [hop] STATUS:OK + +hnote over A: ... + +== Circuit Establishment == + +B -> R: [hop] CONNECT to A +R -> A: [stop] CONNECT from B +A -> R: [stop] STATUS:OK +R -> B: [hop] STATUS:OK + +B <-> A: Connection +@enduml +``` + +
+ + +The first part of the interaction is _A_'s reservation of a relay slot +in _R_. This is accomplished by opening a connection to _R_ and +sending a `RESERVE` message in the `hop` protocol; if the reservation +is successful, the relay responds with a `STATUS:OK` message and +provides _A_ with a reservation voucher. _A_ keeps the connection to +_R_ alive for the duration of the reservation, refreshing the +reservation as needed. + +The second part of the interaction is the establishment of a circuit +switch connection from _B_ to _A_ through _R_. It is assumed that _B_ +has obtained a circuit multiaddr for _A_ of the form +`/p2p/QmR/p2p-circuit/p2p/QmA` out of band using some peer discovery +service (eg. the DHT or a rendezvous point). + +In order to connect to _A_, _B_ then connects to _R_, opens a `hop` +protocol stream and sends a `CONNECT` message to the relay. The relay +verifies that it has a reservation and connection for _A_ and opens a +`stop` protocol stream to _A_, sending a `CONNECT` message. + +Peer _A_ then responds to the relay with a `STATUS:OK` message, which +responds to _B_ with a `STATUS:OK` message in the open `hop` stream +and then proceeds to bridge the two streams into a relayed connection. +The relayed connection flows in the `hop` stream between the +connection initiator and the relay and in the `stop` stream between +the relay and the connection termination point. + +### Hop Protocol + +The Hop protocol governs interaction between clients and the relay; +it uses the protocol ID `/libp2p/circuit/relay/0.2.0/hop`. + +There are two parts of the protocol: +- reservation, by peers that wish to receive relay service +- connection initiation, by peers that wish to connect to a peer through the relay. + +#### Reservation + +In order to make a reservation, a peer opens a connection to the relay and sends a `HopMessage` with +`type = RESERVE`: + +``` +HopMessage { + type = RESERVE +} +``` + +The relay responds with a `HopMessage` of `type = STATUS`, indicating whether the reservation has +been accepted. + +If the reservation is accepted, then the message has the following form: +``` +HopMessage { + type = STATUS + status = OK + reservation = Reservation {...} + limit = Limit {...} +} +``` + +If the reservation is rejected, the relay responds with a `HopMessage` of the form +``` +Reservation { + type = STATUS + status = ... +} +``` +where the `status` field has a value other than `OK`. Common rejection status codes are: +- `PERMISSION_DENIED` if the reservation is rejected because of peer filtering using ACLs. +- `RESERVATION_REFUSED` if the reservation is rejected for some other reason, e.g. because there are too + many reservations. + + +The `reservation` field provides information about the reservation itself; the struct has the following fields: +``` +Reservation { + expire = ... + addrs = [...] + voucher = ... +} +``` + +- the `expire` field contains the expiration time as a UTC UNIX time. The reservation becomes invalid after this time and it's the responsibility of the client to refresh. +- the `addrs` field contains all the public relay addrs, including the peer ID of the relay node but not the + trailing `p2p-circuit` part; the client can use this list to construct its + own `p2p-circuit` relay addrs for advertising by encapsulating + `p2p-circuit/p2p/QmPeer` where `QmPeer` is its peer ID. +- the `voucher` is the binary representation of the reservation voucher -- see [Reservation Vouchers](#reservation-vouchers) for details. + +The `limit` field, if present, provides information about the limits applied by the relay in relayed connection. When omitted, it indicates that the relay does not apply any limits. + +The struct has the following fields: +``` +Limit { + duration = ... + data = ... +``` +- the `duration` field indicates the maximum duration of a relayed connection in seconds; if 0, there is no limit applied. +- the `data` field indicates the maximum number of bytes allowed to be transmitted in each direction; if 0 there is no limit applied. + +Note that the reservation remains valid until its expiration, as long +as there is an active connection from the peer to the relay. If the +peer disconnects, the reservation is no longer valid. + +The server may drop a connection according to its connection +management policy after all reservations expired. The expectation is +that the server will make a best effort attempt to maintain the +connection for the duration of any reservations and tag it to prevent +accidental termination according to its connection management policy. +If a relay server becomes oerloaded however, it may still drop a +connection with reservations in order to maintain its resource quotas. + +***Note: implementations _should not_ accept reservations over already relayed connections*** + +#### Connection Initiation + +In order to initiate a connection to a peer through a relay, the initiator opens a connection and sends a `HopMessage` of `type = CONNECT`: +``` +HopMessage { + type = CONNECT + peer = Peer {...} +} +``` + +The `peer` field contains the peer `ID` of the target peer and optionally the address of that peer for the case of active relay: +``` +Peer { + id = ... + addrs = [...] +``` + +***Note that active relay functionality is considered deprecated for security reasons, at least in public relays.*** +However, the protocol reserves the field to support the functionality for the rare cases where it is actually desirable to use active relay functionality in a controlled environment. + +If the relay has a reservation (and thus an active connection) from the peer, then it opens the second hop of the connection using the `stop` protocol; the details are not relevant for the `hop` protocol and the only thing that matters is whether it succeeds in opening the relay connection or not. +If the relayed connection is successfully established, then the relay responds with `HopMessage` with `type = STATUS` and `status = OK`: +``` +HopMessage { + type = STATUS + sgtatus = OK + limit = Limit {...} +} +``` +At this point the original `hop` stream becomes the relayed connection. +The `limit` field, if present, communicates to the initiator the +limits applied to the relayed connection with the semantics described +[above](#reservation). + +If the relayed connection cannot be established, then the relay responds with a `HopMessage` of `type = STATUS` and the `status` field having a value other than `OK`. +Common failure status codes are: +- `PERMISSION_DENIED` if the connection is rejected because of peer filtering using ACLs. +- `NO_RESERVATION` if there is no active reservation for the target peer +- `RESOURCE_LIMIT_EXCEEDED` if there are two many relayed connections from the initiator or to the target peer. +- `CONNECTION_FAILED` if the relay failed to terminate the connection to the target peer. + +***Note: implementations _should not_ accept connection initiations over already relayed connections*** + +### Stop Protocol + +The Stop protocol governs connection termination between the relay and the target peer; +it uses the protocol ID `/libp2p/circuit/relay/0.2.0/stop`. + +In order to terminate a relayed connection, the relay opens a stream +using an existing connection to the target peer. If there is no +existing connection, an active relay may attempt to open one using the +initiator supplied address, but as discussed in the previous section +this functionality is generally deprecated. + +The relay sends a `StopMessage` with `type = CONNECT` and the following form: +``` +StopMessage { + type = CONNECT + peer = Peer { ID = ...} + limit = Limit { ...} +} +``` +- the `peer` field contains a `Peer` struct with the peer `ID` of the connection initiator. +- the `limit` field, if present, conveys the limits applied to the relayed connection with the semantics described [above](#reservation). + +If the target peer accepts the connection it responds to the relay with a `StopMessage` of `type = STATUS` and `status = OK`: +``` +StopMessage { + type = STATUS + status = OK +} +``` + +At this point the original `stop` stream becomes the relayed connection. + +If the target fails to terminate the connection for some reason, then it responds to the relay with a `StopMessage` of `type = STATUS` and the `status` code set to something other than `OK`. +Common failure status codes are: +- `CONNECTION_FAILED` if the target internally failed to create the relayed connection for some reason. + +### Reservation Vouchers + +Successful relay slot reservations should come with _Reservation Vouchers_. +These are cryptographic certificates signed by the relay that testify that it is willing to provide +service to the reserving peer. +The intention is to eventually require the use of reservation vouchers for dialing relay addresses, +but this is not currently enforced so the vouchers are only advisory. + +The voucher itself is a [Signed Envelope](../RFC/0002-signed-envelopes.md). +The envelope domain is `libp2p-relay-rsvp` and uses the multicodec code `0x0302`. + +The payload of the envelope has the following form, in canonicalized protobuf format: +``` +message Voucher { + required bytes relay = 1; + required bytes peer = 2; + required uint64 expiration = 3; +} +``` +- the `relay` field is the peer ID of the relay. +- the `peer` field is the peer ID of the reserving peer. +- the `expiration` field is the UNIX UTC expiration time for the reservation. + +The wire representation is canonicalized, where elements of the message are written in field id order, with no unknown fields. + + +## Protobuf + +``` +message HopMessage { + enum Type { + RESERVE = 0; + CONNECT = 1; + STATUS = 2; + } + + required Type type = 1; + + optional Peer peer = 2; + optional Reservation reservation = 3; + optional Limit limit = 4; + + optional Status status = 5; +} + +message StopMessage { + enum Type { + CONNECT = 0; + STATUS = 1; + } + + required Type type = 1; + + optional Peer peer = 2; + optional Limit limit = 3; + + optional Status status = 4; +} + +message Peer { + required bytes id = 1; + repeated bytes addrs = 2; +} + +message Reservation { + optional uint64 expire = 1; // Unix expiration time (UTC) + repeated bytes addrs = 2; // relay addrs for reserving peer + optional bytes voucher = 3; // reservation voucher +} + +message Limit { + optional uint32 duration = 1; // seconds + optional uint64 data = 2; // bytes +} + +enum Status { + OK = 100; + RESERVATION_REFUSED = 200; + RESOURCE_LIMIT_EXCEEDED = 201; + PERMISSION_DENIED = 202; + CONNECTION_FAILED = 203; + NO_RESERVATION = 204; + MALFORMED_MESSAGE = 400; + UNEXPECTED_MESSAGE = 401; +} +``` diff --git a/relay/circuit-v2.svg b/relay/circuit-v2.svg new file mode 100644 index 000000000..476539781 --- /dev/null +++ b/relay/circuit-v2.svg @@ -0,0 +1,39 @@ +AARRBBReservation[hop] RESERVE[hop] STATUS:OKReservation timeout approaching.Refresh.[hop] RESERVE[hop] STATUS:OK...Circuit Establishment[hop] CONNECT to A[stop] CONNECT from B[stop] STATUS:OK[hop] STATUS:OKConnection