transports/onion: Add dial-only implementation of Transport

[umgefahren]

Description

Adding a libp2p transport built on top of the newly released Arti. Currently only dialing is supported, since Onion Services are not supported by Arti yet and it will likely take some time until it's supported, since it's not funded yet.

The implementation should be considered as a first step towards a stable Tor based transport.

Links to any relevant issues

This implementation follows the discussion in #708 and should close this issue.

Open Questions

I added libp2p-onion to the default features. However this might be a bad idea, since arti has a lot of dependencies and therefore increases the build time of libp2p greatly. (There are no default features anymore)

Since there is no documentation yet and it's very untested (because of the transports nature automated testing is hard) libp2p-onion maybe shouldn't be integrated in libp2p at all now.

Change checklist

• I have performed a self-review of my own code
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• A changelog entry has been made in the appropriate crates

[umgefahren]

Documentation is missing, but I will add that, once the code itself is approved.

[thomaseizinger]

Did an initial pass, thanks for tackling this!

[thomaseizinger]

Thanks! I've left a few more comments with a mix of nits and other feedback :)

[mxinden]

Very happy to see this happening! Thanks @umgefahren for the work and thanks @thomaseizinger for the reviews. Please ping me in case you need my input anywhere.

[mxinden]

Cross-referencing related effort on the nim-libp2p side: vacp2p/nim-libp2p#765

//CC @Menduist and @diegomrsantos

[thomaseizinger]

Thanks for the continued work here 😊

I left a few more comments. Looking forward to merging this!

[thomaseizinger]

Thanks!

Let me know what you think of the commit I pushed. A few minor doc fixes, otherwise happy to merge this :)

Would you mind doing a smoke test with the example to check that we aren't shipping something outright broken?

[thomaseizinger]

It seems we have a CI failure from cargo deny because it can't interpret some license properly :/

[mxinden]

Is it ok that the interoperability check always fails? And since the semver-checks were introduced in #2635 the CI doesn't get green anymore sadly.

Interoperability tests are back to green with libp2p/test-plans#60.

[thomaseizinger]

I will reach out to our legal team as well, just in case. I don't think we need to block on this.

No objections from our legal team.

On what in particular? Removing the licence headers in each file?

[mxinden]

I will reach out to our legal team as well, just in case. I don't think we need to block on this.

No objections from our legal team.

On what in particular? Removing the licence headers in each file?

No. Regarding the two crates with licensing concerns.

[marten-seemann]

I'd be hesitant to merge this code into the main repository, for multiple reasons:

1. False sense of security: libp2p, and applications built on top of libp2p, broadcast their IDs and multiaddresses at many different places. It's really easy to accidentally reveal your identity, probably even if you turn off the obvious candidates like Identify, AutoNAT and DCuTR. libp2p has not been designed with privacy goals as strict as Tor's in mind, and making sure that the stack has the privacy properties that people expect when they hear that "libp2p over Tor is possible" would be a massive engineering effort. There's probably a lot more ways to compromise yourself other than AutoNAT just publishing your non-Tor addresses.
   In all likelihood, adding a Tor transport would not yield the privacy properties that people are looking for. Keep in mind that Tor is used by dissidents all around the world, who are not experts in network engineering. It is our responsibility to not endanger their safety by overpromising on the privacy properties of the software we publish.
2. Usability: DHT lookups are already slow. Adding the Tor hops to the equation will make things even more painful. There might be optimizations one could achieve from a deeper integration between libp2p and Tor (after all, Tor is running a DHT as well), but that would be a larger effort and require a really in-depth understanding of both IPFS and Tor.
3. Ethical: If the libp2p Tor transport gained a lot of adoption, this might pose a problem for the Tor network. It's a fairly small network with less than 10k relays, handling important traffic of people who actually rely on the privacy properties of the network.

Merging a Tor transport into a PL-maintained libp2p implementation sends a signal that this transport is properly tested and vetted, no matter how many disclaimers we include. As a modular stack, it should be possible to have this code live outside of rust-libp2p (and outside the libp2p org). This would make it possible for people to play around and experiment with this code, without giving it the blessing it would receive from being a part of rust-libp2p.

[marten-seemann]

Adding more context here. The ask for a Tor transport is as old as IPFS / libp2p itself. See for example this discussion from 2016 (!): libp2p/go-libp2p#79

[Menduist]

I'll reply to your points from nim-libp2p point of view, since we'll soon merge our Tor transport. As always, you are free to do want you want, I'm just giving another perspective :)

1. We are going to recommend (or even force) application developers to create a second switch for the Tor transport only. Trying to isolate the Tor transport from other transports properly in a single switch is very tricky and will probably fail, since a lot of state is shared (conn manager, identity, etc)
2. The application developer will then have to balance, and think about which operations are safe to put on the regular switch, and which one are privacy risk and should go on the Tor switch (at the cost of latency & bandwidth lost etc). For instance, participating in a gossipsub mesh is probably fine on the regular switch, but publishing on it should go on the Tor switch
3. I mean, the Tor network exists to be used. Of course people can abuse it and use a lot of resources from it without actually needing to. But they don't need our help for that. For me, Tor transport is a first step in privacy protection in libp2p at the network level. If we start to see a lot of adoption, it may be the sign that we need to build onion routing (or equivalent) in libp2p networks themselves, instead of leaching another network. But today we can't know if that appetite for network level privacy exist

EDIT: and just to clarify: the goal of the Tor transport is not to say "you just enable this flag and your application becomes magically private", it's here to say "if you are building an application with privacy in mind, you can use this flag as one of the tool used to keep it secure". But it's so easy to leak informations at every layer, the whole application has to be built with privacy in mind

[diegomrsantos]

We are considering:

• Making the TorTransport private
• Creating a TorSwitch where it's not possible to add other transports and name resolving is disabled (see Tor design paper section 4.3 Opening and closing streams). It will force the use of a new dedicated switch with a new peer id.

[umgefahren]

First, I want to thank for all the thoughts and comments on this PR. I see pretty much every point made here, but I have some additional things to say, and I want to offer a way to go forward.

Regarding @marten-seemann comments:

• Of course, "libp2p over Tor" may imply more privacy than this offers. But I don’t think it’s that big of an issue. If we are honest about the caveats that lay here, we can somewhat trust developers to handle the transport with the care it requires. After all, libp2p is not faced towards direct users, so the responsibility is somewhat with the libp2p consumers and thus not directly with the users of Tor. If we include some notices in the documentation and continue our effort here, I consider it a hand able risk.
• Usability is an issue though. I can’t really argue against that. Although there might some usage profiles where the drawbacks are worth the benefits. Maybe the Tor transport is only used to retrieve data directly from a peer via Request-Response. Something like this would require the "switch" described later.
• While I understand your ethical considerations, I also agree with @Menduist. While we don’t make abuse easier, we make it a lot easier to write privacy focused software and possibly help the groups under attack we want to protect.

Regarding @Menduist:

I pretty much agree with all you said. I think we in this repository should look at methods on how we can steer behaviors towards using a given transport and also take a look at nim-libp2p here. But I think this is beyond the scope of this PR. I will open an issue about it, as soon as this is merged.

Finally, I want to leave some comments of my own here.

The reason why I wrote this transport was because I’m really interested in bringing transport level privacy to libp2p. The new iteration of the Dione project I wrote would benefit heavily from it and I could think of a lot of other applications where transport level privacy would be good. I recognize it will be quite hard to properly integrate that into libp2p, but we should do it anyway. I would also recommend looking into writing a generic (in customizable) but tested and capable onion router as libp2p behavior and transport. I would say it’s well in the scope of the project and very doable. In fact, there are Onion Routers written with libp2p (i.e. hoprnet).

Recommendations

If it’s fine by @mxinden and @thomaseizinger, I’m still in favor of merging here, although I would like to take some steps previously.

1. Include a privacy notice in the documentation.
2. Rename everything onion to Tor (we could add other onion routers later and libp2p-onion would fit better if we were ever to develop our own onion router).

We should monitor how this transport develops and is being used. Throwing it out is of course always an option. However, I would do that, once we have better alternatives, transport level privacy seems to be a concern to some people.

[marten-seemann]

The requests for Tor support in libp2p / IPFS are as old as the project itself. Over the years, the IPFS and libp2p team has repeatedly decided to not merge any Tor transport into the main project.

Where is the push to merge this into the main repository coming from? Why does this transport need the implicit blessing that comes with living in the main repo? One of the advantages of libp2p being a modular networking stack is that it is possible to include your own transports from repositories outside the main project.

[umgefahren]

You are right, of course it could live outside of rust-libp2p. If this doesn't get merged, I will extract it into it's own repository. But I don't want to be the person making that call.

[thomaseizinger]

Where is the push to merge this into the main repository coming from? Why does this transport need the implicit blessing that comes with living in the main repo? One of the advantages of libp2p being a modular networking stack is that it is possible to include your own transports from repositories outside the main project.

From a maintainer PoV, I definitely see the appeal of certain transports living outside of the monorepo. This does however require us (@libp2p/rust-libp2p-maintainers) to make a commitment to API stability and reducing the number of breaking changes each version. Otherwise, consuming those out-of-tree transports is just a pain.

What if we introduce a libp2p/rust-libp2p-community? This could be a place for community-driven transports and behaviours where people such as @umgefahren can maintain their crates without requiring our blessing.

[thomaseizinger]

This does however require us (@libp2p/rust-libp2p-maintainers) to make a commitment to API stability and reducing the number of breaking changes each version.

I opened a separate discussion for this here: #3072

[umgefahren]

What if we introduce a libp2p/rust-libp2p-community? This could be a place for community-driven transports and behaviours where people such as @umgefahren can maintain their crates without requiring our blessing.

That sounds like a great compromise to me. Should I close this PR and open a new one at the new repository?

[thomaseizinger]

What if we introduce a libp2p/rust-libp2p-community? This could be a place for community-driven transports and behaviours where people such as @umgefahren can maintain their crates without requiring our blessing.

That sounds like a great compromise to me. Should I close this PR and open a new one at the new repository?

Let's wait for what the other maintainers have to say about this :)

[thomaseizinger]

Where is the push to merge this into the main repository coming from? Why does this transport need the implicit blessing that comes with living in the main repo? One of the advantages of libp2p being a modular networking stack is that it is possible to include your own transports from repositories outside the main project.

From a maintainer PoV, I definitely see the appeal of certain transports living outside of the monorepo. This does however require us (@libp2p/rust-libp2p-maintainers) to make a commitment to API stability and reducing the number of breaking changes each version. Otherwise, consuming those out-of-tree transports is just a pain.

What if we introduce a libp2p/rust-libp2p-community? This could be a place for community-driven transports and behaviours where people such as @umgefahren can maintain their crates without requiring our blessing.

Friendly ping @mxinden @jxs @elenaf9 :)

[mergify]

This pull request has merge conflicts. Could you please resolve them @umgefahren? 🙏

[thomaseizinger]

I think for the time being, it is better to leave this out of tree. Let's discuss more over at #3072 how we can stabilize the APIs to make maintaining transports and behaviours out of tree more pleasant.

@umgefahren Feel free to tag me or any other maintainer any time you want feedback or if you are struggling with an upgrade.

In the meantime, we can extend our README with a link to your repository to give it some visibility. Also, libp2p-tor and libp2p-onion are still available on crates.io :)

[umgefahren]

Thanks, I will publish the work I've done here on my own repository.

[umgefahren]

If someone is interested: https://github.com/umgefahren/libp2p-tor