Refactor proposal

[jhiesey]

This is a tentative plan for a refactor followed by a new DHT protocol

[agentofuser]

Would this be a good opportunity to add support for transport privacy? ipfs/notes#37

There might be something to borrow from https://github.com/multiverse-os/oht

[jhiesey]

@agentofuser that wouldn't really fit within libp2p's idea of the DHT. I'm not familiar with oht, but it looks like a much more monolithic project unlike libp2p which is highly modular.

We have discussed that transport privacy would be nice to have to go along with the new DHT design, but it's far out of the scope of this refactor.

[anacrolix]

I don't think the components/drivers as described belong inside the DHT implementation as such. The DHT node instance should be constrained to the core DHT concepts: sending and receiving the DHT RPC messages, and managing its awareness of nodes around itself on the DHT through the k-bucket table. Where possible we should move extra concepts we wish to pile on top outside that core implementation (that's not to say outside the package, but just include awareness of it in the node).

The drivers described: pubsub, routing (what others?) should be implemented as requiring a DHT node instance to operate with, which is unaware of how it's being used.

[jhiesey]

@anacrolix I agree that it would be nice to move most of these components outside the DHT. But we still need to consider this system as a whole, including the DHT-based discovery components, when designing the DHT interface itself. So we can call this a refactor of what is currently called the DHT into a "core" DHT and a set of DHT-based discovery modules.

So it sounds like the DHT itself would contain what I called the "Network Handler" and "Outgoing Peer Routing Driver" (including the k-bucket structure), and the other components would be outside.

[gpestana]

How modular is this approach? It would be nice to allow developers to "bring their own" DHT components implementations (e.g. a routing driver that would be privacy preserving) while keeping compatibility with other libp2p peers.

[Stebalien]

I'm not really seeing anything here about protocol changes, how we're going to upgrade to multiple DHTs, how we're going to unify "provider" records and other records, etc. I'm not convinced refactoring the DHT before planning these changes is worth it as we'll just need to refactor again and again.

For example, "providers" really shouldn't be distinct from other values.