Crawler based DHT client

[aschmahmann]

closes #619

Still very WIP. Should merge after the crawler gets merged.

TODO:

• Extract crawler fixes into crawler branch and merge
• Expose two implementations of the MessageSender interface that we already have Will have to wait for another PR
  • One for sending one off messages to peers
  • One that tries to reuse streams and keep them alive
• Add support for options in constructor
• Add docs + comments - We have some, but we're going to have to come back for more here when we are able to do more of the cleanup work

[whyrusleeping]

What does the general extra load on the network look like for this? Crawling 'every peer' every hour seems... potentially expensive, especially as the network grows.

Additionally, theres an added resource consumption to consider here, we should quantify that.

And finally, what specific metrics should we be gathering on the network to help us better understand these implications?

[aschmahmann]

All excellent questions.

What does the general extra load on the network look like for this?

I'm not sure in that I haven't done large scale testground tests to simulate behavior here, or thought of a good way to test this on a smaller scale.

However, at a high level resource consumption should probably look like this:

• Client resource usage
  • I mostly don't care about this at the moment, local testing shows it uses more RAM and not a ton of CPU but it's an experimental alternative client if they don't like it they can turn it off
• Server resource usage
  • Record storage (HDD, RAM, CPU)
    • I expect this number across the public IPFS network to increase pretty significantly since
      • I'm pretty sure that, due to issues with the standard DHT client and how go-ipfs uses it, only a fraction of peoples records are actually making it out to the network
      • This may encourage more use of IPNS as well as IPFS in general
    • If this number really increases drastically that's both problematic, but also really cool. A user with 1M CIDs should be storing about 1M * 20 (the bucket size) * 100 bytes (a CID is 30 bytes but let's just assume we have a bunch of overhead) = 2GB of storage across the network
      • If we assume something like 20k DHT server nodes that's 100kb per node
      • So if we assume DHT server nodes would care about a 100MB increase in data storage that would mean an extra Billion CIDs getting on the network which would be pretty cool 😎
        • If this started becoming a problem we'd have to see how many CIDs came from a few major sources and then work to prevent abuse (e.g. a max number of records per peerID/IP address)
        • IPNS records and especially public keys are bigger than CIDs
  • Connections (CPU, RAM, Sockets)
    • The standard client
      • periodically queries subsets of the network to update its routing
      • queries the network and forms new connections every time the user/application performs a query (e.g. peer address discovery, provider records/CIDs, IPNS records) and then throws the data away once we use it (e.g. querying the same key twice in a row gives you very little savings)
    • The alternative client
      • periodically queries the entire network to update its routing (it currently does this once an hour in bulk, but could probably be spread out and a number that wasn't picked out of a hat could be chosen 😅)

what specific metrics should we be gathering on the network to help us better understand these implications

The winning question! Here are some that are on my mind (but more welcome, cc @petar @barath)

• Churn rate on the network -> it can inform how frequently we need to crawl the network
• Measure the number (and type) of queries a server node (or the group of hydra nodes) receive per hour as well as how many peerIDs we learn about during that time -> we should be able to compare with the number of queries we'd have in the new system (if you exclude the final Puts)
• Measure resource usage on an individual node as it stores more CIDs -> so we can make some projections around stress per node as the network usage increases

[robertschris]

This is an interesting DHT model I haven't thought of yet.

While I don't expect this to replace the default DHT mode, it does offer a great potential for bulk providers for one reason, it gets more efficient the more throughput of query that get fed through it.

Over time as short lived and unreliable nodes get culled from the peer table only long running high quality peers should remain and failure rates should drop quite quickly (if the query rate is high enough).

Some thoughts on this...

1. You should add a Bloom Filter(or similar) filled with all failed/unreliable peers.
   Adding a BF containing all the multiadresses that you have failed to connect to in the past should offer a nice performance boost as over time you will stop trying to use peers we already know are unreliable but keep getting told about.

2. Adding the ability for trusted nodes(part of a cluster?) to delegate their put requests to these bulk provider nodes should help as this DHT mode is both expensive to upkeep and gets better the more you use it, so using it like a supernode for put requests should be quite effective.

3. The warmup time is problematic and will increase with network size. Some shortcuts I can think of though...
   a. Save your peer table on shutdown and reload it on startup, unless it is too old and stale then discard.
   b. If you have trusted peers that are already running and stable you can grab their peer table.

4. The default table refresh strategy should be "trickle" based to spread out the upkeep load more evenly, also full crawls are expensive to the network as a whole and should be used sparingly. My first thought on this would be to periodically run a set of "Star" positioned requests and add(perhaps test first) all the peers we hear about while traversing these findprovs queries.

This should net you a few hundred peers throughout the DHT to add to your table each time you run it and then run this either on a fixed interval(every 30 seconds?) or dynamically whenever the node thinks it needs to refresh the table.

Additionally it may be required to once in a while(once a week/month?) to clear the unreliable peer BF and do a full crawl to both keep the false positive error rate on the BF down and to give peers another shot at proving to be reliable.

[aschmahmann]

@robertschris 👍 to basically all your suggestions.

Some thoughts on your suggestions:

1. Keeping some peer history would probably be good. e.g. peers might only be online for 16hrs a day and then off for the other 8, could be nice to track. Tracking the peers here doesn't seem the highest priority IMO, but rather it'd be great to update the table more gradually then only once per crawl
2. This is definitely a good thing to do and will make running these nodes as infrastructure really nice. However, it is either already doable or requires a protocol update depending on the circumstance.
   • Doable: PutValue can be done in a delegated way already
   • Protocol Upgrade required: Provide cannot be done in a delegated way since we don't announce "X has data with multihash Y" instead we announce "I have multihash Y". This requires Signed Provider records for the DHT #559, or more generally Remove special cases for Peer and Provider records #584.
3. I agree these seem like good ideas that should be available

• In the shorter term what I'll likely do here is bundle the existing client in with this one and use the standard client until this one is fully operational.

4. Related to 3 and 1 the key with the network crawling is not so much that the whole network is crawled but that the routing table is a complete trie instead of segmented into buckets of 20. Properly implementing accelerated lookups from the Kademlia paper XOR-trie based routing table #572 and filling the trie gradually instead of crawling the whole network at once seems reasonable. If it doesn't happen in this PR I would certainly like to see it happen in a follow up one.

I'd like to clarify that there are 3 separate optimizations here that work independently from each other (although they're complementary):

• XOR Trie based routing table (and crawling)
• Not waiting on any particular peer to respond to us (e.g. waiting for some fraction of the target peers to respond instead of the closest X peers)
• Allowing for bulk (provide) operations

Some of these could likely be added into the standard DHT client, and some things from the standard client (like the routing logic) could potentially be reused here in a more limited way to help with gradually filling the trie. Additionally, when this new client is deemed sufficiently stable I'd like to see it be useful as a DHT server as well since these nodes could accelerate lookups by less powerful peers that are using the standard client lookup logic.

I'm hoping to do a writeup this month about the work so far, time permitting 🤞. Generally I am hoping that this shows how client code can really give us a lot of progress here and to then be able to push on where the protocol needs changes and on the impacts of some of the tunable client + protocol parameters (e.g. number of peers in a response, routing table refresh periods, how many peers to wait for responses from to consider an operation successful, periodicity of repeating provides/puts, etc.)

[aschmahmann]

Note: this is a breaking change as these used to take a *IpfsDHT. We could have them take a thing that returns a host since that's all that was being used but this might be workable as well while we explore this space.

[MichaelMure]

I just wanted to say that this is a pretty cool work, and will likely be useful for Infura. 👍 👍