Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[part 2] grpc-based dht #51

Merged
merged 57 commits into from
Jun 28, 2020
Merged

[part 2] grpc-based dht #51

merged 57 commits into from
Jun 28, 2020

Conversation

justheuristic
Copy link
Member

@justheuristic justheuristic commented Jun 28, 2020

TL;DR we offloaded RPC calls to GRPC (cpp under the hood), this resulted in

  • (+) around 6.5x faster rpc calls in 256 node benchmark
  • (+) rpc replies are no longer throttled with more than 256 active connections
  • (+) protocol can now perform bulk store & find requests, e.g. protocol.store(peer_addr, [key1, key2, ...], ...)
  • (-) node initialization now takes 0.5-1 seconds (was: almost instant)
  • (-) Non-bulk RPCs to protocol have become more complex (i.e. you must store a list with one item)
    • however, these RPCs are not exposed to end user; in contrast, node.store / node.get and DHT.* didn't change

Next PR:

  • aiohttp logging; better yet, use logger from asyncio_handlers
  • reusing TCP connections
  • Bulk node.store / node.get
  • proactive caching

Future concerns: (add to issues)

  • on protocol shutdown: notify peers
  • update RoutingTable to avoid O(n * log(n)) complexity in local peers (should be: O(k * log n) where k = bucket size)
  • when this pr finishes, replace grpc.experimental.aio with grpc.aio
  • when grpc v1.31 releases to pypi, remove manual grpc installation from .circleci
  • add test for DHT stability by randomly killing peers
  • update benchmark_dht
  • clean up utils

…roves data availability if a lot of peers join)
* initial dht crawl is no longer blocking
…ach other "new nodes" and sent EVERYTHING to EVERYONE on each rpc call. Now DHT nodes will only request store on .store call OR when a new peer knocks on their DHT
* implement DHTProtocol via gRPC
* DHTProtocol now stores bytes only (not enforcing msgpack)
* add grpc requirements
@justheuristic justheuristic marked this pull request as draft June 28, 2020 03:04
@justheuristic justheuristic marked this pull request as ready for review June 28, 2020 13:19
# Conflicts:
#	hivemind/dht/node.py
#	hivemind/dht/protocol.py
#	tests/test_dht.py
@justheuristic justheuristic merged commit 8bded39 into master Jun 28, 2020
@justheuristic justheuristic deleted the protobuf branch June 28, 2020 14:18
This was referenced Jun 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants