Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/dht persist seed #382

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
60a62b9
initial sketch of persisting/seeding functionality.
raulk Apr 8, 2019
4bb1ee9
wip.
raulk May 23, 2019
50add9d
gx publish 4.4.34
Stebalien Mar 20, 2019
e76bd42
GoModules: Checksum mismatch:
im-kulikov Mar 21, 2019
a897931
Add go1.12 to TravisCI
im-kulikov Mar 21, 2019
bbd4e52
Revert "GoModules: Checksum mismatch:"
Stebalien Mar 21, 2019
d39ad78
go modules: tidy go.sum
Stebalien Mar 21, 2019
3dd349b
ci: update to go 1.12.1
Stebalien Mar 21, 2019
a0bcf19
make it work in wasm
whyrusleeping Mar 21, 2019
2150884
dialQueue: start the control loop later
Stebalien Mar 27, 2019
f116774
gomod: update libp2p to v0.0.4
vyzo Apr 4, 2019
5dce96b
update readme badges
yusefnapora Apr 9, 2019
0e770a7
farewell gx; thanks for serving us well.
raulk Apr 11, 2019
1430c59
gomod: update go-libp2p
vyzo Apr 12, 2019
4bd4c31
providers: use raw cids as map keys
Stebalien Apr 13, 2019
ca8376d
providers: don't load into the cache on write
Stebalien Apr 13, 2019
75b0e7a
providers: improve time parsing error checking
Stebalien Apr 13, 2019
d33462b
providers: delete expired provider records on load
Stebalien Apr 13, 2019
1334e7a
providers: add some documentation
Stebalien Apr 13, 2019
fe54a2a
providers: make sure to close provider query, even on error
Stebalien Apr 13, 2019
19962a0
providers: use the non-locking LRU
Stebalien Apr 13, 2019
0a5dff7
providers: optimize GC
Stebalien Apr 13, 2019
87652bc
providers: improve test coverage
Stebalien Apr 13, 2019
144b5c1
dep: update go-datastore
Stebalien Apr 13, 2019
e761ef1
Add opencensus basic metrics (#317)
lanzafame Apr 15, 2019
7433908
Remove the old protocol from the defaults
anacrolix Apr 15, 2019
036cb2b
fix(providers): gc
Stebalien Apr 18, 2019
9749126
providers: flush batch on process close
Stebalien Apr 18, 2019
e8ded8c
providers: test partial GC
Stebalien Apr 18, 2019
c22eafe
providers: run datastore GC concurrently
Stebalien Apr 18, 2019
4d0aa5e
query: fix error "leak"
Stebalien Apr 24, 2019
9aedde8
Expose metrics views in a slice (#327)
anacrolix Apr 25, 2019
d841bc2
query: fix a goroutine leak when the routing table is empty
Stebalien Apr 26, 2019
6247c2a
fix: reduce memory used by buffered writers
Stebalien May 8, 2019
5e82d14
dep: update go-libp2p
Stebalien May 17, 2019
bebd23b
dep: update go-libp2p
Stebalien May 22, 2019
5192d82
dep: update go-libp2p
Stebalien May 22, 2019
bc59935
drop the go-context dependency
Stebalien May 24, 2019
9fbb6d7
kill idle streams after 10 minutes
Stebalien May 24, 2019
7522ede
Update deps to latest libp2p
hsanjuan May 24, 2019
32fd59c
migrate to consolidated types. (#344)
raulk May 26, 2019
64ccdd6
Add timeout
michaelavila Jun 14, 2019
a72c18e
Replace delimited reader with msgio VarintReader
bigs Jun 17, 2019
ac23848
Replace delimited reader in message sender
bigs Jun 17, 2019
068f1d5
Address review comments
bigs Jun 18, 2019
67cfea2
feat: make sure to leave time to send provider records
Stebalien Jun 29, 2019
7fff57f
update go.mod
aarshkshah1992 Jul 14, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ version: 2
jobs:
build:
docker:
- image: circleci/golang:1.11
- image: circleci/golang:1.12
environment:
GO111MODULE: "on"
steps:
Expand Down
1 change: 0 additions & 1 deletion .gx/lastpubver

This file was deleted.

4 changes: 1 addition & 3 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,13 @@ os:
language: go

go:
- 1.11.x
- 1.12.x

env:
global:
- GOTFLAGS="-race"
- IPFS_REUSEPORT=false
matrix:
- BUILD_DEPTYPE=gx
- BUILD_DEPTYPE=gomod


Expand All @@ -25,7 +24,6 @@ script:

cache:
directories:
- $GOPATH/src/gx
- $GOPATH/pkg/mod
- /home/travis/.cache/go-build

Expand Down
15 changes: 0 additions & 15 deletions Makefile

This file was deleted.

12 changes: 8 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# go-libp2p-kad-dht

[![](https://img.shields.io/badge/made%20by-Protocol%20Labs-blue.svg?style=flat-square)](http://ipn.io)
[![](https://img.shields.io/badge/project-libp2p-blue.svg?style=flat-square)](http://github.com/libp2p/libp2p)
[![](https://img.shields.io/badge/freenode-%23ipfs-blue.svg?style=flat-square)](http://webchat.freenode.net/?channels=%23ipfs)
[![standard-readme compliant](https://img.shields.io/badge/standard--readme-OK-green.svg?style=flat-square)](https://github.com/RichardLitt/standard-readme)
[![](https://img.shields.io/badge/made%20by-Protocol%20Labs-blue.svg?style=flat-square)](https://protocol.ai)
[![](https://img.shields.io/badge/project-libp2p-yellow.svg?style=flat-square)](https://libp2p.io)
[![](https://img.shields.io/badge/freenode-%23libp2p-yellow.svg?style=flat-square)](http://webchat.freenode.net/?channels=%23yellow)
[![GoDoc](https://godoc.org/github.com/libp2p/go-libp2p-kad-dht?status.svg)](https://godoc.org/github.com/libp2p/go-libp2p-kad-dht)
[![Build Status](https://travis-ci.org/libp2p/go-libp2p-kad-dht.svg?branch=master)](https://travis-ci.org/libp2p/go-libp2p-kad-dht)
[![Discourse posts](https://img.shields.io/discourse/https/discuss.libp2p.io/posts.svg)](https://discuss.libp2p.io)

> A Kademlia DHT implementation on go-libp2p

Expand Down Expand Up @@ -37,3 +37,7 @@ Small note: If editing the README, please conform to the [standard-readme](https
## License

[MIT](LICENSE) © Protocol Labs Inc.

---

The last gx published version of this module was: 4.4.34: QmXuNFLZc6Nb5akB4sZsxK3doShsFKT1sZFvxLXJvZQwAW
56 changes: 40 additions & 16 deletions dht.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,21 @@ import (
"bytes"
"context"
"errors"
"fmt"
"sync"
"time"

"github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p-core/network"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p-core/peerstore"
"github.com/libp2p/go-libp2p-core/protocol"
"github.com/libp2p/go-libp2p-core/routing"

"go.opencensus.io/tag"
"golang.org/x/xerrors"

"github.com/libp2p/go-libp2p-kad-dht/metrics"
opts "github.com/libp2p/go-libp2p-kad-dht/opts"
pb "github.com/libp2p/go-libp2p-kad-dht/pb"
providers "github.com/libp2p/go-libp2p-kad-dht/providers"
Expand All @@ -19,15 +29,9 @@ import (
logging "github.com/ipfs/go-log"
goprocess "github.com/jbenet/goprocess"
goprocessctx "github.com/jbenet/goprocess/context"
host "github.com/libp2p/go-libp2p-host"
kb "github.com/libp2p/go-libp2p-kbucket"
inet "github.com/libp2p/go-libp2p-net"
peer "github.com/libp2p/go-libp2p-peer"
pstore "github.com/libp2p/go-libp2p-peerstore"
protocol "github.com/libp2p/go-libp2p-protocol"
record "github.com/libp2p/go-libp2p-record"
recpb "github.com/libp2p/go-libp2p-record/pb"
routing "github.com/libp2p/go-libp2p-routing"
base32 "github.com/whyrusleeping/base32"
)

Expand All @@ -38,11 +42,11 @@ var logger = logging.Logger("dht")
const NumBootstrapQueries = 5

// IpfsDHT is an implementation of Kademlia with S/Kademlia modifications.
// It is used to implement the base IpfsRouting module.
// It is used to implement the base Routing module.
type IpfsDHT struct {
host host.Host // the network services we need
self peer.ID // Local peer (yourself)
peerstore pstore.Peerstore // Peer Registry
host host.Host // the network services we need
self peer.ID // Local peer (yourself)
peerstore peerstore.Peerstore // Peer Registry

datastore ds.Datastore // Local data

Expand All @@ -68,7 +72,7 @@ type IpfsDHT struct {
// guarantee, but we can use them to aid refactoring.
var (
_ routing.ContentRouting = (*IpfsDHT)(nil)
_ routing.IpfsRouting = (*IpfsDHT)(nil)
_ routing.Routing = (*IpfsDHT)(nil)
_ routing.PeerRouting = (*IpfsDHT)(nil)
_ routing.PubKeyFetcher = (*IpfsDHT)(nil)
_ routing.ValueStore = (*IpfsDHT)(nil)
Expand Down Expand Up @@ -136,7 +140,7 @@ func makeDHT(ctx context.Context, h host.Host, dstore ds.Batching, protocols []p
cmgr.UntagPeer(p, "kbucket")
}

return &IpfsDHT{
dht := &IpfsDHT{
datastore: dstore,
self: h.ID(),
peerstore: h.Peerstore(),
Expand All @@ -148,6 +152,10 @@ func makeDHT(ctx context.Context, h host.Host, dstore ds.Batching, protocols []p
routingTable: rt,
protocols: protocols,
}

dht.ctx = dht.newContextWithLocalTags(ctx)

return dht
}

// putValueToPeer stores the given key/value pair at the peer 'p'
Expand Down Expand Up @@ -175,7 +183,7 @@ var errInvalidRecord = errors.New("received invalid record")
// key. It returns either the value or a list of closer peers.
// NOTE: It will update the dht's peerstore with any new addresses
// it finds for the given peer.
func (dht *IpfsDHT) getValueOrPeers(ctx context.Context, p peer.ID, key string) (*recpb.Record, []*pstore.PeerInfo, error) {
func (dht *IpfsDHT) getValueOrPeers(ctx context.Context, p peer.ID, key string) (*recpb.Record, []*peer.AddrInfo, error) {

pmes, err := dht.getValueSingle(ctx, p, key)
if err != nil {
Expand Down Expand Up @@ -271,12 +279,12 @@ func (dht *IpfsDHT) Update(ctx context.Context, p peer.ID) {
}

// FindLocal looks for a peer with a given ID connected to this dht and returns the peer and the table it was found in.
func (dht *IpfsDHT) FindLocal(id peer.ID) pstore.PeerInfo {
func (dht *IpfsDHT) FindLocal(id peer.ID) peer.AddrInfo {
switch dht.host.Network().Connectedness(id) {
case inet.Connected, inet.CanConnect:
case network.Connected, network.CanConnect:
return dht.peerstore.PeerInfo(id)
default:
return pstore.PeerInfo{}
return peer.AddrInfo{}
}
}

Expand Down Expand Up @@ -413,3 +421,19 @@ func (dht *IpfsDHT) Ping(ctx context.Context, p peer.ID) error {
}
return nil
}

// newContextWithLocalTags returns a new context.Context with the InstanceID and
// PeerID keys populated. It will also take any extra tags that need adding to
// the context as tag.Mutators.
func (dht *IpfsDHT) newContextWithLocalTags(ctx context.Context, extraTags ...tag.Mutator) context.Context {
extraTags = append(
extraTags,
tag.Upsert(metrics.KeyPeerID, dht.self.Pretty()),
tag.Upsert(metrics.KeyInstanceID, fmt.Sprintf("%p", dht)),
)
ctx, _ = tag.New(
ctx,
extraTags...,
) // ignoring error as it is unrelated to the actual function of this code.
return ctx
}
10 changes: 5 additions & 5 deletions dht_bootstrap.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@ import (
"fmt"
"time"

"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p-core/routing"

u "github.com/ipfs/go-ipfs-util"
peer "github.com/libp2p/go-libp2p-peer"
pstore "github.com/libp2p/go-libp2p-peerstore"
routing "github.com/libp2p/go-libp2p-routing"
multiaddr "github.com/multiformats/go-multiaddr"
"github.com/multiformats/go-multiaddr"
_ "github.com/multiformats/go-multiaddr-dns"
)

Expand Down Expand Up @@ -112,7 +112,7 @@ func newRandomPeerId() peer.ID {
}

// Traverse the DHT toward the given ID.
func (dht *IpfsDHT) walk(ctx context.Context, target peer.ID) (pstore.PeerInfo, error) {
func (dht *IpfsDHT) walk(ctx context.Context, target peer.ID) (peer.AddrInfo, error) {
// TODO: Extract the query action (traversal logic?) inside FindPeer,
// don't actually call through the FindPeer machinery, which can return
// things out of the peer store etc.
Expand Down
Loading