Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hole Punching #806

Merged
merged 127 commits into from
Apr 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
127 commits
Select commit Hold shift + click to select a range
2033712
[skip ci] Start of upgrade refacto
Menduist Nov 22, 2022
df97314
kinda working
Menduist Nov 23, 2022
1ab9cc9
remove upgraded system
Menduist Nov 23, 2022
8b76889
re-add custom matcher
Menduist Nov 23, 2022
b954dad
Simplify connManager
Menduist Nov 23, 2022
c0316e3
fix
Menduist Nov 23, 2022
12ac83f
fix tests
Menduist Nov 24, 2022
7ad1ea8
try fix test
Menduist Nov 25, 2022
263b9e8
Fix GossipSub race condition
Menduist Nov 25, 2022
7b98af6
fix more race conditions
Menduist Nov 25, 2022
99bc8ee
Fix GossipSub race condition
Menduist Nov 25, 2022
6992d81
Merge remote-tracking branch 'origin/unstable' into upgraderefacto
Menduist Jan 2, 2023
a6cec9f
Better fix
Menduist Jan 2, 2023
696a5b4
Merge branch 'fixgossiprace' into upgraderefacto
Menduist Jan 2, 2023
92a425c
Fix typo
Menduist Jan 3, 2023
95fb447
Merge branch 'fixgossiprace' into upgraderefacto
Menduist Jan 3, 2023
7a74dd3
fix ci
Menduist Jan 3, 2023
5510ef7
Merge remote-tracking branch 'origin/unstable' into upgraderefacto
Menduist Jan 10, 2023
281a15a
Merge remote-tracking branch 'origin/unstable' into upgraderefacto
Menduist Jan 24, 2023
6acf4b4
Merge remote-tracking branch 'origin/unstable' into upgraderefacto
Menduist Jan 25, 2023
c279e5a
Fix pubsub
Menduist Jan 25, 2023
5c6fe92
fix short agent
Menduist Jan 30, 2023
5dc290b
Update libp2p/multistream.nim
Menduist Jan 30, 2023
c5f2dbd
Merge remote-tracking branch 'origin/unstable' into upgraderefacto
Menduist Feb 21, 2023
de06b46
Add getWrapped to YamuxChannel
Menduist Feb 21, 2023
2ab82c0
Merge remote-tracking branch 'origin/upgraderefacto' into upgraderefacto
Menduist Feb 21, 2023
685966c
fix autonat
Menduist Feb 21, 2023
8618c85
Merge branch 'unstable' into upgraderefacto
diegomrsantos Feb 22, 2023
b07e66b
Merge remote-tracking branch 'origin/unstable' into upgraderefacto
Menduist Mar 2, 2023
c9cc2bc
Trigger events before identify
Menduist Mar 2, 2023
1fee13c
ObservedMAManager
diegomrsantos Mar 2, 2023
19de3d7
improvements
diegomrsantos Mar 3, 2023
7627325
replace the heap by a seq
diegomrsantos Mar 3, 2023
92bfd83
Move manager to identify
diegomrsantos Mar 6, 2023
4fc7b6d
Simplify api
diegomrsantos Mar 6, 2023
22d6c05
Improvements after code review
diegomrsantos Mar 8, 2023
aee8dcf
Merge branch 'unstable' into observed-addr-manager
diegomrsantos Mar 8, 2023
7cb27cc
More fixes
diegomrsantos Mar 8, 2023
c462503
move procs here for reuse
diegomrsantos Mar 10, 2023
e6908d0
improve naming
diegomrsantos Mar 10, 2023
a378b2d
remove proc
diegomrsantos Mar 10, 2023
f4b34f2
Add an AddressMapper to the AutoRelayService
diegomrsantos Mar 13, 2023
e1c4c9e
Remove AddressMapper when AutonatService stops
diegomrsantos Mar 14, 2023
a38cd88
Bind to local addr when behind a NAT
diegomrsantos Mar 7, 2023
2289e82
Use ReusePort when starting
diegomrsantos Mar 8, 2023
21d9835
Merge branch 'unstable' into transport-hole-punching
diegomrsantos Mar 24, 2023
113d4af
update chronos
diegomrsantos Mar 27, 2023
d72e110
use SocketFlags
diegomrsantos Mar 27, 2023
8988e68
remove duplicate ports check
diegomrsantos Mar 27, 2023
34f5f0d
Make Unknown the NetworkReachability default value
diegomrsantos Mar 27, 2023
341f983
Messages
diegomrsantos Dec 19, 2022
ece6802
Basic version
diegomrsantos Dec 19, 2022
e415d9c
Basic messages test
diegomrsantos Dec 19, 2022
cda7705
dcutr
diegomrsantos Jan 18, 2023
dd69bb0
small refactoring
diegomrsantos Feb 14, 2023
55b9ae4
Split dcutr files
diegomrsantos Feb 15, 2023
5e8744d
Fix several issues
diegomrsantos Mar 10, 2023
080ae37
fix server side
diegomrsantos Mar 10, 2023
f568a1e
fix typo
diegomrsantos Mar 10, 2023
9c881cf
upgrade during simultaneous conn
diegomrsantos Mar 16, 2023
e98ad5b
fix for nim 1.2
diegomrsantos Mar 16, 2023
83c2daa
fix compilation
diegomrsantos Mar 27, 2023
84ad89f
fix text and make it run
diegomrsantos Mar 27, 2023
2745a49
fix for nim 1.2
diegomrsantos Mar 27, 2023
051b1cf
small fixes
diegomrsantos Mar 28, 2023
b7d97b2
use a proc param instead
diegomrsantos Mar 28, 2023
b16f7a5
simplify test
diegomrsantos Mar 28, 2023
b467d9d
simplify upgrade
diegomrsantos Mar 28, 2023
59be505
fix for nim 1.2
diegomrsantos Mar 28, 2023
8745c6e
fix for nim 1.2
diegomrsantos Mar 28, 2023
7d40521
remove unnecessary fields from client
diegomrsantos Mar 29, 2023
187fe6e
naming boolean flags
diegomrsantos Mar 29, 2023
daeeec9
Update libp2p/dial.nim
diegomrsantos Mar 29, 2023
6ee8c4e
remove unnecessary proc
diegomrsantos Mar 29, 2023
e57630d
always use peerId in secure
diegomrsantos Mar 29, 2023
a98c490
guessDialableAddr must use listenAddrs
diegomrsantos Mar 29, 2023
e1e6839
improve imports and test comment
diegomrsantos Mar 29, 2023
7beef85
remove sleep
diegomrsantos Apr 3, 2023
06a8f83
testing fix for i386
diegomrsantos Apr 4, 2023
9b1eb3e
update chronos version
diegomrsantos Apr 4, 2023
1b5e4fc
fix test
diegomrsantos Apr 4, 2023
836ba24
support only tcp addrs
diegomrsantos Apr 5, 2023
2fcea83
server connects in parallel
diegomrsantos Apr 5, 2023
6f5bdfd
client connects in parallel
diegomrsantos Apr 5, 2023
2aa4253
remove unnecessary proc
diegomrsantos Apr 5, 2023
2d8d56c
remove unnecessary line
diegomrsantos Apr 6, 2023
18addd2
simplifying send proc
diegomrsantos Apr 6, 2023
4b03405
Merge remote-tracking branch 'origin/unstable' into dcutr
diegomrsantos Apr 6, 2023
d5c4d94
testing error cases with stub switch
diegomrsantos Apr 9, 2023
ff0dc49
reraising error
diegomrsantos Apr 11, 2023
25258f9
Apply suggestions from code review
diegomrsantos Apr 11, 2023
2c9b495
Update libp2p/protocols/connectivity/dcutr/client.nim
diegomrsantos Apr 12, 2023
873911c
changes for code review
diegomrsantos Apr 12, 2023
1823073
reraising error on the server
diegomrsantos Apr 12, 2023
2d22ba1
typo
diegomrsantos Apr 12, 2023
3349949
Basic version which asks peers recently connected about our nat status
diegomrsantos Nov 21, 2022
6758964
Add more test
diegomrsantos Nov 22, 2022
d8eb0dd
Refactor hp service
diegomrsantos Nov 29, 2022
80b9d35
Remove sleepAsync
diegomrsantos Nov 30, 2022
690d926
Fix hole punching test
diegomrsantos Dec 1, 2022
fc8c176
Rename NetworkReachability values
diegomrsantos Dec 2, 2022
0897a25
Ask a configurable number of random peers
diegomrsantos Dec 2, 2022
1d5d237
Ask only peers with at least one out connection
diegomrsantos Dec 2, 2022
ca75e99
Fix rebase issues
diegomrsantos Dec 16, 2022
bb6f0af
Direct conn
diegomrsantos Jan 27, 2023
41125d3
Waiting for PR in chronos
diegomrsantos Feb 14, 2023
d2bae1d
small refactoring
diegomrsantos Feb 14, 2023
0a4c383
Fix some issues and set reachability in transports
diegomrsantos Mar 8, 2023
ea22efc
add dcutr to hp
diegomrsantos Mar 10, 2023
c53bed6
relayed conn isn't necessary in tryStartingDirectConn
diegomrsantos Mar 10, 2023
4938064
improve naming
diegomrsantos Mar 10, 2023
7101700
fix export
diegomrsantos Mar 10, 2023
b58cc2a
temp fix for multiaddr problems
diegomrsantos Mar 14, 2023
ed9f26b
remove unnecessary proc
diegomrsantos Mar 16, 2023
49592d1
remove comment
diegomrsantos Mar 16, 2023
d57bc41
improve import
diegomrsantos Mar 16, 2023
f808cf7
fix for nim 1.2
diegomrsantos Mar 16, 2023
f5cd073
improve naming
diegomrsantos Mar 17, 2023
a0cefe5
several fixes and more tests
diegomrsantos Mar 21, 2023
f8b578a
remove echo
diegomrsantos Mar 21, 2023
9f86add
fixes after rebase
diegomrsantos Apr 5, 2023
39ac49f
increase test coverage
diegomrsantos Apr 10, 2023
5f4745e
fixes for code review
diegomrsantos Apr 12, 2023
5a800d6
Merge branch 'unstable' into hole-punching
diegomrsantos Apr 14, 2023
efb1c79
Merge branch 'unstable' into hole-punching
diegomrsantos Apr 14, 2023
4d2d21c
improve log
diegomrsantos Apr 14, 2023
f05b2c3
handle relay addrs in the address book
diegomrsantos Apr 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions libp2p/protocols/connectivity/autonat/client.nim
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,6 @@ import ../../../switch,
../../../peerid
import core

export core

logScope:
topics = "libp2p autonat"

Expand Down
8 changes: 4 additions & 4 deletions libp2p/protocols/connectivity/autonat/service.nim
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,12 @@ import chronos, metrics
import ../../../switch
import ../../../wire
import client
from core import NetworkReachability, AutonatUnreachableError
import ../../../utils/heartbeat
import ../../../crypto/crypto

export options, core.NetworkReachability

logScope:
topics = "libp2p autonatservice"

Expand All @@ -30,7 +33,7 @@ type
newConnectedPeerHandler: PeerEventHandler
addressMapper: AddressMapper
scheduleHandle: Future[void]
networkReachability: NetworkReachability
networkReachability*: NetworkReachability
confidence: Option[float]
answers: Deque[NetworkReachability]
autonatClient: AutonatClient
Expand Down Expand Up @@ -71,9 +74,6 @@ proc new*(
dialTimeout: dialTimeout,
enableAddressMapper: enableAddressMapper)

proc networkReachability*(self: AutonatService): NetworkReachability {.inline.} =
return self.networkReachability

proc callHandler(self: AutonatService) {.async.} =
if not isNil(self.statusAndConfidenceHandler):
await self.statusAndConfidenceHandler(self.networkReachability, self.confidence)
Expand Down
2 changes: 2 additions & 0 deletions libp2p/protocols/connectivity/dcutr/client.nim
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ import ../../protocol,
../../../switch,
../../../utils/future

export DcutrError

type
DcutrClient* = ref object
connectTimeout: Duration
Expand Down
1 change: 1 addition & 0 deletions libp2p/protocols/connectivity/dcutr/server.nim
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import ../../protocol,
../../../switch,
../../../utils/future

export DcutrError
export chronicles

type Dcutr* = ref object of LPProtocol
Expand Down
2 changes: 1 addition & 1 deletion libp2p/protocols/connectivity/relay/relay.nim
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ proc createReserveResponse(
status: some(Ok))
return ok(msg)

proc isRelayed(conn: Connection): bool =
proc isRelayed*(conn: Connection): bool =
var wrappedConn = conn
while not isNil(wrappedConn):
if wrappedConn of RelayConnection:
Expand Down
105 changes: 105 additions & 0 deletions libp2p/services/hpservice.nim
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Nim-LibP2P
# Copyright (c) 2022 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))
# * MIT license ([LICENSE-MIT](LICENSE-MIT))
# at your option.
# This file may not be copied, modified, or distributed except according to
# those terms.

when (NimMajor, NimMinor) < (1, 4):
{.push raises: [Defect].}
else:
{.push raises: [].}

import std/[tables, sequtils]

import chronos, chronicles

import ../switch, ../wire
import ../protocols/rendezvous
import ../services/autorelayservice
import ../discovery/[rendezvousinterface, discoverymngr]
import ../protocols/connectivity/relay/relay
import ../protocols/connectivity/autonat/service
import ../protocols/connectivity/dcutr/[client, server]


logScope:
topics = "libp2p hpservice"

type
HPService* = ref object of Service
newConnectedPeerHandler: PeerEventHandler
onNewStatusHandler: StatusAndConfidenceHandler
autoRelayService: AutoRelayService
autonatService: AutonatService
isPublicIPAddrProc: IsPublicIPAddrProc

IsPublicIPAddrProc* = proc(ta: TransportAddress): bool {.gcsafe, raises: [Defect].}

proc new*(T: typedesc[HPService], autonatService: AutonatService, autoRelayService: AutoRelayService,
isPublicIPAddrProc: IsPublicIPAddrProc = isGlobal): T =
return T(autonatService: autonatService, autoRelayService: autoRelayService, isPublicIPAddrProc: isPublicIPAddrProc)

proc tryStartingDirectConn(self: HPService, switch: Switch, peerId: PeerId): Future[bool] {.async.} =
await sleepAsync(500.milliseconds) # wait for AddressBook to be populated
for address in switch.peerStore[AddressBook][peerId]:
try:
let ta = initTAddress(address)
if ta.isOk() and self.isPublicIPAddrProc(ta.get()):
await switch.connect(peerId, @[address], true, false)
debug "Direct connection created."
return true
except CatchableError as err:
debug "Failed to create direct connection.", err = err.msg
continue
return false

method setup*(self: HPService, switch: Switch): Future[bool] {.async.} =
var hasBeenSetup = await procCall Service(self).setup(switch)
hasBeenSetup = hasBeenSetup and await self.autonatService.setup(switch)

if hasBeenSetup:
let dcutrProto = Dcutr.new(switch)
switch.mount(dcutrProto)

self.newConnectedPeerHandler = proc (peerId: PeerId, event: PeerEvent): Future[void] {.async.} =
try:
let conn = switch.connManager.selectMuxer(peerId).connection
if isRelayed(conn) and conn.transportDir == Direction.In:
if await self.tryStartingDirectConn(switch, peerId):
await conn.close()
return
let dcutrClient = DcutrClient.new()
var natAddrs = switch.peerStore.getMostObservedProtosAndPorts()
if natAddrs.len == 0:
natAddrs = switch.peerInfo.listenAddrs.mapIt(switch.peerStore.guessDialableAddr(it))
await dcutrClient.startSync(switch, peerId, natAddrs)
await sleepAsync(2000.milliseconds) # grace period before closing relayed connection
await conn.close()
except CatchableError as err:
debug "Hole punching failed during dcutr", err = err.msg

switch.connManager.addPeerEventHandler(self.newConnectedPeerHandler, PeerEventKind.Joined)

self.onNewStatusHandler = proc (networkReachability: NetworkReachability, confidence: Option[float]) {.gcsafe, async.} =
if networkReachability == NetworkReachability.NotReachable:
discard await self.autoRelayService.setup(switch)
elif networkReachability == NetworkReachability.Reachable:
discard await self.autoRelayService.stop(switch)

# We do it here instead of in the AutonatService because this is useful only when hole punching.
for t in switch.transports:
t.networkReachability = networkReachability

self.autonatService.statusAndConfidenceHandler(self.onNewStatusHandler)
return hasBeenSetup

method run*(self: HPService, switch: Switch) {.async, public.} =
await self.autonatService.run(switch)

method stop*(self: HPService, switch: Switch): Future[bool] {.async, public.} =
discard await self.autonatService.stop(switch)
if not isNil(self.newConnectedPeerHandler):
switch.connManager.removePeerEventHandler(self.newConnectedPeerHandler, PeerEventKind.Joined)
2 changes: 1 addition & 1 deletion libp2p/wire.nim
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ proc initTAddress*(ma: MultiAddress): MaResult[TransportAddress] =
res.port = Port(fromBytesBE(uint16, pbuf))
ok(res)
else:
err("MultiAddress must be wire address (tcp, udp or unix)")
err("MultiAddress must be wire address (tcp, udp or unix): " & $ma)

proc connect*(
ma: MultiAddress,
Expand Down
1 change: 1 addition & 0 deletions tests/stubs/autonatclientstub.nim
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import ../../libp2p/[protocols/connectivity/autonat/client,
peerid,
multiaddress,
switch]
from ../../libp2p/protocols/connectivity/autonat/core import NetworkReachability, AutonatUnreachableError, AutonatError

type
AutonatClientStub* = ref object of AutonatClient
Expand Down
48 changes: 48 additions & 0 deletions tests/stubs/switchstub.nim
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Nim-LibP2P
# Copyright (c) 2023 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))
# * MIT license ([LICENSE-MIT](LICENSE-MIT))
# at your option.
# This file may not be copied, modified, or distributed except according to
# those terms.

{.used.}

when (NimMajor, NimMinor) < (1, 4):
{.push raises: [Defect].}
else:
{.push raises: [].}

import chronos
import ../../libp2p/[peerid, multiaddress, switch]

type
SwitchStub* = ref object of Switch
switch*: Switch
connectStub*: proc(): Future[void] {.async.}

method connect*(
self: SwitchStub,
peerId: PeerId,
addrs: seq[MultiAddress],
forceDial = false,
reuseConnection = true,
upgradeDir = Direction.Out) {.async.} =
if (self.connectStub != nil):
await self.connectStub()
else:
await self.switch.connect(peerId, addrs, forceDial, reuseConnection, upgradeDir)

proc new*(T: typedesc[SwitchStub], switch: Switch, connectStub: proc (): Future[void] {.async.} = nil): T =
return SwitchStub(
switch: switch,
peerInfo: switch.peerInfo,
ms: switch.ms,
transports: switch.transports,
connManager: switch.connManager,
peerStore: switch.peerStore,
dialer: switch.dialer,
nameResolver: switch.nameResolver,
services: switch.services,
connectStub: connectStub)
9 changes: 9 additions & 0 deletions tests/stubs/torstub.nim
Original file line number Diff line number Diff line change
@@ -1,3 +1,12 @@
# Nim-LibP2P
# Copyright (c) 2023 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))
# * MIT license ([LICENSE-MIT](LICENSE-MIT))
# at your option.
# This file may not be copied, modified, or distributed except according to
# those terms.

{.used.}

when (NimMajor, NimMinor) < (1, 4):
Expand Down
Loading