Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core,btc,app: Manage SPV wallet peers #1931

Merged
merged 9 commits into from
Nov 23, 2022
Merged

Conversation

martonp
Copy link
Contributor

@martonp martonp commented Oct 28, 2022

This change allows the user to see the peers the BTC SPV wallets are connected to, and add new peers.

  • client/asset: Adds a new PeerManager interface with three methods: Peers, AddPeer and RemovePeer. Wallets that implement this interface have the trait WalletTraitPeerManager.
  • client/core: Adds new methods that call the PeerManager methods.
  • client/btc:
    • A SPVPeerManager type is added which manages a key value db and a neutrino chain service. When the user adds a new peer, it is stored in the key value db, and will be automatically connected the next time the wallet is opened.
    • The BTCWallet interface now has three new functions, Peers, AddPeer and RemovePeer, and the BTC, BCH, and LTC SPV wallets each use a SPVPeerManager in order to handle these new functions.
  • app:
    Screen Shot 2022-10-28 at 1 21 28 PM

Screen Shot 2022-11-03 at 6 07 58 PM

go.mod Outdated Show resolved Hide resolved
@chappjc
Copy link
Member

chappjc commented Oct 28, 2022

I really like the Manage Peers view. Good stuff @martonp!

@martonp
Copy link
Contributor Author

martonp commented Oct 30, 2022

@chappjc This will be ready to review once dcrlabs/neutrino-bch#9 and dcrlabs/neutrino-ltc#6 are merged.

Copy link
Member

@buck54321 buck54321 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approach ACK, for sure. This is really well done.

// Peers returns the list of peers that the wallet is connected to. It also
// returns the peers that the user added that the wallet may not currently
// be connected to.
func (w *SPVPeerManager) Peers() ([]*asset.WalletPeer, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why w?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll update, I had these methods on the btcSPVWallet type earlier.

client/asset/btc/spv_wrapper.go Outdated Show resolved Hide resolved
@martonp martonp force-pushed the addPeers branch 2 times, most recently from 073abb4 to dd373b6 Compare October 31, 2022 21:47
@martonp martonp marked this pull request as ready for review October 31, 2022 21:58
Copy link
Member

@JoeGruffins JoeGruffins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the red X could use a column name or a hover over message. It's probably obvious for most but I was confused if maybe it meant I was not connected at first, until I clicked it.

image

Also looks great! Finally I am able to connect to a bchd node and bch is starting to sync on testnet.

client/asset/btc/spv_peer_manager.go Outdated Show resolved Hide resolved
client/asset/bch/spv.go Outdated Show resolved Hide resolved
client/asset/btc/spv_wrapper.go Show resolved Hide resolved
client/asset/interface.go Outdated Show resolved Hide resolved
@JoeGruffins
Copy link
Member

JoeGruffins commented Nov 1, 2022

Probably unrelated to these changes, but client won't shut down as bch is trying to sync.

2022-11-01 07:59:04.627 [TRC] CORE[btc][SPV]: Stopping neutrino DB.
2022-11-01 07:59:04.644 [INF] CORE[btc][SPV]: SPV wallet closed
2022-11-01 07:59:04.644 [INF] CORE: Locking DCR wallet
2022-11-01 07:59:04.644 [ERR] CORE: Failed to lock DCR wallet: wallet locked: account is already locked
2022-11-01 07:59:04.644 [INF] CORE[dcr][SPV]: Unloading wallet
2022-11-01 07:59:04.647 [INF] CORE[dcr][SPV]: SPV wallet closed
2022-11-01 07:59:04.647 [INF] CORE: Locking BCH wallet
2022-11-01 07:59:04.647 [INF] CORE[bch][SPV]: Unloading wallet

10 minutes now... giving up

Even going back to master it seems bch has bricked my dexc. Can start but not shut down. Cannot disable either now.
image

Unrelated to this pr. Will make an issue after some investigation.

@chappjc
Copy link
Member

chappjc commented Nov 1, 2022

Unrelated to this pr. Will make an issue after some investigation.

Will keep an eye out for the issue. There are a number of neutrino-related issues that only really pop out with testnet or mainnet where sync can take actual time.

  1. Wallet sync progress hits 100% way too soon: client/asset/dcr: restart required after restoring native dcr wallet with funds #1816 (comment) It was hung during that too of course.
  2. When the wallets appear to be hung, I always find that in the assetdb neutrino.log it is requesting cfilters, nice and slow, usually 1-by-1 instead of batch. That's something that can be improved with neutrino+query+rescan: improve rescan speed lightninglabs/neutrino#236, which I tested at one point.
  3. Even with the above fix, neutrino sync is non-cancellable. We'd have to figure out why it doesn't respond to shutdown requests when in sync.

EDIT: just made that into an issue: #1934
Can be orthogonal to the bricking of dexc depending on what you find.

Copy link
Member

@buck54321 buck54321 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wallet is ready, but I'm not seeing the button.

image

client/asset/btc/btc.go Outdated Show resolved Hide resolved
client/asset/btc/spv_peer_manager.go Outdated Show resolved Hide resolved
client/asset/btc/spv_peer_manager.go Outdated Show resolved Hide resolved
client/webserver/site/src/html/wallets.tmpl Outdated Show resolved Hide resolved
@martonp
Copy link
Contributor Author

martonp commented Nov 3, 2022

@buck54321 I hadn't updated the cache busters. You should be able to see the button now.

@martonp martonp force-pushed the addPeers branch 3 times, most recently from c387299 to 34d186b Compare November 3, 2022 22:19
Copy link
Member

@JoeGruffins JoeGruffins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, it was working well before the last changes. Not sure what happened but now unable to add peers.

The neutrino logs show:

2022-11-04 12:23:22.381 [DBG] NTRNO: Peer 127.0.0.1:28333 is banned for another 23h4m36.618483257s
2022-11-04 12:23:22.382 [DBG] NTRNO: Peer 127.0.0.1:28336 is banned for another 23h4m36.617422658s
2022-11-04 12:23:22.384 [DBG] NTRNO: Peer 127.0.0.1:28335 is banned for another 23h4m36.615500595s
2022-11-04 12:29:13.708 [DBG] NTRNO: Peer 127.0.0.1:28337 is banned for another 22h58m45.291805946s

And my bchd has logs that correspond with attempting to add:

2022/11/04 12:29:13 http: TLS handshake error from 127.0.0.1:32816: EOF

Also, removing peers shows an error, but also seems to work if I reload the page:
image

@martonp
Copy link
Contributor Author

martonp commented Nov 4, 2022

@JoeGruffins Seems like your node was banned by neutrino for some reason, maybe due to the node not serving the compact filters? I'll add a reconnect button on the screen that will unban and attempt to connect.

Copy link
Member

@JoeGruffins JoeGruffins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like my suggestion of using s.ChainService.ConnectNode(addr, true) is not working for me. It is not adding the node for some reason. I can look into it more.

Comment on lines 186 to 194
// loadSavedPeersFromFile replaces the contents of dexc-peers.json.
func (w *SPVPeerManager) writeSavedPeersToFile(peers map[string]peerSource) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

loadSavedPeersFromFile -> writeSavedPeersToFile

Copy link
Member

@JoeGruffins JoeGruffins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bch issue does not appear to be related to these changes.

@chappjc
Copy link
Member

chappjc commented Nov 9, 2022

My bch issue does not appear to be related to these changes.

So is ConnectNode working for you now?

@JoeGruffins
Copy link
Member

JoeGruffins commented Nov 9, 2022

So is ConnectNode working for you now?

No. Well, yes for bitcoind <-> neturino, no for bchd <-> neutrino. But @martonp was looking into it and it looks like it is a communications issue.

From @martonp:
I've synced a bch testnet node as well.. It's unable to connect because I'm getting an error here: https://github.com/gcash/bchd/blob/5964724713ae9aef9ad334dee56496f4e3bce7e6/peer/peer.go#L2321

I think this warrants a separate issue. I am planning to make one after these changes are finalized, or marton can.

@chappjc chappjc self-requested a review November 10, 2022 19:58
@chappjc
Copy link
Member

chappjc commented Nov 16, 2022

I've synced a bch testnet node as well.. It's unable to connect because I'm getting an error here: https://github.com/gcash/bchd/blob/5964724713ae9aef9ad334dee56496f4e3bce7e6/peer/peer.go#L2321

I think this warrants a separate issue. I am planning to make one after these changes are finalized, or marton can.

I don't understand. The BCH SPV wallet has worked fine with the bchd harness in the past. Testnet too. Are you saying that it's now broken, or it's just broken when manually adding peers in this PR?

@chappjc
Copy link
Member

chappjc commented Nov 17, 2022

I don't understand. The BCH SPV wallet has worked fine with the bchd harness in the past. Testnet too. Are you saying that it's now broken, or it's just broken when manually adding peers in this PR?

I guess you were just talking about this PR. BCH SPV wallet still seems fine to me:

2022-11-17 12:24:36.890 [INF] SYNC: New valid peer 127.0.0.1:49296 (inbound) (neutrino:0.0.4-beta/)                
2022-11-17 12:24:36.891 [DBG] PEER: Received getheaders (locator 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206, stop 00000000
00000000000000000000000000000000000000000000000000000000) from 127.0.0.1:49296 (inbound)                                                        
2022-11-17 12:24:36.891 [DBG] PEER: Sending headers (num 413) to 127.0.0.1:49296 (inbound)
2022-11-17 12:24:36.897 [DBG] PEER: Received getcfheaders (start_height=1, stop_hash=6528106e9d3a34525c603d68818aa24ad412d2c7980b617b595bcedc81e
4ee20) from 127.0.0.1:49296 (inbound)                                                                                                           
2022-11-17 12:24:36.898 [DBG] PEER: Sending cfheaders (stop_hash=6528106e9d3a34525c603d68818aa24ad412d2c7980b617b595bcedc81e4ee20, num_filter_ha
shes=413) to 127.0.0.1:49296 (inbound)                                                                                                          
2022-11-17 12:24:36.901 [DBG] PEER: Received getcfilters from 127.0.0.1:49296 (inbound)                                                         
2022-11-17 12:24:36.901 [DBG] PEER: Sending cfilter to 127.0.0.1:49296 (inbound)

However, bchd still seems to try to discover other peers on regnet and I end up with it connecting to the same bitcoin-cash-node twice and seeing lots of apparently harmless DBG level errors. Silenced with:

diff --git a/dex/testing/btc/base-harness.sh b/dex/testing/btc/base-harness.sh
index afa05287d..b9c565194 100755
--- a/dex/testing/btc/base-harness.sh
+++ b/dex/testing/btc/base-harness.sh
@@ -393,7 +393,7 @@ if [ ! -z "$GODAEMON" ]; then
       OMEGA_RPC_PORT=21558
 
 cat > "${NODE_CONF}" <<EOF
-addpeer=127.0.0.1:${ALPHA_LISTEN_PORT}
+connect=127.0.0.1:${ALPHA_LISTEN_PORT}
 listen=:21577
 rpcuser=user
 rpcpass=pass
@@ -401,7 +401,7 @@ rpclisten=0.0.0.0:${OMEGA_RPC_PORT}
 regtest=1
 rpccert=${OMEGA_DIR}/rpc.cert
 rpckey=${OMEGA_DIR}/rpc.key
-debuglevel=trace
+debuglevel=debug
 EOF
 
 cat > "${CLIENT_CONF}" <<EOF

@chappjc
Copy link
Member

chappjc commented Nov 17, 2022

@JoeGruffins Seems like your node was banned by neutrino for some reason, maybe due to the node not serving the compact filters? I'll add a reconnect button on the screen that will unban and attempt to connect.

I don't think the bchd node should be trying to connect to the neutrino peer, only the other way around. I changed the omega node config from addpeer to connect, and created the following PR as well: gcash/bchd#519
Will take that out of draft after testing this dcrdex PR more with that change.

@martonp
Copy link
Contributor Author

martonp commented Nov 17, 2022

I was able to connect to bchd on testnet as well. I removed the default peer now to avoid confusion.. the default peer was "localhost:28333" which attempted to connect to [::1]:28333, and would fail if bchd was set to listen to 127.0.0.1:2883.

Copy link
Member

@chappjc chappjc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partial review because I want to respond to latest comment from @martonp

client/asset/btc/spv_peer_manager.go Outdated Show resolved Hide resolved
client/asset/btc/spv_peer_manager.go Outdated Show resolved Hide resolved
client/asset/btc/spv_peer_manager.go Outdated Show resolved Hide resolved
client/asset/btc/spv_peer_manager.go Outdated Show resolved Hide resolved
}

// Attempt to look up an IP address associated with the parsed host.
ips, err := net.LookupIP(host)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the last few weeks we've been bit by localhost not resolving on some systems, depending on build tags, OS config, and DNS provider. Two things we might want to do: change our pre-configured localhost peers to 127.0.0.1, and put a manual resolution in this function to resolve localhost to 127.0.0.1 because we can't always rely on net.LookupIP to do it. If we really want to respect the possibility that the user has IPv6 loopback (::1), we could check if host was localhost in the err branch and only then fallback to 127.0.0.1.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We won't get any error if it resolves to the IPv6 address and there's nothing listening over there, neutrino will just keep retrying. So we'd have to separately make a connection to try it out. I think just manually resolving to 127.0.0.1 is fine.. do you think we should put a message on the screen stating that localhost resolves to 127.0.0.1, and users should type [::1] if that's what they need?

Copy link
Member

@chappjc chappjc Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to add messaging about how we resolve localhost IMO. Resolvers are very inconsistent about localhost, and given we won't get an error if it incorrectly resolves to IPv6 and neutrino will retry and fail forever, we should take steps to prevent that.

users should type [::1] if that's what they need?

Totally. If they really want [::1], they can type that.

client/asset/btc/spv_peer_manager.go Outdated Show resolved Hide resolved
client/asset/btc/spv_peer_manager.go Outdated Show resolved Hide resolved
// to the user. If a user previously added a peer that originally connected
// but now the address cannot be resolved to an IP, it should be displayed
// that the wallet was unable to connect to that peer.
w.peers[addr] = &walletPeer{source: source, resolvedName: resolvedAddr}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't resolvedAddr be an empty string here?

client/asset/btc/spv_peer_manager.go Outdated Show resolved Hide resolved
@chappjc
Copy link
Member

chappjc commented Nov 17, 2022

I was able to connect to bchd on testnet as well. I removed the default peer now to avoid confusion.. the default peer was "localhost:28333" which attempted to connect to [::1]:28333, and would fail if bchd was set to listen to 127.0.0.1:2883.

Made a relevant comment in my review here #1931 (comment)

But ACK to removing the localhost peer, regardless of the resolution issues. Since BCH testnet4 at least has a DNS seeder (testnet4.imaginary.cash), unlike LTC testnet. It only returns 1 node unfortunately, itself apparently, but it's something.

client/asset/btc/btc.go Outdated Show resolved Hide resolved
client/asset/btc/spv.go Outdated Show resolved Hide resolved
client/asset/interface.go Outdated Show resolved Hide resolved
client/rpcserver/handlers.go Outdated Show resolved Hide resolved
client/rpcserver/handlers.go Outdated Show resolved Hide resolved
This change allows the user to see the peers the BTC SPV wallets are
connected to, and add new peers.
- `client/asset`: Adds a new `PeerManager` interface with three methods:
  `Peers`, `AddPeer` and `RemovePeer`. Wallets that implement this
  interface have the trait `WalletTraitPeerManager`.
- `client/core`: Adds new methods that call the `PeerManager` methods.
- `client/btc`:
  - A `SPVPeerManager` type is added which manages a key value db and a
    neutrino chain service. When the user adds a new peer, it is stored
    in the key value db, and will be automatically connected the next
    time the wallet is opened.
  - The `BTCWallet` interface now has three new functions,  `Peers`,
    `AddPeer` and `RemovePeer`, and the BTC, BCH, and LTC SPV wallets
    each use a `SPVPeerManager` in order to handle these new functions.
@JoeGruffins
Copy link
Member

JoeGruffins commented Nov 22, 2022

I don't understand. The BCH SPV wallet has worked fine with the bchd harness in the past. Testnet too. Are you saying that it's now broken, or it's just broken when manually adding peers in this PR?

BCH spv has never worked for me, besides one iteration of this pr.

Could it be because I have ipv6 disabled on my computer? I see some discussion about that above.

@chappjc
Copy link
Member

chappjc commented Nov 22, 2022

Could it be because I have ipv6 disabled on my computer? I see some discussion about that above.

That's likely. Now that @martonp has it going to 127.0.0.1, I bet you'll be good now.

@JoeGruffins
Copy link
Member

Oh yeah! It is working now on sim and testnet. I guess that was it.

@chappjc
Copy link
Member

chappjc commented Nov 22, 2022

There are a few unaddressed comments from other reviews, but otherwise good. Please circle back on the comments from @JoeGruffins and @buck54321, thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants