Road to Tor v3 support (BIP155) #2411
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
furszy
changed the title
This was referenced Jun 8, 2021
Merged
furszy
force-pushed
the
2021_road_to_tor
branch
2 times, most recently
from
df14e88
to
715bcad
Compare
furszy
force-pushed
the
2021_road_to_tor
branch
5 times, most recently
from
74f7d5f
to
dc0bf03
Compare
furszy
force-pushed
the
2021_road_to_tor
branch
2 times, most recently
from
5bf3237
to
becca5f
Compare
|
will this solve #2454 ? (currently unable to use tor/onion addresses with master nodes (tor ver 2.0) |
|
Not related, this is for Tor v3 onion addresses support. Just answered you there. |
furszy
force-pushed
the
2021_road_to_tor
branch
4 times, most recently
from
820e87a
to
b10aebf
Compare
furszy
added a commit
that referenced
this pull request
Jul 1, 2021
a470c34 Backport attributes.h and connect it to base58.h functions only (practicalswift) 0247f6f util: Don't allow base58-decoding of std::string:s containing non-base58 characters (practicalswift) 70c480c tests: Add tests for base58-decoding of std::string:s containing non-base58 characters (practicalswift) 9d481be Add bounds checks in key_io before DecodeBase58Check (Pieter Wuille) eac71b5 Finish Encode/Decode destination functions move from base58 to key_io. (furszy) 2e9376c Pass a maximum output length to DecodeBase58 and DecodeBase58Check (Pieter Wuille) c93e19f Clean duplicate usage of DecodeSecret & EncodeSecret. (furszy) 4d4160e Stop using CBase58Data for ext keys (furszy) e861cda Backport string ToUpper and ToLower. (furszy) f6c2872 util: Add Join helper to join a list of strings (MarcoFalke) 32c1e42 Add tests for util/vector.h's Cat and Vector (Pieter Wuille) dc42563 Add some general std::vector utility functions (Pieter Wuille) Pull request description: Decoupled from #2411 Tor's v3 addr support, built on top of #2359. This PR finishes the address encoding cleanup, removing the `CBase58`, `CBitcoinSecret`, `CBitcoinExtKey`, and `CBitcoinExtPubKey` classes, in favor of using the KeyIO::Encode/Decode functions. Furthermore, all PIVX-specific address logic is moved to key_io.{h,cpp}, leaving base58.{h,cpp} as a pure utility that implements the base58 encoding/decoding logic. Plus, includes some general utility functions for std::vector and std::string. Adaptation of the following PRs: * bitcoin#11372. * bitcoin#16670. (without faebf62) * bitcoin#16889. * bitcoin#17511. * bitcoin#17721. ACKs for top commit: random-zebra: rebase utACK a470c34 Fuzzbawls: ACK a470c34 Tree-SHA512: 7a3e1ea0f86c7dab960a5761a666dc7eb291d749e1e9cc24583eec2d6114ca47bc6b9ad50c1c7ff2ecba7f3f60100ce7c0ee8522dc3a2f29d6d79cb052187e0d
furszy
added a commit
that referenced
this pull request
Jul 1, 2021
21f05c1 net: drop unused connman param (Cory Fields) 50853a2 net: use an interface class rather than signals for message processing (furszy) 67757cd net: pass CConnman via pointer rather than reference (furszy) Pull request description: Another decouple from #2411. Coming from bitcoin#10756. > See individual commits. > Benefits: > * Allows us to begin moving stuff out of CNode and into CNodeState. > * Drops boost dependency and overhead > * Drops global signal registration > * Friendlier backtraces ACKs for top commit: random-zebra: ACK 21f05c1 Fuzzbawls: ACK 21f05c1 Tree-SHA512: 8a1ace6d9b8dd2a36d0e1b1465b1f71e7c0a5fcd9a33462210a724cc6eafc119166bd0ee9e9db7862ac41dc62bfa3373c51f575d138b42ec0e140b2524e831d4
furszy
force-pushed
the
2021_road_to_tor
branch
2 times, most recently
from
12f581c
to
4107da6
Compare
|
Rebased on master after decoupled PRs merge + squashed some commits. From 179 commits to 132 now :). Next in the line is #2412 🤘. |
furszy
added a commit
that referenced
this pull request
Jul 5, 2021
bd4b846 Remove old serialization primitives (Pieter Wuille) 060d62b Convert the last, non-trivial, serialization functions to the new form (furszy) 8c74c09 Convert LimitedString to formatter (Pieter Wuille) f021897 Fix CDiskBlockIndex serialization of dummy fields for old DB versions (random-zebra) 1ee0cb2 Convert CDiskBlockIndex to new serialization. (furszy) 221bf49 Convert wallet to new serialization (furszy) cf06950 Convert to new serialization (step 3) (furszy) dc0fc95 Remove old MESS_VER_STRMESS message version try-catch. (furszy) 35fca11 Convert Qt to new serialization (Pieter Wuille) 3f7826e Add comments to CustomUintFormatter (Pieter Wuille) eccd473 Convert to new serialization (step 2). Focused on object's serializations that doesn't require an special treatment. (furszy) 0f15784 Convert everything except wallet/qt to new serialization (step 1) (Pieter Wuille) 3d3ee64 Convert merkleblock to new serialization (Pieter Wuille) 13577fb Add SER_READ and SER_WRITE for read/write-dependent statements (Russell Yanofsky) 7344c1a Extend CustomUintFormatter to support enums (Russell Yanofsky) c4d6228 Merge BigEndian functionality into CustomUintFormatter (Pieter Wuille) 3765d6c Add static_asserts to ser_X_to_Y() methods (Samer Afach) 806213a Fix a violation of C++ standard rules that unions cannot be switched. (Samer Afach) d6380c4 Add CustomUintFormatter (Pieter Wuille) fd29a50 Make VectorFormatter support stateful formatters (Russell Yanofsky) 4e2afad Convert CCompactSize to proper formatter (Pieter Wuille) bb99030 Get rid of VARINT default argument (Pieter Wuille) e107a0c Convert undo.h to new serialization framework (Pieter Wuille) a926ba3 Make std::vector and prevector reuse the VectorFormatter logic (Pieter Wuille) 1dfddce Add custom vector-element formatter (Pieter Wuille) df4e1ba Add a constant for the maximum vector allocation (5 Mbyte) (Pieter Wuille) c2fdeaf Convert compression.h to new serialization framework (Pieter Wuille) aa35991 Add FORMATTER_METHODS, similar to SERIALIZE_METHODS, but for formatters (Pieter Wuille) 3e38199 Move compressor utility functions out of class (Pieter Wuille) 7376a95 Convert chain to new serialization (Pieter Wuille) bbfc55c Convert VARINT to the formatter/Using approach (Pieter Wuille) 39c58a1 Add a generic approach for (de)serialization of objects using code in other classes (Pieter Wuille) ace3895 Convert addrdb/addrman to new serialization (Pieter Wuille) 6bb135e Introduce new serialization macros without casts (Pieter Wuille) ace7d14 Drop minor GetSerializeSize template (Ben Woosley) f05e692 Drop unused GetType() from CSizeComputer (furszy) 5c36b3d Introduce BigEndian wrapper and use it for netaddress ports (Pieter Wuille) fb3c646 Migrate last FLATDATA calls to use Span. (furszy) 1ef2d90 Support serializing Span<unsigned char> and use that instead of FLATDATA (Pieter Wuille) 8fef544 Add Slice: a (pointer, size) array view that acts like a container (Pieter Wuille) Pull request description: Decoupled from #2411, built on top of #2359. Focused on creating the Span class and updating the serialization framework and every object using it up to latest upstream structure (3-4 years ahead of what we currently are in master). We will be up-to-date with them in the area after finishing with #2411 entirely (there are few more updates to the serialization code that comes down #2411 commits line that cannot cherry-pick here). Adapted the following PRs: * bitcoin#12886. * bitcoin#12916. * bitcoin#13558. * bitcoin#17850. * bitcoin#17896. * bitcoin#12752. * bitcoin#17957. * bitcoin#18021. * bitcoin#18087. * bitcoin#18112 (only from 353f376 that we don't support). * bitcoin#18167. * bitcoin#18317. * bitcoin#19032. ACKs for top commit: random-zebra: ACK bd4b846 Fuzzbawls: ACK bd4b846 Tree-SHA512: fe1b31d0976dff97bfeed0f9efeeb4c6c161277529880ede990c9b3fb0ea680f25b4be01b739f6bf7eeca79fa7687c9c2146c403c96e86bc6b052c6dd88e6930
Adapted version of btc@fa1da3d4bfc0511a89f5b19d5a4d89e55ff7ccde
Allow-this-application-to-accept-inbound-connections permission popups and is generally safer. To prevent binding to 127.0.0.1, set self.bind_to_localhost_only = False.
Skip the parts that cannot be run on the host due to lack of IPv6 support or a second interface to bind on, and warn appropriately. Without no strong requirements (besides being Linux only, in which case the test is skipped) left, just add this test to the default in test_runner. Includes suggested changes by John Newbery.
- Even when rpcallowip is specified, only bind localhost - Explicitly bind in run_allowip_test
Even though the format of `peers.dat` was changed in an incompatible way (old software versions <0.21 cannot understand the new file format), it is not guaranteed that old versions will fail to parse it. There is a chance that old versions parse its contents as garbage and use it. Old versions expect the "key size" field to be 32 and fail the parsing if it is not. Thus, we put something other than 32 in it. This will make versions between 0.11.0 and 0.20.1 deterministically fail on the new format. Versions prior to bitcoin#5941 (<0.11.0) will still parse it as garbage. Also, introduce a way to increment the `peers.dat` format in a way that does not necessary make older versions refuse to read it.
Combination of upstream's bitcoin#19961, bitcoin#21753 and bitcoin#22172
v2 addresses were 16 chars long, new v3 addr is 56 chars long and doesn't fit on the topbar. A future good work will be to create a tor-only information dialog, grouping and presenting the tor related information properly.
Don't export `in6addr_loopback` because that upsets `contrib/devtools/symbol-check.py` Fixes bitcoin#20127
And correct MN start not working if the wallet is running both sides.
furszy
force-pushed
the
2021_road_to_tor
branch
from
6e75b36
to
7e74076
Compare
|
Rebased after 2492 merge, added enforcement height for testnet (for block 332,300) and zebra's be46655 for the spork dance. As we are running out of time (only 7 days left until the next superblock), would be better to leave every other PR tagged for v5.3 that does not contain a consensus change or a bugfix for a subsequent v5.4 (that we can release a week or two after v5.3 without any rush. Mostly for #2280 and #2406 that will need some more work and further proper testing) and only be focused on testing what is here and in master currently (more than +150 PRs..) so we can pack it on time. |
random-zebra
previously approved these changes
Aug 10, 2021
There was a problem hiding this comment.
tested running v3 onion masternode on testnet (removing 2feabac) all good.
tested gitian build (merge be46655 on acdbcf8)
d5a129376b55a3f2b5ee68f60a577b828eab605dfef7d84d3716a58b0f884981 pivx-5.2.99-aarch64-linux-gnu-debug.tar.gz
b752241e653e3d7075df326e5557efef82490962f1b5e69ddf03b96b0b2eb4c8 pivx-5.2.99-aarch64-linux-gnu.tar.gz
b2d0c6130c8feb699d1459aebb7f24587ced5fc24dfe3730ca7667d5a88b426b pivx-5.2.99-arm-linux-gnueabihf-debug.tar.gz
90ff8abd1ac1d49736194071582bcb3bbf39bf6bb9fddec84289da8a19dc7459 pivx-5.2.99-arm-linux-gnueabihf.tar.gz
2246d21135ace3b8a61562e05250fdad676766f466f79e0af9561dda437929cb pivx-5.2.99-i686-pc-linux-gnu-debug.tar.gz
f59dd658993ed83fef81b80fe22e609994103726ae6bf761a6f16f10e9c9ea0a pivx-5.2.99-i686-pc-linux-gnu.tar.gz
6b7de40df7364f4a6ca4a9850c78bc91022e88337deee9496b10a6e93a8b65ab pivx-5.2.99-x86_64-linux-gnu-debug.tar.gz
0543f47b5ca9d27bbf88c9bfd0331d974e9b3d060d254fcf4e7b254c6ac778fa pivx-5.2.99-x86_64-linux-gnu.tar.gz
41932488bb747edbb89bbe0f938f6458ae9b3e2105270184c6c6a3e5624fa774 src/pivx-5.2.99.tar.gz
323e92f7e1cd81c04ced477dda950ecd5ccaa543bbb54d1ae834c1d0a6874028 pivx-linux-5.2-res.yml
1f08c53ca11292ebb2ed0d70bbff6f45ee0bf5462da05752fb7d42c49a81dc71 pivx-5.2.99-win-unsigned.tar.gz
32c1bb799871b3582a8a7874752b61b234ca7b074911ff18fe0dfbc070db224c pivx-5.2.99-win64-debug.zip
909c7f9311ef830ba362ff25daa64051b860711bc9f22822a9a93e7e9f3b36e1 pivx-5.2.99-win64-setup-unsigned.exe
902fe60b9b20c4f022aa325a3771e8227aec15366ac1fccade557a79c877a4b2 pivx-5.2.99-win64.zip
582312239f815b41381283c1c6fee1c32e4312385548d2e8c9f9a159d69ba696 src/pivx-5.2.99.tar.gz
1afadc029ab2c9c9087e827ebfbea394824bd2a50c039419a51987e9708c44ed pivx-win-5.2-res.yml
d1606c66356bee93e8180331a18574b3b0ea42eb5cfc23373b16ae68c8ff91ba pivx-5.2.99-osx-unsigned.dmg
140e28f5a31eb8c386706c76ad485a5d7db9aa437434e8f85b857b51d97a4cad pivx-5.2.99-osx-unsigned.tar.gz
fa64a0064d47193c6d162f49b7c3a946f23cc84eae4986356672e10d1f800cbc pivx-5.2.99-osx64.tar.gz
c204f92e39afd8bd357f698199cfc332823fdbfc99402e7e862524cc8d650b48 src/pivx-5.2.99.tar.gz
a132f9d4a4fb891f1b434c346fc5aba9786f2a678c19f0e7c9af41dcbb2150b0 pivx-osx-6.0-res.yml
ACK be46655
So upgraded peers don't create, accept nor broadcast an mnb containing a new addr v2 to non-upgraded peers before the new NU enforcement (introduced in bitcoin#2492).
furszy
force-pushed
the
2021_road_to_tor
branch
from
be46655
to
ecde04a
Compare
Fuzzbawls
approved these changes
Aug 11, 2021
There was a problem hiding this comment.
ACK ecde04a
Played with this quite a bit more, noticed an issue with the MN list not being able to properly identify TORv3 MNs as running on the Onion network. Double checked that the MNB message was using the correct CService address, so it is likely due to old nodes that don't understand ADDRv2 addresses relaying the MNB. Will need to check this again once testnet is updated.
|
Post-merge comment, just confirming that I was able to run it yesterday on testnet as well, it works perfectly. I only had the permission issues which were solved with the @furszy's guidance, but these issues are not related to this PR. Nice work! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Conjunction of a large number of back ports, updates and refactorings that made with the final goal of implementing v3 Onion addresses support (BIP155 https://github.com/bitcoin/bips/blob/master/bip-0155.mediawiki) before the tor v2 addresses EOL, scheduled, by the Tor project, for (1) July 15th: v2 addr support removal from the code base, and (2) October 15th: v2 addr network disable, where every peer in our network running under Tor will loose the connection and drop the network.
As BIP155 describes, this is introducing a new P2P message to gossip longer node addresses over the P2P network. This is required to support new-generation Onion addresses, I2P, and potentially other networks that have longer endpoint addresses than fit in the 128 bits of the current addr message.
In order to achieve the end goal, had to:
Important note:
This PR it is not meant to be merged as a standalone PR, will decouple smaller ones moving on. Adding on each sub-PR its own description isolated from this big monster.
Second note:
This is still a work-in-progress, not ready for testing yet. I'm probably missing to mention few PRs that have already adapted to our sources. Just making it public so can decouple the changes, we can start merging them and i can continue working a bit more confortable (rebase a +170 commits separate branch is not fun..).
List of back ported and adapted PRs:
Span and Serialization:
Net, NetAddress and AddrMan:
Keys and Addresses encoding:
Util:
Bench:
BIP155:
I'm currently working on the PRs marked as "pending", this isn't over, but I'm pretty pretty close :). What a long road..