Concurrent network requests #58
Comments
|
Agree.
About how many connections do we speak here? I wonder whether we run a danger that public nodes will get pushed to their performance limits much faster if future wallets overwhelm them with connections. |
|
I'll check resource impact with connections soon. I planned to cap the connection pool at 20. It can also easily be implemented to only open >1 connections if the client has many blocks to scan and disconnect any lingering connections when scanning is done. |
|
One clarifying note: when using libcurl's multi interface, if the server does not support multiplexing and the client tries to make concurrent requests, then libcurl initiates new connections to the server and handles requests concurrently under the hood. Thus, switching to libcurl today and using the multi interface for concurrent requests would in fact negate the usefulness of a new client-side connection pool since it abstracts the "pooling" away. However, the client-side connection pool is simple enough code that I'm still of the view it makes sense to move forward with it. |
j-berman
added a commit
to j-berman/monero
that referenced
this issue
Apr 4, 2024
*How it works*
Assume the user's wallet must start scanning blocks from height
5000.
1. The scanner begins by launching 10 RPC requests in parallel to
fetch chunks of blocks as follows:
```
request 0: { start_height: 5000, max_block_count: 20 }
request 1: { start_height: 5020, max_block_count: 20 }
...
request 9: { start_height: 5180, max_block_count: 20 }
```
2. As soon as any single request completes, the wallet immediately
parses that chunk.
- This is all in parallel. For example, as soon as request 7
responds, the wallet immediately begins parsing that chunk in
parallel to any other chunks it's already parsing.
3. If a chunk does not include a total of max_block_count blocks,
and the chunk is not the tip of the chain, this means there was a
"gap" in the chunk request. The scanner launches another parallel
RPC request to fill in the gap.
- This gap can occur because the server will **not** return a
chunk of blocks greater than 100mb (or 20k txs) via the
/getblocks.bin` RPC endpoint
([`FIND_BLOCKCHAIN_SUPPLEMENT_MAX_SIZE`](https://github.com/monero-project/monero/blob/053ba2cf07649cea8134f8a188685ab7a5365e5c/src/cryptonote_core/blockchain.cpp#L65))
- The gap is from `(req.start_height + res.blocks.size())` to
`(req.start_height + req.max_block_count)`.
4. As soon as the scanner finishes parsing the chunk, it
immediately submits another parallel RPC request.
5. In parallel, the scanner identifies a user's received (and
spent) enotes in each chunk.
- For example, as soon as request 7 responds and the wallet
parses it, the wallet scans that chunk in parallel to any other
chunks it's already scanning.
6. Once a single chunk is fully scanned locally, the scanner
launches a parallel task to fetch and scan the next chunk.
7. Once the scanner reaches the tip of the chain (the terminal
chunk), the scanner terminates.
*Some technical highlights*
- The wallet scanner is backwards compatible with existing daemons
(though it would need to point to an updated daemon to realize the
perf speed-up).
- On error cases such as the daemon going offline, the same wallet
errors that wallet2 uses (that the wallet API expects) are
propagated up to the higher-level Seraphis lib.
- The implementation uses an http client connection pool (reusing
the epee http client) to support parallel network requests
([related](seraphis-migration/wallet3#58)).
- A developer using the scanner can "bring their own blocks/network
implementation" to the scanner by providing a callback function of
the following type as a param to the async scanner constructor:
`std::function<bool(const cryptonote::COMMAND_RPC_GET_BLOCKS_FAST::request, cryptonote::COMMAND_RPC_GET_BLOCKS_FAST::response)>`
UkoeHB
pushed a commit
to UkoeHB/monero
that referenced
this issue
May 7, 2024
Implement async wallet scanner.
Adds a new functional test for direct wallet2 -> live RPC daemon
interactions. This sets up a framework to test pointing the
Seraphis wallet lib to a live daemon.
Tests sending and scanning:
- a normal transfer
- a sweep single (0 change)
- to a single subaddress
- to 3 subaddresses (i.e. scanning using additional pub keys)
* scan machine: option to force reorg avoidance increment first pass
- when pointing to a daemon that does not support returning empty
blocks when the client requests too high of a height, we have to
be careful in our scanner to always request blocks below the chain
tip, in every request.
- by forcing the reorg avoidance increment on first pass, we make
sure clients will always include the reorg avoidance increment
when requesting blocks from the daemon, so the client can expect
the request for blocks should *always* return an ok height.
* core tests: check conversion tool on all legacy enote version types
Stil TODO:
- check complete scanning on all enote types
- hit every branch condition for all enote versions
* conn pool mock: epee http client connection pool
- Enables concurrent network requests using the epee http client.
- Still TODO for production:
1) close_connections
2) require the pool respect max_connections
* enote finding context: IN LegacyUnscannedChunk, OUT ChunkData
- finds owned enotes by legacy view scanning a chunk of blocks
* async: function to remove minimum element from token queue
- Useful when we want to remove elements of the token queue in an
order that is different than insertion order.
* async scanner: scan via RPC, fetching & scanning parallel chunks
*How it works*
Assume the user's wallet must start scanning blocks from height
5000.
1. The scanner begins by launching 10 RPC requests in parallel to
fetch chunks of blocks as follows:
```
request 0: { start_height: 5000, max_block_count: 20 }
request 1: { start_height: 5020, max_block_count: 20 }
...
request 9: { start_height: 5180, max_block_count: 20 }
```
2. As soon as any single request completes, the wallet immediately
parses that chunk.
- This is all in parallel. For example, as soon as request 7
responds, the wallet immediately begins parsing that chunk in
parallel to any other chunks it's already parsing.
3. If a chunk does not include a total of max_block_count blocks,
and the chunk is not the tip of the chain, this means there was a
"gap" in the chunk request. The scanner launches another parallel
RPC request to fill in the gap.
- This gap can occur because the server will **not** return a
chunk of blocks greater than 100mb (or 20k txs) via the
/getblocks.bin` RPC endpoint
([`FIND_BLOCKCHAIN_SUPPLEMENT_MAX_SIZE`](https://github.com/monero-project/monero/blob/053ba2cf07649cea8134f8a188685ab7a5365e5c/src/cryptonote_core/blockchain.cpp#L65))
- The gap is from `(req.start_height + res.blocks.size())` to
`(req.start_height + req.max_block_count)`.
4. As soon as the scanner finishes parsing the chunk, it
immediately submits another parallel RPC request.
5. In parallel, the scanner identifies a user's received (and
spent) enotes in each chunk.
- For example, as soon as request 7 responds and the wallet
parses it, the wallet scans that chunk in parallel to any other
chunks it's already scanning.
6. Once a single chunk is fully scanned locally, the scanner
launches a parallel task to fetch and scan the next chunk.
7. Once the scanner reaches the tip of the chain (the terminal
chunk), the scanner terminates.
*Some technical highlights*
- The wallet scanner is backwards compatible with existing daemons
(though it would need to point to an updated daemon to realize the
perf speed-up).
- On error cases such as the daemon going offline, the same wallet
errors that wallet2 uses (that the wallet API expects) are
propagated up to the higher-level Seraphis lib.
- The implementation uses an http client connection pool (reusing
the epee http client) to support parallel network requests
([related](seraphis-migration/wallet3#58)).
- A developer using the scanner can "bring their own blocks/network
implementation" to the scanner by providing a callback function of
the following type as a param to the async scanner constructor:
`std::function<bool(const cryptonote::COMMAND_RPC_GET_BLOCKS_FAST::request, cryptonote::COMMAND_RPC_GET_BLOCKS_FAST::response)>`
---------
Co-authored-by: jeffro256 <jeffro256@tutanota.com>
UkoeHB
pushed a commit
to UkoeHB/monero
that referenced
this issue
Aug 2, 2024
Implement async wallet scanner.
Adds a new functional test for direct wallet2 -> live RPC daemon
interactions. This sets up a framework to test pointing the
Seraphis wallet lib to a live daemon.
Tests sending and scanning:
- a normal transfer
- a sweep single (0 change)
- to a single subaddress
- to 3 subaddresses (i.e. scanning using additional pub keys)
* scan machine: option to force reorg avoidance increment first pass
- when pointing to a daemon that does not support returning empty
blocks when the client requests too high of a height, we have to
be careful in our scanner to always request blocks below the chain
tip, in every request.
- by forcing the reorg avoidance increment on first pass, we make
sure clients will always include the reorg avoidance increment
when requesting blocks from the daemon, so the client can expect
the request for blocks should *always* return an ok height.
* core tests: check conversion tool on all legacy enote version types
Stil TODO:
- check complete scanning on all enote types
- hit every branch condition for all enote versions
* conn pool mock: epee http client connection pool
- Enables concurrent network requests using the epee http client.
- Still TODO for production:
1) close_connections
2) require the pool respect max_connections
* enote finding context: IN LegacyUnscannedChunk, OUT ChunkData
- finds owned enotes by legacy view scanning a chunk of blocks
* async: function to remove minimum element from token queue
- Useful when we want to remove elements of the token queue in an
order that is different than insertion order.
* async scanner: scan via RPC, fetching & scanning parallel chunks
*How it works*
Assume the user's wallet must start scanning blocks from height
5000.
1. The scanner begins by launching 10 RPC requests in parallel to
fetch chunks of blocks as follows:
```
request 0: { start_height: 5000, max_block_count: 20 }
request 1: { start_height: 5020, max_block_count: 20 }
...
request 9: { start_height: 5180, max_block_count: 20 }
```
2. As soon as any single request completes, the wallet immediately
parses that chunk.
- This is all in parallel. For example, as soon as request 7
responds, the wallet immediately begins parsing that chunk in
parallel to any other chunks it's already parsing.
3. If a chunk does not include a total of max_block_count blocks,
and the chunk is not the tip of the chain, this means there was a
"gap" in the chunk request. The scanner launches another parallel
RPC request to fill in the gap.
- This gap can occur because the server will **not** return a
chunk of blocks greater than 100mb (or 20k txs) via the
/getblocks.bin` RPC endpoint
([`FIND_BLOCKCHAIN_SUPPLEMENT_MAX_SIZE`](https://github.com/monero-project/monero/blob/053ba2cf07649cea8134f8a188685ab7a5365e5c/src/cryptonote_core/blockchain.cpp#L65))
- The gap is from `(req.start_height + res.blocks.size())` to
`(req.start_height + req.max_block_count)`.
4. As soon as the scanner finishes parsing the chunk, it
immediately submits another parallel RPC request.
5. In parallel, the scanner identifies a user's received (and
spent) enotes in each chunk.
- For example, as soon as request 7 responds and the wallet
parses it, the wallet scans that chunk in parallel to any other
chunks it's already scanning.
6. Once a single chunk is fully scanned locally, the scanner
launches a parallel task to fetch and scan the next chunk.
7. Once the scanner reaches the tip of the chain (the terminal
chunk), the scanner terminates.
*Some technical highlights*
- The wallet scanner is backwards compatible with existing daemons
(though it would need to point to an updated daemon to realize the
perf speed-up).
- On error cases such as the daemon going offline, the same wallet
errors that wallet2 uses (that the wallet API expects) are
propagated up to the higher-level Seraphis lib.
- The implementation uses an http client connection pool (reusing
the epee http client) to support parallel network requests
([related](seraphis-migration/wallet3#58)).
- A developer using the scanner can "bring their own blocks/network
implementation" to the scanner by providing a callback function of
the following type as a param to the async scanner constructor:
`std::function<bool(const cryptonote::COMMAND_RPC_GET_BLOCKS_FAST::request, cryptonote::COMMAND_RPC_GET_BLOCKS_FAST::response)>`
---------
Co-authored-by: jeffro256 <jeffro256@tutanota.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Context
If the wallet was able to make concurrent network requests to a daemon, then it could make requests for chunks of blocks in parallel and immediately start scanning whichever chunks get to the client first. This is how @UkoeHB's async scanner was designed. When implemented, I observed a download speed-up of ~40% on my machine pointing to a remote daemon, which translated to a ~40% speed-up in scanning compared to wallet2, since download speed is my bottleneck (source).
The epee http client wallet2 uses does not support concurrent network requests over a single connection. In order for a client to make concurrent network requests to a daemon with epee's http client, it must initiate multiple connections to the daemon, which has perf overhead. This is how I implemented the mock async scanner mentioned above (source).
libcurl's http client offers the ability to make concurrent network requests over a single connection, however, this is only supported when the server supports http/2 multiplexing or http/3. Both of which seem to require significant changes to the Monero server connection code to support.
Immediate path forward
I think the optimal immediate path forward is to move forward with a hardened client-side connection pool that wraps N http connections to a daemon and is particularly useful for scanning. I already implemented a mock connection pool in my scanner impl linked above, and I think that can easily be improved and hardened.
This shouldn't require any changes to the daemon and should be easily doable in days since it's pretty much already implemented.
Future considerations
monerodrpc server to support http/3 in order to support concurrent request/response handling (plus other perf benefits including faster connection time), and re-consider using libcurl in the wallet.Both of these would be nice to have but I think networking dev resources are probably better allocated elsewhere at the moment (e.g. p2p encryption), since we can realize a major scanning speed-up without major server-side code that would require a fair amount of dev effort and review.
The text was updated successfully, but these errors were encountered: