CHIP-0026: New Wallet Sync Protocol

CHIPs/chip-0026.md

@@ -0,0 +1,248 @@
+| CHIP Number   | 0026                                                                     |
+| :------------ | :----------------------------------------------------------------------- |
+| Title         | New Wallet Sync Protocol                                                 |
+| Description   | Wallet protocol messages for syncing coins and transactions from a node. |
+| Author        | [Brandon Haggstrom](https://github.com/Rigidity)                         |
+| Editor        | [Dan Perry](https://github.com/danieljperry)                             |
+| Comments-URI  | [CHIPs repo, PR #100](https://github.com/Chia-Network/chips/pull/100)    |
+| Status        | Final                                                                    |
+| Category      | Standards Track                                                          |
+| Sub-Category  | Network                                                                  |
+| Created       | 2024-03-05                                                               |
+| Requires      | None                                                                     |
+| Replaces      | None                                                                     |
+| Superseded-By | None                                                                     |
+
+## Abstract
+
+This CHIP proposes a new set of protocol messages for syncing a light wallet against a full node. It solves many pain points with the current protocol, prevents DoS (Denial of Service) issues for certain use cases, and enables certain optimizations while syncing. In addition, this protocol will enable wallets to subscribe to transactions in the mempool.
+
+## Motivation
+
+Currently the wallet protocol enables you to register for updates for a set of coin ids or puzzle hashes. The initial coin states will be included in the response, and further coin state updates will be sent to you as new blocks are farmed or whenever a reorg occurs. This does a good job at keeping the wallet informed about the current state of its coins, but it has a few shortcomings. This protocol is an attempt to address these issues and make light wallets more stable and efficient.
+
+First of all, if you subscribe to a set of puzzle hashes and there are more coins than would fit in a response, it will be truncated. This makes it impossible to download the full set of coins if it exceeds a certain size. This protocol solves this by paginating the response, sending coin data grouped by block height in each batch. This way you can sync up to the peak eventually, regardless of how many coins you are downloading.
+
+There is also currently no way to request the current state of coins without subscribing to them. Additionally, subscriptions cannot be removed later either, meaning you will eventually hit the subscription limit through normal use, even if you only need the coin information one time (for example when constructing a CAT spend). The new protocol allows you to do both of these things, preventing unnecessary subscriptions where possible.
+
+The new wallet sync protocol also forces you to handle reorgs, by rejecting the request if the claimed header hash does not match the height provided. Reorgs can occur both while you are syncing a set of puzzle hashes (if there is a lot of data to be synced), and after you reconnect your wallet to the network (if a reorg happened to occur right after you went offline). If a wallet does not handle reorgs properly, it can result in incorrect coin state data being stored in its database.
+
+You will also be able to opt in to receiving updates for relevant transactions as they enter or leave the mempool.
+
+## Backwards Compatibility
+
+The changes to the wallet protocol are fully backwards compatible.
+
+## Rationale
+
+Another way to accomplish the goals for syncing outlined above would be to stream coin states from the node to the wallet until you have reached the peak. This way you can receive a consistent snapshot of the blockchain database and apply updates as needed from there. However, this forces the node to spend an arbitrarily long amount of time sending data to the wallet until complete, which is a major concern for both performance and opening up the potential for DoS attacks.
+
+The reasoning for doing it this way, by requesting coin state in batches, is that you can rate limit the requests consistently, preventing wallets from easily overloading the node with expensive operations. The wallet can handle reorgs on the fly by backtracking (or finding a common fork point), and can ask the node to automatically subscribe it to future updates once the initial coin state is synced. This solves both the performance concern and the requirement to have consistent data.
+
+Instead of syncing from a starting height, the new protocol uses a previous height and header hash to sync off of. The reasoning here is that you can use the last known peak when reconnecting to the network to start from (assuming no reorg has occurred). As well as this, and to prevent reorg issues, the header hash is now required. If the wallet does not know the header hash, it can sync from genesis instead.
+
+The protocol update PR has been shared with the community a couple times, and though it hasn't reached community consensus, feedback thus far has seemed positive. This CHIP is to facilitate feedback on both the protocol and its implementation, and to ensure that it solves various developer and user concerns with the current light wallet protocol.
+
+## Specification
+
+This CHIP's design allows wallets to opt in to receive mempool updates via a new capability.
+
+Whenever a new subscription to puzzle hashes (via `RequestPuzzleState`) or coin IDs (via `RequestCoinState`) is added, the wallet will receive a list of every transaction ID that relates to those subscriptions in a `MempoolItemsAdded` message.
+
+Whenever transactions are added or removed from the mempool, the protocol will send the transaction ID to every peer that has subscribed to anything inside of it (spent or created coin IDs, puzzle hashes, or hints).
+
+The full node then gives coin state updates, as well as mempool transaction updates.
+
+### Message Types
+
+The messages types have the following values:
+
+```py
+mempool_items_added = 104
+mempool_items_removed = 105
+request_cost_info = 106
+respond_cost_info = 107
+```
+
+### Remove Puzzle Subscriptions
+
+```py
+class RequestRemovePuzzleSubscriptions:
+    puzzle_hashes: Optional[List[bytes32]]
+
+class RespondRemovePuzzleSubscriptions:
+    puzzle_hashes: List[bytes32]
+```
+
+Removes puzzle hashes from the subscription list (or all of them if `None`), returning the hashes that were actually removed.
+
+### Remove Coin Subscriptions
+
+```py
+class RequestRemoveCoinSubscriptions:
+    coin_ids: Optional[List[bytes32]]
+
+class RespondRemoveCoinSubscriptions:
+    coin_ids: List[bytes32]
+```
+
+Removes coin ids from the subscription list (or all of them if `None`), returning the ids that were actually removed.
+
+### Request Puzzle State
+
+```py
+class RequestPuzzleState:
+    puzzle_hashes: List[bytes32]
+    previous_height: Optional[uint32]
+    header_hash: bytes32
+    filters: CoinStateFilters
+    subscribe_when_finished: bool
+
+class RespondPuzzleState:
+    puzzle_hashes: List[bytes32]
+    height: uint32
+    header_hash: bytes32
+    is_finished: bool
+    coin_states: List[CoinState]
+
+class RejectPuzzleState:
+    reason: uint8  # RejectStateReason
+
+class CoinStateFilters:
+    include_spent: bool
+    include_unspent: bool
+    include_hinted: bool
+    min_amount: uint64
+
+class RejectStateReason(IntEnum):
+    REORG = 0
+    EXCEEDED_SUBSCRIPTION_LIMIT = 1
+```
+
+Requests coin states that match the given puzzle hashes (or hints).
+
+Unlike `RegisterForPhUpdates`, this does not add subscriptions for the puzzle hashes automatically.
+When `subscribe_when_finished` is set to `True`, it will add subscriptions, but only once the last batch has been requested.
+
+As well as this, previously it was impossible to get all coin records if the number of items exceeded the limit.
+This implementation allows you to continue where you left off with `previous_height` and `header_hash`.
+
+If a reorg of relevant blocks occurs while doing so, `previous_height` will no longer match `header_hash`.
+This can be handled by a wallet by simply backtracking a bit, or restarting the sync from genesis. It could be inconvenient, but at least you can detect it.
+In the event that a reorg is detected by a node, `RejectPuzzleState` will be returned. This is the only scenario it will be rejected directly like this.
+
+Additionally, it is now possible to filter out spent, unspent, or hinted coins, as well as coins below a minimum amount.
+This can reduce the risk of spamming or DoS of a wallet in some cases, and improve performance.
+
+If `previous_height` is `None`, you are syncing from genesis. The `header_hash` should match the genesis challenge of the network you are connected to.
+
+### Request Coin State
+
+```py
+class RequestCoinState:
+    coin_ids: List[bytes32]
+    previous_height: Optional[uint32]
+    header_hash: bytes32
+    subscribe: bool
+
+class RespondCoinState:
+    coin_ids: List[bytes32]
+    coin_states: List[CoinState]
+
+class RejectCoinState:
+    reason: uint8  # RejectStateReason
+
+class RejectStateReason(IntEnum):
+    REORG = 0
+    EXCEEDED_SUBSCRIPTION_LIMIT = 1
+```
+
+Request coin states that match the given coin ids.
+
+Unlike `RegisterForCoinUpdates`, this does not add subscriptions for the coin ids automatically.
+When `subscribe` is set to `True`, it will add and return as many coin ids to the subscriptions list as possible.
+
+Unlike the new `RequestPuzzleState` message, this does not implement batching for simplicity. The order is also not guaranteed.
+However, you can still specify the `previous_height` and `header_hash` to start from.
+
+If a reorg of relevant blocks has occurred, `previous_height` will no longer match `header_hash`.
+This can be handled by a wallet depending on the use case (for example by restarting from zero). It could be inconvenient, but at least you can detect it.
+In the event that a reorg is detected by a node, `RejectCoinState` will be returned. This is the only scenario it will be rejected directly like this.
+
+If `previous_height` is `None`, you are syncing from genesis. The `header_hash` should match the genesis challenge of the network you are connected to.
+
+### Mempool Updates Capability
+
+A new `MEMPOOL_UPDATES` capability, with a value of `5`. This opts in to receiving the below mempool update messages.
+
+### Mempool Items Added
+
+```py
+class MempoolItemsAdded:
+    transaction_ids: List[bytes32]
+```
+
+This message will only be sent if `MEMPOOL_UPDATES` is supported by both the wallet and the node. It is sent whenever a new valid transaction enters the mempool, as long as it matches one or more of your subscriptions.
+
+In addition, when you first subscribe to one or more puzzle hashes or coin ids using the new subscription messages in described earlier, you will receive an initial `MempoolItemsAdded` message containing a list of transaction ids of existing mempool items that match those subscriptions. It is possible to receive transactions which you have already received previously, so it's up to the wallet to filter them out.
+
+### Mempool Items Removed
+
+```py
+class MempoolRemoveReason(Enum):
+    CONFLICT = 1
+    BLOCK_INCLUSION = 2
+    POOL_FULL = 3
+    EXPIRED = 4
+
+class RemovedMempoolItem:
+    transaction_id: bytes32
+    reason: uint8 # MempoolRemoveReason
+
+class MempoolItemsRemoved:
+    removed_items: List[RemovedMempoolItem]
+```
+
+This message will only be sent if `MEMPOOL_UPDATES` is supported by both the wallet and the node. It is sent whenever a transaction leaves the mempool for any of the above reasons, as long as it matches one or more of your subscriptions.
+
+### Request Cost Info
+
+```py
+class RequestCostInfo:
+    pass
+
+class RespondCostInfo:
+    max_transaction_cost: uint64
+    max_block_cost: uint64
+    max_mempool_cost: uint64
+    mempool_cost: uint64
+    mempool_fee: uint64
+    bump_fee_per_cost: uint8
+```
+
+This gives various information about the costs of transactions, blocks, and the mempool. It can be useful for ensuring a transaction is valid prior to submitting it, as well as estimating fees.
+
+## Test Cases
+
+There are various test cases for syncing and subscribing to coin state updates in [test_new_wallet_protocol.py](https://github.com/Chia-Network/chia-blockchain/blob/8cbde05d05d9c447300c9faac21cade9acc6d7db/tests/wallet/test_new_wallet_protocol.py).
+
+## Reference Implementation
+
+The new wallet sync and subscription protocol is implemented in the following Pull Requests from the `chia-blockchain` GitHub repository:
+
+- [#17340](https://github.com/Chia-Network/chia-blockchain/pull/17340) -- new subscription and wallet sync protocol
+- [#17980](https://github.com/Chia-Network/chia-blockchain/pull/17980) -- mempool updates
+- [#18052](https://github.com/Chia-Network/chia-blockchain/pull/18052) -- performance improvement
+- [#18096](https://github.com/Chia-Network/chia-blockchain/pull/18096) -- split capabilities for each service
+
+## Security
+
+This wallet protocol update is not an attempt to improve, nor should it have any effect on, validation against untrusted full nodes. You should always keep multiple peers connected so that you can detect whether or not a peer is giving you bad data, and prevent omission.
+
+## Additional Assets
+
+None
+
+## Copyright
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).