diff --git a/README.mediawiki b/README.mediawiki index 815e8c3c67..bc174cf350 100644 --- a/README.mediawiki +++ b/README.mediawiki @@ -1030,6 +1030,13 @@ Those proposing changes should consider that ultimately consent may rest with th | Standard | Draft |- +| [[bip-0331.mediawiki|331]] +| Peer Services +| Ancestor Package Relay +| Gloria Zhao +| Standard +| Draft +|- | [[bip-0338.mediawiki|338]] | Peer Services | Disable transaction relay message diff --git a/bip-0331.mediawiki b/bip-0331.mediawiki new file mode 100644 index 0000000000..4cbb3e478d --- /dev/null +++ b/bip-0331.mediawiki @@ -0,0 +1,412 @@ +
+  BIP: 331
+  Layer: Peer Services
+  Title: Ancestor Package Relay
+  Author: Gloria Zhao 
+  Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-0331
+  Status: Draft
+  Type: Standards Track
+  Created: 2022-08-08
+  License: BSD-3-Clause
+
+ +==Abstract== + +Peer-to-peer protocol messages enabling nodes to request and relay the unconfirmed ancestor package +of a given transaction, and to request and relay transactions in batches. + +==Motivation== + +(1) Help incentive-compatible transactions propagate. + +Since v0.13, Bitcoin Core has used ancestor packages instead of individual transactions to evaluate +the incentive compatibility of transactions in the mempool +[https://github.com/bitcoin/bitcoin/pull/7594 Add tracking of ancestor packages] and +selecting them for inclusion in blocks +[https://github.com/bitcoin/bitcoin/pull/7600 Select transactions using feerate-with-ancestors]. +Incentive-compatible mempool and miner policies help create a fair, fee-based market for block +space. While miners maximize transaction fees in order to earn higher block rewards, non-mining +users participating in transaction relay reap many benefits from employing policies that result in a +mempool with similar contents, including faster compact block relay and more accurate fee +estimation. Additionally, users may take advantage of mempool and miner policy to bump the priority +of their transactions by attaching high-fee descendants (Child Pays for Parent or CPFP). + +Only individually considering transactions for submission to the mempool creates a limitation in +the node's ability to determine which transactions have the highest feerates, since it cannot take +into account descendants until all the transactions are in the mempool. Similarly, it cannot use a +transaction's descendants when considering which of two conflicting transactions to keep (Replace by +Fee or RBF). + +When a user's transaction does not meet a mempool's minimum feerate and they cannot create a +replacement transaction directly, their transaction will simply be rejected by this mempool. They +also cannot attach a descendant to pay for replacing a conflicting transaction; it would be rejected +for spending inputs that do not exist. + +This limitation harms users' ability to fee-bump their transactions. Further, it presents a security +issue in contracting protocols which rely on presigned, time-sensitive transactions'''Examples of time-sensitive pre-signed transactions in L2 protocols.''' +* [https://github.com/lightning/bolts/blob/master/03-transactions.md#htlc-timeout-and-htlc-success-transactions HTCL-Timeout in LN Penalty] +* [https://github.com/revault/practical-revault/blob/master/transactions.md#cancel_tx Unvault Cancel in Revault] +* [https://github.com/discreetlogcontracts/dlcspecs/blob/master/Transactions.md#refund-transaction Refund Transaction in Discreet Log Contracts] +* [https://gist.github.com/instagibbs/60264606e181451e977e439a49f69fe1 Updates in Eltoo] + to prevent cheating. +In other words, a key security assumption of many contracting protocols is that all parties can +propagate and confirm transactions in a timely manner. Increasing attention has been brought to +"pinning attacks," a type of censorship in which the attacker uses mempool policy restrictions to +prevent a transaction from being relayed or getting mined. +'''Concerns for pinning attacks in L2 protocols''' +* [https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-May/020458.html Bringing a nuke to a knife fight: Transaction introspection to stop RBF pinning] +* [https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-April/002639.html RBF Pinning with Counterparties and Competing Interest] +* [https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-June/002758.html Pinning : The Good, The Bad, The Ugly] +* [https://github.com/t-bast/lightning-docs/blob/master/pinning-attacks.md T-bast's Pinning Attacks Document] +* [https://gist.github.com/instagibbs/60264606e181451e977e439a49f69fe1 Eltoo Pinning] + + +These transactions must meet a certain confirmation target to be effective, but their feerates +are negotiated well ahead of broadcast time. If the forecast feerate was too low and no +fee-bumping options are available, attackers can steal money from their counterparties. Always +overestimating fees may sidestep this issue (but only while mempool traffic is low and +predictable), but this solution is not guaranteed to work and wastes users' money. For some attacks, +the available defenses require nodes to have a bird's-eye view of Bitcoin nodes' mempools, which is +an unreasonable security requirement. + +The best solution is to enable nodes to consider packages of transactions as a unit, e.g. one or +more low-fee ancestor transactions with a high-fee descendant, instead of separately. A package-aware +mempool policy can help determine if it would actually be economically rational to accept a +transaction to the mempool if it doesn't meet fee requirements individually. Network-wide adoption +of these policies would create a more purely-feerate-based market for block space and allow +contracting protocols to adjust fees (and therefore mining priority) at broadcast time. + +Theoretically, developing a safe and incentive-compatible package mempool acceptance policy is +sufficient to solve this issue. Nodes could opportunistically accept packages (e.g. by trying +combinations of transactions rejected from their mempools), but this practice would likely be +inefficient at best and open new Denial of Service attacks at worst. As such, this proposal +suggests adding new p2p messages enabling nodes to request and share package-validation-related +information with one another, resulting in a more efficient and reliable way to propagate packages. + +(2) Eliminate the need for txid-based transaction relay and make orphan handling more robust. + +Txid-based transaction relay is problematic since a transaction's witness may be malleated without +changing its txid; a node cannot use txid to deduplicate transactions it has already downloaded +or validated. Ideally, two nodes that both support BIP339 wtxid-based transaction relay shouldn't +ever need to use txid-based transaction relay. + +A single use case of txid-based relay remains: handling "orphan" transactions that spend output(s) +from an unconfirmed transaction the receiving node is unaware of. Orphan transactions are very +common for new nodes that have just completed Initial Block Download and do not have an up-to-date +mempool. Nodes also download transactions from multiple peers. If the peer from which a child +transaction was requested responds faster than the peer from which its parent was requested, that +child is seen as an orphan transaction. + +Nodes may handle orphans by storing them in a cache and requesting any missing parent(s) by txid +(prevouts specify txid, not wtxid). These parents may end up being orphans as well, if they also +spend unconfirmed inputs that the node is unaware of. This method of handling orphans is problematic +for two reasons: it requires nodes to allocate memory for unvalidated data received on the p2p +network and it relies on txid-based relay between two wtxid-relay peers. + +This proposal makes orphan-fetching more efficient and no longer require txid-based relay. + +==Definitions== + +Given any two transactions Tx0 and Tx1 where Tx1 spends an output of Tx0, Tx0 is a '''parent''' of +Tx1 and Tx1 is a '''child''' of Tx0. + +A transaction's '''ancestors''' include, recursively, its parents, the parents of its parents, etc. +A transaction's '''descendants''' include, recursively, its children, the children of its children, +etc. A transaction's parent is its ancestor, but an ancestor is not necessarily a parent. + +A '''package''' is a list of transactions, representable by a connected Directed Acyclic +Graph (a directed edge exists between a transaction that spends the output of another transaction). +In this proposal, a package is limited to unconfirmed transactions. + +An '''ancestor package''' consists of an unconfirmed transaction with all of its unconfirmed +ancestors. + +==Specification== + +===Intended Protocol Flow=== + +This package relay protocol satisfies both use cases (orphan transaction handling and high-feerate +transaction paying for low-feerate ancestors). + +Package Relay consists of two phases: a package information round and a transaction data round. +'''Why are package information and transaction data rounds both necessary?''' + +Several alternative designs were considered. One should measure alternative solutions based on the +resources used to communicate (not necessarily trustworthy) information: We would like to minimize +network bandwidth, avoid downloading a transaction more than once, avoid downloading transactions +that are eventually rejected, and minimize storage allocated for not-yet-validated transactions. + +Consider the following scenarios (where the mempool minimum feerate is 3sat/vB and all transactions +are 1000vB in virtual size): + +* Package {A, B} where A pays 0 satoshis and B pays 8000 satoshis in fees. +* Package {C, D} where C pays 0 satoshis and D pays 1200 satoshis in fees. +* Package {E, F, G, H, J} where the transactions pay 4000, 8000, 0, 2000, and 4000 satoshis in fees, respectively. + +
+ +'''No Package Information Round:''' Instead of having a package information round, just use the +child's wtxid to refer to the package and always send the entire package together. This would make +it very likely for honest nodes to redownload duplicate transactions. +[[File:./bip-0331/no_package_info.png|900px]] +
+ +'''Package Information Only:''' Just having package information gives enough information for the +receiver to accept the packages. Omit the "getpkgtxns" and "pkgtxns" messages and just download the +transactions individually using existing messages. While this option is a good fallback if batched +transaction download fails for some reason, it shouldn't be used as the default because it always +requires storage of unvalidated transactions. +[[File:./bip-0331/package_info_only.png|1200px]] +
+ +====Orphan Transaction Handling==== + +Upon receiving an orphan transaction, a node may request ancestor package information delineating +the wtxids of the transaction's unconfirmed ancestors. This is done without using txid-based relay. +The package information can be used to request transaction data. As these transactions are dependent +upon one another to be valid, the transactions can be requested and sent as a batch. + +[[File:./bip-0331/protocol_flow.png|1200px]] + +====Fee-Bumped Transactions==== + +Too-low-feerate transactions (i.e. below the node's minimum mempool feerate) with high-feerate +descendants can also be relayed this way. If the peers are using BIP133 fee filters and a +low-feerate transaction is below the node's fee filter, the sender will not announce it. The +high-feerate transaction will be sent by the sender, and received and handled as an orphan by the +receiver, the transactions are validated as a package, and so the protocol naturally works for this +use case. + +This does not mean BIP133 is required for package relay to work, provided that nodes do not +immediately reject transactions previously found to be too low feerate. If the low-feerate +transaction was sent, the receiver would reject it but should later request and accept it after +learning that it is the ancestor of the high-feerate transaction. + +[[File:./bip-0331/package_cpfp_flow.png|800px]] + +This protocol is receiver-initiated only; nodes do not proactively announce packages to their peers. +'''Why no sender-initiated protocol?''' Sender-initiated package +relay can, theoretically, save a round trip by notifying the receiver ahead of time that they will +probably need to request and validate a group of transactions together in order for them to be +accepted. As with any proactive communication, there is a chance that the receiver already knows +this information, so this network bandwidth may be wasted. Shortened latency is less significant +than wasted bandwidth. + +The logic used to decide when to announce a package proactively determines whether it is a net +increase or decrease for overall bandwidth usage. However, it's difficult to design anything to save +bandwidth without any idea of what its bandwidth usage actually looks like in practice. We'll want +to design the sender-initiated protocol carefully, and inform the design decisions using data +collected from the mainnet p2p network. However, there is no historical transaction data to use +because the goal is to enable currently-rejected transactions to propagate. In order to get this +right, I propose we hold off on sender-initiated for now, deploy receiver-initiated package relay, +observe its usage and figure out where we can save a round trip, and then introduce a +well-researched sender-initiated package relay protocol. + +Ancestor package relay is negotiated between two peers during the version handshake using a +"sendpackages" message containing version=0. It requires both peers to support wtxid-based relay +because package transactions are referenced by their wtxid. Peers wishing to relay multiple versions +of packages should send one "sendpackages" message for each version. +'''Is it possible to support other types of packages in the future?''' Yes. Attempting to +support arbitrary packages in mempool validation may result in very complex logic, new Denial of +Service attack vectors, and policy limitations that could be leveraged to censor transactions (aka +"pinning attacks"). This protocol is extensible to support multiple types of packages based on +future desired use cases. +'''Why send one message per version instead of sending a single message for the highest +supported version?''' It should also be possible to support some subset of the existing package +types. For example, if a node's mempool policy doesn't support or a node implementation deprecates +version 2, it should be easy to announce support for version 1, but not version 2, packages. + +[[File:./bip-0331/version_negotiation.png|600px]] + +===Combined Hash=== + +A "combined hash" serves as a unique "package id" for some list of unique transactions. + +The combined hash of a package of transactions is equal to the hash of each transaction's wtxid, concatenated in lexicographical order. + +The combined hash can be calculated using just the wtxids of the transactions. + +===New Messages=== + +Four new protocol messages and two inv types are added. + +====sendpackages==== + +{| +| Field Name || Type || Size || Purpose +|- +|version || uint32_t || 4 || Denotes a package version supported by the node. +|- +|} + +# The "sendpackages" message has the structure defined above, with pchCommand == "sendpackages". + +# During version handshake, nodes should send a "sendpackages" message to indicate they support package relay. Nodes should send a "sendpackages" message for each version they support. + +# Upon receipt of a "sendpackages" message with a version that is not supported, a node must treat the peer as if it never received the message. + +# The "sendpackages" message MUST be sent before sending a "verack" message. If a "sendpackages" message is received afer "verack", the sender may be disconnected. + +# A node that sends "sendpackages" MUST also send "wtxidrelay". Upon receipt of a "verack", if the sender has sent a "sendpackages" but not "wtxidrelay", the sender may be disconnected. + +# Upon successful connection ("verack" sent by both peers), a node may relay packages with the peer if they did not set "fRelay" to false in the "version" message, both peers sent "wtxidrelay", and both peers sent "sendpackages" for matching version(s). Peers should relay packages corresponding to versions that both sent "sendpackages" for.'''Is it ok to send "sendpackages" to a peer that specified fRelay=false in their "version" message?''' +Yes, this is allowed in order to reduce the number of negotiation steps. This means nodes can +announce features without first checking what the other peer has sent, and then apply logic at the +end. See [https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-May/020510.html discussion]. + + +# Version=0 packages correspond to ancestor packages, i.e. "ancpkginfo" and "MSG_ANCPKGINFO". + +====ancpkginfo==== +{| +| Field Name || Type || Size || Purpose +|- +|txns_length||CompactSize||1 or 3 bytes|| The number of transactions provided. +|- +|txns||List of wtxids||txns_length * 32|| The wtxids of each transaction in the package. +|} + +# The "ancpkginfo" message has the structure defined above, with pchCommand == "ancpkginfo". + +# The "txns" field should contain a list of up to 26 wtxids which constitute the ancestor package of the last wtxid (the "representative transaction"). Apart from the last wtxid, the rest of the wtxids may be in any order.'''Why not include feerate information to help the receiver decide whether these transactions are worth downloading?''' +Peers are not trusted to provide correct information, so a reported low feerate should not stop a +node from downloading the package or verifying what the transactions' feerates actually are. +Additionally, a simple feerate is typically insufficient; the receiver must also know the dependency +relationships between transactions and their respective sizes. +'''Should a peer be punished if they provide incorrect package info, e.g. a list of unrelated transactions?''' +Ideally, there should be a way to enforce that peers are providing correct information to each +other. However, two peers may have different views of what a transaction's unconfirmed ancestors +are based on their chainstate. For example, during a reorg or when two blocks are found at the same +time, one peer may see a transaction as confirmed while the other peer does not. +As such, it is impossible to accurately enforce this without also knowing the peer's chainstate. +It was [https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-May/020493.html originally proposed] +to include a block hash in "ancpkginfo" to avoid unwarranted disconnections. However, it does not +make much sense to stop or delay transaction data requests due to mismatched chainstates, and the +chainstate may change again between package information and transaction data rounds. Instead, +differences in chainstate should be handled at the validation level. The node should de-duplicate +recently-confirmed transactions and make a best effort to validate the transactions it has already +downloaded. See [https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-June/020558.html discussion]. +'''Should topological order (i.e. a parent always appears somewhere in the list before +its child) be required?''' +It is not possible to determine whether a list of transactions is topologically sorted without first +establishing that the list contains a full ancestor package. + + +# Upon receipt of a "ancpkginfo" message, the node may use it to request the transactions not already in its mempool. + +# Upon receipt of a malformed "ancpkginfo" message, the sender may be disconnected. An "ancpkginfo" +# message is malformed if it contains more than 26 transactions, duplicate wtxids, or specifies transactions spending the same inputs (which the receiver would learn after downloading the transactions). + +# A node MUST NOT send a "ancpkginfo" message that has not been requested by the recipient. Upon receipt of an unsolicited "ancpkginfo", a node may disconnect the sender. + +====MSG_ANCPKGINFO==== + +# A new inv type (MSG_ANCPKGINFO == 0x6) is added, for use only in getdata requests pertaining to ancestor packages. + +# As a getdata request type, it indicates that the sender wants a "ancpkginfo" containing all of the unconfirmed ancestors of a transaction, referenced by wtxid. + +# Upon receipt of a "getdata(MSG_ANCPKGINFO)" request, the node should respond with an "ancpkginfo" message corresponding to the transaction's unconfirmed ancestor package, or with NOTFOUND. The wtxid of the requested transaction should be the last item in the list. + +# The inv type should not be used in announcements, i.e. "inv(MSG_ANCPKGINFO)" must never be sent. If a "inv(MSG_ANCPKGINFO)" is received, the sender may be disconnected. + +# This inv type should only be used if both peers agreed to send version 0 packages. If a "getdata" message with type MSG_ANCPKGINFO is received from a peer with which ancestor package relay was not negotiated, no response should be sent and the sender may be disconnected. + +====getpkgtxns==== + +{| +| Field Name || Type || Size || Purpose +|- +|txns_length||CompactSize||1 or 3 bytes|| The number of transactions requested. +|- +|txns||List of wtxids||txns_length * 32|| The wtxids of each transaction in the package. +|} + +# The "getpkgtxns" message has the structure defined above, with pchCommand == "getpkgtxns". + +# A "getpkgtxns" message should be used to request all or some of the transactions previously announced in a "ancpkginfo" message, specified by witness transaction id. This message is intended to allow nodes to avoid downloading and storing transactions that cannot be validated immediately. + +# Upon receipt of a "getpkgtxns" message, a node should respond with either a "pkgtxns" containing the requested transactions or "notfound" messages indicating one or more of the transactions is unavailable.'''Why isn't package relay negotiation required for getpkgtxns?''' +It was [https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-May/020496.html suggested] that +this message is not package-specific and generally useful for requesting transactions as +batches. + +====pkgtxns==== + +{| +| Field Name || Type || Size || Purpose +|- +|txns_length||CompactSize||1 or 3 bytes|| The number of transactions provided. +|- +|txns||List of transactions||variable|| The transactions in the package. +|} + +# The "pkgtxns" message has the structure defined above, with pchCommand == "pkgtxns". + +# A "pkgtxns" message should contain the transaction data requested using "getpkgtxns". + +# Upon receipt of a "pkgtxns" message, the node should validate the transactions together as a package rather than individually. + +# A "pkgtxns" message should only be sent to a peer that requested the package using "getpkgtxns". If a node receives an unsolicited package, the sender may be disconnected.'''Why isn't package relay negotiation required for pkgtxns?''' +It was [https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-May/020496.html suggested] that +this message is not package-specific and generally useful for downloading transactions as +batches. + +====MSG_PKGTXNS==== + +# A new inv type (MSG_PKGTXNS == 0x7) is added, for use only in "notfound" messages pertaining to package transactions. + +# As a notfound type, it indicates that the sender is unable to send all the transactions requested in a prior "getpkgtxns" message. The hash used is equal to the combined sha256 hash of the wtxids in the getpkgtxns request. + +# This inv type should only be used in "notfound" messages, i.e. "inv(MSG_PKGTXNS)" and "getdata(MSG_PKGTXNS)" should never be sent. + + +==Compatibility== + +Older clients remain fully compatible and interoperable after this change. Clients implementing this +protocol will only attempt to send and request packages if agreed upon during the version handshake. +'''Will package relay cause non-package relay nodes to waste bandwidth on low-feerate transactions?''' +If a node supports package relay, it may accept low-feerate transactions (e.g. paying zero fees) +into its mempool, but non-package relay nodes would most likely reject them. To mitigate bandwidth +waste, a package relay node should not announce descendants of below-fee-filter transactions to +non-package relay peers. + +'''Is Package Erlay possible?''' +A client using BIP330 reconciliation-based transaction relay (Erlay) is able to use package relay +without interference. After reconciliation, any transactions with unconfirmed ancestors may be +relayed using ancestor package relay. +[[File:./bip-0331/package_erlay.png|800px]] + + +==Implementation== + +Sample implementation for Bitcoin Core: +https://github.com/glozow/bitcoin/tree/2023-01-package-relay. A prerequisite for implementing a safe +package relay protocol is a mempool acceptance policy that safely validates packages of +transactions. +'''Package Mempool Acceptance Policy''' +Accepting packages from peers should not significantly increase a node's DoS attack surface. +Additionally, the mempool acceptance policy should result in the most incentive-compatible subset of +the package be accepted into the mempool. Otherwise, more pinning attacks or censorship vectors may +be possible. For example, It should not be assumed that packages are CPFPs. An ancestor package may +include a high-feerate parent and low-feerate child; the policy may choose to accept the parent but +not the child. If one or more transactions are policy-invalid, other transactions that are not +dependent upon them should still be considered. +[https://github.com/bitcoin/bitcoin/pull/22290 Sample implementation: #22290] + + +==Acknowledgements== + +Thank you to Suhas Daftuar, John Newbery, Anthony Towns, and Martin Zumsande for input on the design. + +Much of this work is inspired by ideas and code by Suhas Daftuar and Antoine Riard. +'''Prior Work on Package Relay''' +* [https://gist.github.com/sdaftuar/8756699bfcad4d3806ba9f3396d4e66a Strawman Proposal] +* [https://github.com/bitcoin/bitcoin/issues/14895 Package relay design questions] +* [https://github.com/bitcoin/bitcoin/pull/16401 Add package acceptance logic to mempool] +* [https://github.com/bitcoin/bitcoin/pull/19621 [RFC] Package-relay: sender-initiated] + + +==References and Rationale== + + + diff --git a/bip-0331/no_package_info.png b/bip-0331/no_package_info.png new file mode 100644 index 0000000000..54b20f97dd Binary files /dev/null and b/bip-0331/no_package_info.png differ diff --git a/bip-0331/package_cpfp_flow.png b/bip-0331/package_cpfp_flow.png new file mode 100644 index 0000000000..6c9462c540 Binary files /dev/null and b/bip-0331/package_cpfp_flow.png differ diff --git a/bip-0331/package_erlay.png b/bip-0331/package_erlay.png new file mode 100644 index 0000000000..fd3661ff5d Binary files /dev/null and b/bip-0331/package_erlay.png differ diff --git a/bip-0331/package_info_only.png b/bip-0331/package_info_only.png new file mode 100644 index 0000000000..2bd0272606 Binary files /dev/null and b/bip-0331/package_info_only.png differ diff --git a/bip-0331/protocol_flow.png b/bip-0331/protocol_flow.png new file mode 100644 index 0000000000..9683e92cfe Binary files /dev/null and b/bip-0331/protocol_flow.png differ diff --git a/bip-0331/receiver_init_dialogue.png b/bip-0331/receiver_init_dialogue.png new file mode 100644 index 0000000000..aa287226e8 Binary files /dev/null and b/bip-0331/receiver_init_dialogue.png differ diff --git a/bip-0331/version_negotiation.png b/bip-0331/version_negotiation.png new file mode 100644 index 0000000000..00d7536d52 Binary files /dev/null and b/bip-0331/version_negotiation.png differ