[Mempool] Fix mempool limiting and replace-by-fee for PrioritiseTransaction #7062

sdaftuar · 2015-11-19T17:36:34Z

This fixes the behavior of the mempool limiting to respect any fee deltas applied via PrioritiseTransaction.

~~(Note that the first 5 commits here are introduced by @morcos in #6898.)~~

It also includes a couple other fixes relating to PrioritiseTransaction:

Updates RBF logic
Updates mempool acceptance logic

sdaftuar · 2015-11-19T20:11:08Z

This will create merge conflicts for #6871 -- let's try to get that reviewed and merged first, and I can resolve the merge conflicts in this PR afterward.

petertodd · 2015-11-20T20:35:35Z

Note that this leaks information about what transactions you've prioritised to the outside world even more so than before. Probably unavoidable right now, but in the long run it'd be better if we had a separate mempool for mining that kept priority deltas. Transactions themselves can be shared across both mempools and reference counted.

petertodd · 2015-11-20T20:36:45Z

src/rpcblockchain.cpp

            "    \"time\" : n,             (numeric) local time transaction entered pool in seconds since 1 Jan 1970 GMT\n"
            "    \"height\" : n,           (numeric) block height when transaction entered pool\n"
            "    \"startingpriority\" : n, (numeric) priority when transaction entered pool\n"
            "    \"currentpriority\" : n,  (numeric) transaction priority now\n"
            "    \"descendantcount\" : n,  (numeric) number of in-mempool descendant transactions (including this one)\n"
            "    \"descendantsize\" : n,   (numeric) size of in-mempool descendants (including this one)\n"
-            "    \"descendantfees\" : n,   (numeric) fees of in-mempool descendants (including this one)\n"
+            "    \"descendantfees\" : n,   (numeric) modified fees of in-mempool descendants (including this one)\n"


The term "modified fees" won't mean anything to the reader. How about "fees of in-mempool descendants (including this one) with priority deltas applied"

I was trying to make a reference to the term "modifedfee" which I added above without having to redefine the concept, what if I just do: "modified fees (see above) of in-mempool descendants (including this one)"

petertodd · 2015-11-20T20:43:48Z

Otherwise, utACK, though will want to recheck rebased version once #6871 is merged of course.

sdaftuar · 2015-11-30T21:55:23Z

Rebased.

@petertodd This pull has now been updated to use fee deltas for the replace-by-fee calculation, and the replace-by-fee rpc test has been updated to exercise that logic. (I cherry-picked #7137, so if that gets merged first I'll rebase this one, or I can close that pull if that's easier.)

petertodd · 2015-12-01T01:50:56Z

qa/rpc-tests/replace-by-fee.py

+                # the transactions we needed mined were mined, and
+                # don't worry about what is left.
+                break
+            mempool_size = new_size


FYI, I've found bugs before by erroring out if any txs are stuck; not sure this is a good idea.

Fair enough -- I ran into an issue where the really large transaction created in one of the tests was too big to mined, but I should be able to rework that test and make this check error out instead.

sdaftuar · 2015-12-01T17:57:19Z

Rebased and changed rpc test to error out if the mempool ends up with stuck transactions.

sdaftuar · 2015-12-01T19:34:30Z

Rebased now that #6898 is merged

sdaftuar · 2015-12-02T16:28:37Z

Updated so that now mempool acceptance is also determined by including a transaction's fee delta from PrioritiseTransaction.

In doing so, I realized that GetMinRelayFee is not very useful, so I scrapped it. With the DEFAULT_BLOCK_PRIORITY_SIZE set to 0, this function was stopping all free transactions from making it in (unless they were prioritised using PrioritiseTransaction). Scrapping it means that there is a behavior change from before, where now we will allow in transactions between 50kb and 100kb that are free as long as they have sufficient priority. Given that when the mempool is full we don't allow free transactions at all, and given that we expire things after 3 days, I think this should be okay.

EDIT: Actually, there's a bug in GetMinRelayFee so that the comparison with DEFAULT_BLOCK_PRIORITY_SIZE was incorrect; the comparison is comparing two things as unsigned int's, but the right hand side is negative... So this test isn't preventing anything from getting into the mempool now that the DEFAULT_BLOCK_PRIORITY_SIZE is 0.

if (nBytes < (DEFAULT_BLOCK_PRIORITY_SIZE - 1000))      
        nMinFee = 0;

morcos · 2015-12-02T16:50:39Z

utACK (modulo comment above)

sdaftuar · 2015-12-02T17:59:06Z

Addressed @morcos' nit (and squashed into first commit).

Redo the feerate index to be based on mining score, rather than fee. Update mempool_packages.py to test prioritisetransaction's effect on package scores.

One test in AcceptToMemoryPool was to compare a transaction's fee agains the value returned by GetMinRelayFee. This value was zero for all small transactions. For larger transactions (between DEFAULT_BLOCK_PRIORITY_SIZE and MAX_STANDARD_TX_SIZE), this function was preventing low fee transactions from ever being accepted. With this function removed, we will now allow transactions in that range with fees (including modifications via PrioritiseTransaction) below the minRelayTxFee, provided that they have sufficient priority.

laanwj · 2015-12-21T16:13:00Z

utACK

901b01d Remove GetMinRelayFee (Suhas Daftuar) 27fae34 Use fee deltas for determining mempool acceptance (Suhas Daftuar) 9ef2a25 Update replace-by-fee logic to use fee deltas (Suhas Daftuar) eb30666 Fix mempool limiting for PrioritiseTransaction (Suhas Daftuar)

…action 1) Fix mempool limiting for PrioritiseTransaction Redo the feerate index to be based on mining score, rather than fee. Update mempool_packages.py to test prioritisetransaction's effect on package scores. 2) Update replace-by-fee logic to use fee deltas 3) Use fee deltas for determining mempool acceptance 4) Remove GetMinRelayFee One test in AcceptToMemoryPool was to compare a transaction's fee agains the value returned by GetMinRelayFee. This value was zero for all small transactions. For larger transactions (between DEFAULT_BLOCK_PRIORITY_SIZE and MAX_STANDARD_TX_SIZE), this function was preventing low fee transactions from ever being accepted. With this function removed, we will now allow transactions in that range with fees (including modifications via PrioritiseTransaction) below the minRelayTxFee, provided that they have sufficient priority. Github-Pull: #7062 Rebased-From: eb30666 9ef2a25 27fae34 901b01d

9b9c616 Fix missing zapwallettxes mode in wallet_hd.py functional test (furszy) d6d0ad9 [logs] fix zapwallettxes startup logs (John Newbery) 006c503 [wallet] fix zapwallettxes interaction with persistent mempool (John Newbery) c6d45c6 Adapting and connecting mempool_persist.py functional test to the test runner. (furszy) 4f26a4e Control mempool persistence using a command line parameter. (John Newbery) 5d949de [Qt] Do proper shutdown (Jonas Schnelli) e60da98 Allow shutdown during LoadMempool, dump only when necessary (Jonas Schnelli) c0a0e81 Moving TxMempoolInfo tx member to CTransactionRef (furszy) f0c2255 Add mempool.dat to doc/files.md (furszy) 8e52226 Add DumpMempool and LoadMempool (Pieter Wuille) 44c635d Add AcceptToMemoryPoolWithTime function (Pieter Wuille) 6bbc6a9 Add feedelta to TxMempoolInfo (Pieter Wuille) 9979f3d [mempool] move removed and conflicts transaction ref list to vector. (furszy) 4f672c2 Make removed and conflicted arguments optional to remove (Pieter Wuille) e51c4b8 Bypass removeRecursive in removeForReorg (Pieter Wuille) 54cf7c0 Get rid of CTxMempool::lookup() entirely (furszy) 35bc2a9 Finished the switch CTransaction storage in mempool to CTransactionRef and introduced the timeLastMempoolReq coming from btc#8080. (furszy) d10583b An adapted version of btc@b5599147533103efea896a1fc4ff51f2d3ad5808 (furszy) cb4fc6c An adapted version of btc@ed7068302c7490e8061cb3a558a0f83a465beeea (furszy) 9645775 Split up and optimize transaction and block inv queues (furszy) 68bc68f Don't do mempool lookups for "mempool" command without a filter (furszy) 7624823 mapNextTx: use pointer as key, simplify value (furszy) 191c62e Return mempool queries in dependency order (Pieter Wuille) 23c9f3e Eliminate TX trickle bypass, sort TX invs for privacy and priority. (furszy) 6ebfd17 tiny test fix for mempool_tests (Alex Morcos) 8c0016e Check all ancestor state in CTxMemPool::check() (furszy) 91c6096 Add ancestor feerate index to mempool (Suhas Daftuar) 64e84e2 Add ancestor tracking to mempool This implements caching of ancestor state to each mempool entry, similar to descendant tracking, but also including caching sigops-with-ancestors (as that metric will be helpful to future code that implements better transaction selection in CreatenewBlock). (furszy) 8325bb4 Fix mempool limiting for PrioritiseTransaction Redo the feerate index to be based on mining score, rather than fee. (furszy) 1fa40ac Remove work limit in UpdateForDescendants() The work limit served to prevent the descendant walking algorithm from doing too much work by marking the parent transaction as dirty. However to implement ancestor tracking, it's not possible to similarly mark those descendant transactions as dirty without having to calculate them to begin with. This commit removes the work limit altogether. With appropriate chain limits (-limitdescendantcount) the concern about doing too much work inside this function should be mitigated. (furszy) ba32375 Rename CTxMemPool::remove -> removeRecursive remove is no longer called non-recursively, so simplify the logic and eliminate an unnecessary parameter (furszy) c30fa16 CTxMemPool::removeForBlock now uses RemoveStaged (furszy) Pull request description: Ending up 2020 with a large PR :). Included a good number of performance and privacy improvements over the mempool and inv processing/sending areas + added mempool cache dump/load. Almost finishing with #1726 work, getting closer to 9725, and getting closer to be able to decouple the message processing thread <-> wallet and the wallet <-> GUI locks dependencies (which will be another long story.. but well, step by step). The final goal is clear, a much faster syncing process using pivx-qt (a good speed up for pivxd as well), smoother visual navigation when big wallets are syncing, less thread synchronization issues, among all of the improvements that will be coming with the backports adaptations. Adapted the following PRs to our sources: bitcoin#7062 —> only eb30666. bitcoin#7174 —> complete. bitcoin#7562 —> only c5d746a. bitcoin#7594 —> complete. bitcoin#7840 —> complete. bitcoin#7997 —> complete. bitcoin#8080 —> complete bitcoin#8126 —> except e9b4780 (we don't have the mapRelay) and c2a4724 (we don't have the relay expiration vector). bitcoin#8448 —> complete bitcoin#8515 —> complete bitcoin#9408 —> complete bitcoin#9966 —> complete bitcoin#10330 —> complete ACKs for top commit: random-zebra: ACK 9b9c616 Fuzzbawls: ACK 9b9c616 Tree-SHA512: 186bd09bbb19b55ec0d46dc27291663a0df2d60d8238c661a8c9b8cbf3677b6f8a92fa56bd7346a3cb5293712eeccc5d6542ee3c441d35a4f61e7d12e2ce489a

morcos mentioned this pull request Nov 19, 2015

Rewrite CreateNewBlock #6898

Merged

jonasschnelli added the Mempool label Nov 20, 2015

petertodd reviewed Nov 20, 2015
View reviewed changes

sdaftuar force-pushed the fix-mempool-limiting branch from f60b903 to 4b0c38c Compare November 30, 2015 21:51

sdaftuar changed the title ~~[Mempool] Fix mempool limiting for PrioritiseTransaction~~ [Mempool] Fix mempool limiting and replace-by-fee for PrioritiseTransaction Nov 30, 2015

petertodd reviewed Dec 1, 2015
View reviewed changes

sdaftuar force-pushed the fix-mempool-limiting branch from 4b0c38c to 7b1af7a Compare December 1, 2015 17:55

sdaftuar force-pushed the fix-mempool-limiting branch from 7b1af7a to 5ba3162 Compare December 1, 2015 19:33

laanwj added this to the 0.12.0 milestone Dec 2, 2015

sdaftuar force-pushed the fix-mempool-limiting branch from 50db147 to 7574ba6 Compare December 2, 2015 16:08

sdaftuar force-pushed the fix-mempool-limiting branch from 7574ba6 to 901b01d Compare December 2, 2015 17:58

sdaftuar and others added 4 commits December 2, 2015 12:59

Fix mempool limiting for PrioritiseTransaction

eb30666

Redo the feerate index to be based on mining score, rather than fee. Update mempool_packages.py to test prioritisetransaction's effect on package scores.

Update replace-by-fee logic to use fee deltas

9ef2a25

Use fee deltas for determining mempool acceptance

27fae34

laanwj merged commit 901b01d into bitcoin:master Dec 21, 2015

sdaftuar mentioned this pull request Jan 18, 2016

[Tests] Eliminate race condition in mempool_packages.py #7368

Merged

luke-jr mentioned this pull request Feb 13, 2016

Bugfix: Rename descendantfees to descendantmodfees #7529

Closed

therealyingtong mentioned this pull request May 6, 2020

Backport removal of Coin Age Priority zcash/zcash#4476

Closed

furszy mentioned this pull request Aug 20, 2020

Up to CTransactionRef point back ports to do list. PIVX-Project/PIVX#1726

Closed

21 tasks

furszy mentioned this pull request Dec 31, 2020

[NET] Inv, mempool and validation improvements PIVX-Project/PIVX#2118

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Mempool] Fix mempool limiting and replace-by-fee for PrioritiseTransaction #7062

[Mempool] Fix mempool limiting and replace-by-fee for PrioritiseTransaction #7062

sdaftuar commented Nov 19, 2015

sdaftuar commented Nov 19, 2015

petertodd commented Nov 20, 2015

petertodd Nov 20, 2015

sdaftuar Nov 24, 2015

petertodd commented Nov 20, 2015

sdaftuar commented Nov 30, 2015

petertodd Dec 1, 2015

sdaftuar Dec 1, 2015

sdaftuar commented Dec 1, 2015

sdaftuar commented Dec 1, 2015

sdaftuar commented Dec 2, 2015

morcos commented Dec 2, 2015

sdaftuar commented Dec 2, 2015

laanwj commented Dec 21, 2015

[Mempool] Fix mempool limiting and replace-by-fee for PrioritiseTransaction #7062

[Mempool] Fix mempool limiting and replace-by-fee for PrioritiseTransaction #7062

Conversation

sdaftuar commented Nov 19, 2015

sdaftuar commented Nov 19, 2015

petertodd commented Nov 20, 2015

petertodd Nov 20, 2015

Choose a reason for hiding this comment

sdaftuar Nov 24, 2015

Choose a reason for hiding this comment

petertodd commented Nov 20, 2015

sdaftuar commented Nov 30, 2015

petertodd Dec 1, 2015

Choose a reason for hiding this comment

sdaftuar Dec 1, 2015

Choose a reason for hiding this comment

sdaftuar commented Dec 1, 2015

sdaftuar commented Dec 1, 2015

sdaftuar commented Dec 2, 2015

morcos commented Dec 2, 2015

sdaftuar commented Dec 2, 2015

laanwj commented Dec 21, 2015