add(mempool): Verify transactions with unmined inputs in the mempool #8857

arya2 · 2024-09-09T19:45:37Z

Motivation

We want to verify transactions in the mempool that spend the transparent outputs of other transactions in the mempool.

Closes #8777.

Solution

Adds a handle to the mempool service in the transaction verifier
Adds AwaitOutput request in the mempool service
Updates the spent_utxos() method in the transaction verifier to try querying the mempool for spent outpoints before returning a TransparentInputNotFound error.
Polls the mempool service from the transaction verifier after verifying a mempool transaction that creates new UTXOs (in case they're spent by other mempool transactions that are still being verified)
Updates getblocktemplate method and ZIP-317 transaction selection to include transactions that spent the outputs of other transactions in the mempool when all of their dependencies have been selected

Tests

Added tests to check that:

Inserting a transaction also updates the transaction dependencies tracked in the verified set
The Storage.remove() method works as expected (the rest of evict_one() should be covered already)
The mempool responds to AwaitOutput requests when the output for the queried outpoint is:
- Added to the mempool
- Already available in the mempool
The transaction verifier:
- Calls the mempool with AwaitOutput requests as expected and processes UnspentOutput responses correctly
- Polls the mempool after verifying a mempool transaction that creates new transparent outputs
- Includes all of the expected spent_mempool_outpoints in its response
Long poll ids remaining consistent as long the mempool contents are unchanged (and everything else is unchanged)
ZIP-317 mempool transaction selection excludes transactions that have dependencies unless their dependencies have already been selected

If needed, this PR could also add tests for:

ZIP-317 mempool transaction selection always returns the right order of transactions (this could be unnecessary since they’re sorted afterwards anyway but it could be a good sanity check)
The getblocktemplate method returns transactions with dependencies in the right order when the transaction hashes would have them sorted in the wrong order
Ideally at least a basic acceptance test that checks that it all works via RPCs

Follow Up Work

The mempool's Downloads sends a result to the response channel too early, it should wait until the transaction is being inserted into the mempool's verified set.

PR Author's Checklist

The PR name will make sense to users.
The PR provides a CHANGELOG summary.
The solution is tested.
The documentation is up to date.
The PR has a priority label.

PR Reviewer's Checklist

The PR Author's checklist is complete.
The PR resolves the issue.

… mempool setup argument, adds and uses an `init_test()` fn for passing a closed channel receiver in tests where no mempool service is needed in the transaction verifier.

…nd a `new_for_tests()` constructor for convenience)

…t receiver, updates tests.

…ol service handle.

…dSet` `transactions()` return type.

…om the mempool's verified set

…onse

…m the tx verifier response

…ion outputs

…nsparent outputs has been verified. adds a TODO for adding a `pending_outputs` field to the mempool Storage

…ng outputs requests when inserting new transactions into the mempool's verified set

…ified set

… the caller handle it (required to avoid moving zip317 tx selection code to mempool)

…nless their dependencies have already been added.

upbqdn

I'm trying to figure out what happens if someone submits txs containing a circular dependency. We should make sure we can handle such cases well.

arya2 · 2024-11-08T23:48:47Z

I'm trying to figure out what happens if someone submits txs containing a circular dependency. We should make sure we can handle such cases well.

The transaction verifier would return an TransparentInputNotFound error for both because (unlike pending_utxos in the state service) the mempool only responds to AwaitOutput requests once the transaction creating that output has been added to the mempool's verified set.

A similar question around ordering the transactions in block templates caused me some strife (the code should work as expected there too, and it's been excluded from release compilations, but it was difficult to reason about). That exact question may have also caused me some strife a little earlier, as well as what happens to long chains of dependent transactions.

Avoids unnecessarily rejecting dependent transactions of randomly evicted mempool transactions. Updates `TransactionDependencies::remove_all()` to omit provided transaction id from the list of removed transaction ids.

zebrad/src/components/mempool/storage/verified_set.rs

Co-authored-by: Marek <mail@marek.onl>

gustavovalverde · 2024-11-14T12:18:29Z

@mergify refresh

mergify · 2024-11-14T12:18:44Z

refresh

✅ Pull request refreshed

gustavovalverde · 2024-11-18T11:33:49Z

@mergify requeue

mergify · 2024-11-18T11:33:53Z

requeue

❌ This pull request head commit has not been previously disembarked from queue.

gustavovalverde · 2024-11-18T11:35:39Z

@mergify queue

mergify · 2024-11-18T11:35:45Z

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 1dfac40

arya2 added 16 commits September 3, 2024 22:25

Adds a parameter to zebra_consensus::router::init() for accepting a…

fde3370

… mempool setup argument, adds and uses an `init_test()` fn for passing a closed channel receiver in tests where no mempool service is needed in the transaction verifier.

Adds a mempool argument to the transaction::Verifier constructor (a…

85fdd88

…nd a `new_for_tests()` constructor for convenience)

Removes Clone impl on transaction::Verifier to add mempool onesho…

174f4ea

…t receiver, updates tests.

Adds TODOs

3c05b03

updates transaction verifier's poll_ready() method to setup the mempo…

77aa2e3

…ol service handle.

Updates VerifiedSet struct used in mempool storage

5c7f45b

Updates mempool service and its Storage to use the updated `Verifie…

ecc2ddd

…dSet` `transactions()` return type.

updates created_outputs when inserting or removing a transaction fr…

e60197c

…om the mempool's verified set

Adds a TODO, updates field docs

745e6a7

Updates spent_utxos() to query the mempool for unspent outputs

c9c0a11

Adds spent_mempool_outpoints as a field on tx verifier mempool resp…

cbb34e6

…onse

Updates mempool Downloads to return the spent_mempool_outpoints fro…

be6fd7b

…m the tx verifier response

Updates Storage.insert() to accept a list of spent mempool transact…

90812b1

…ion outputs

Adds transaction dependencies when inserting a tx in VerifiedSet

d0b2d41

polls mempool svc from tx verifier when a mempool tx that creates tra…

0a72b41

…nsparent outputs has been verified. adds a TODO for adding a `pending_outputs` field to the mempool Storage

Adds pending_outputs field on mempool Storage and responds to pendi…

2fc7829

…ng outputs requests when inserting new transactions into the mempool's verified set

arya2 added A-mempool Area: Memory pool transactions P-Medium ⚡ labels Sep 9, 2024

arya2 self-assigned this Sep 9, 2024

github-actions bot added C-trivial Category: A trivial change that is not worth mentioning in the CHANGELOG labels Sep 9, 2024

arya2 added 9 commits September 16, 2024 16:07

replaces UnminedTxId type with transaction::Hash in mempool's ver…

ae87a27

…ified set

prune pending outputs when rejecting and removing same effects.

61e6e21

Remove dependent transactions from verified set when removing a tx

bebe50e

updates tests

29654b4

appeases clippy.

aed637a

removes unused len() method

a838934

fixes doc links

695a258

Adds transaction dependencies to the FullTransactions response, let…

bc981c0

… the caller handle it (required to avoid moving zip317 tx selection code to mempool)

updates block template construction to avoid including transactions u…

520660f

…nless their dependencies have already been added.

upbqdn reviewed Nov 8, 2024

View reviewed changes

arya2 added 3 commits November 8, 2024 19:21

Applies suggestions from code review

df9c1c1

Avoids unnecessarily rejecting dependent transactions of randomly evicted mempool transactions. Updates `TransactionDependencies::remove_all()` to omit provided transaction id from the list of removed transaction ids.

Applies suggestions from code review.

874372a

Adds minor comments

5829ef3

arya2 force-pushed the verify-orphaned-mempool-txs branch from e213f22 to 5829ef3 Compare November 9, 2024 00:46

arya2 changed the title ~~add(mempool): Verify transactions in the mempool with unmined inputs~~ add(mempool): Verify transactions with unmined inputs in the mempool Nov 9, 2024

arya2 requested review from conradoplg and upbqdn November 9, 2024 00:56

upbqdn reviewed Nov 11, 2024

View reviewed changes

zebrad/src/components/mempool/storage/verified_set.rs Outdated Show resolved Hide resolved

Update zebrad/src/components/mempool/storage/verified_set.rs

813eeca

Co-authored-by: Marek <mail@marek.onl>

upbqdn previously approved these changes Nov 11, 2024

View reviewed changes

upbqdn removed the C-trivial Category: A trivial change that is not worth mentioning in the CHANGELOG label Nov 12, 2024

Remove an outdated comment (#9013)

e5fbe48

arya2 dismissed upbqdn’s stale review via e5fbe48 November 12, 2024 15:21

github-actions bot added the C-trivial Category: A trivial change that is not worth mentioning in the CHANGELOG label Nov 12, 2024

upbqdn self-requested a review November 12, 2024 15:32

upbqdn approved these changes Nov 12, 2024

View reviewed changes

arya2 removed the extra-reviews This PR needs at least 2 reviews to merge label Nov 12, 2024

mergify bot added a commit that referenced this pull request Nov 18, 2024

Merge of #8857

0d80fe0

This was referenced Nov 18, 2024

merge queue: embarking main (77f1460) and [#8857 + #8967] together #9036

Closed

add(rpc): Adds getblockheader RPC method #8967

Merged

mergify bot merged commit 1dfac40 into main Nov 18, 2024
152 of 154 checks passed

mergify bot deleted the verify-orphaned-mempool-txs branch November 18, 2024 12:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add(mempool): Verify transactions with unmined inputs in the mempool #8857

add(mempool): Verify transactions with unmined inputs in the mempool #8857

arya2 commented Sep 9, 2024 •

edited

Loading

upbqdn left a comment •

edited

Loading

arya2 commented Nov 8, 2024 •

edited

Loading

gustavovalverde commented Nov 14, 2024

mergify bot commented Nov 14, 2024

gustavovalverde commented Nov 18, 2024

mergify bot commented Nov 18, 2024

gustavovalverde commented Nov 18, 2024

mergify bot commented Nov 18, 2024 •

edited

Loading

add(mempool): Verify transactions with unmined inputs in the mempool #8857

add(mempool): Verify transactions with unmined inputs in the mempool #8857

Conversation

arya2 commented Sep 9, 2024 • edited Loading

Motivation

Solution

Tests

Follow Up Work

PR Author's Checklist

PR Reviewer's Checklist

upbqdn left a comment • edited Loading

Choose a reason for hiding this comment

arya2 commented Nov 8, 2024 • edited Loading

gustavovalverde commented Nov 14, 2024

mergify bot commented Nov 14, 2024

✅ Pull request refreshed

gustavovalverde commented Nov 18, 2024

mergify bot commented Nov 18, 2024

❌ This pull request head commit has not been previously disembarked from queue.

gustavovalverde commented Nov 18, 2024

mergify bot commented Nov 18, 2024 • edited Loading

✅ The pull request has been merged automatically

arya2 commented Sep 9, 2024 •

edited

Loading

upbqdn left a comment •

edited

Loading

arya2 commented Nov 8, 2024 •

edited

Loading

mergify bot commented Nov 18, 2024 •

edited

Loading