Sequence mismatch can needlessly occur when transactions are submitted during ReCheckTx phase

[cmwaters]

Problem Definition

In celestiaorg/celestia-node#3986, users experienced seeing sequence mismatches (i.e. incorrect nonces) although we have a TxClient that is designed to handle this in a thread-safe way. The problem turns out not to be the signer which appears to use locks correctly but the underlying consensus node.

Say we have a mempool with 10 transactions from the same user from nonce 1-10. Let's say the first 5 are committed in a block and the other 5 are left over. Upon committing the block, the last nonce in state is updated to 5 and we pass through all the transactions back to the application (called rechecktx) - these need to be in order of nonce. If everything works as planned, the nonce tracked by the app goes to 10 and then new transactions starting with nonce 11 can be processed. If however a new transaction with nonce 11 is submitted in between this recheckTx phase, say between rechecking transaction 6 and 7, it will have an incorrect nonce and thus be rejected. Once it gets rejected all other transactions from that user are rejected.

Proposal

The simplest solution to this would be to remove the set of go routines that are spun up for each rechecked tx that call the application. We should probably also work out if this serialization leads to any performance degradation (I can't imagine it does because ABCI is sequential anyway).

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned