Transaction Pool Dropping Certain Transactions

[joepetrowski]

Transaction pool on Westmint is dropping transactions that should be included in a block. Steps:

1. Submit assets.transferKeepAlive(id, account, ...) => It works.
2. Submit assets.block(id, account) => it works.
3. Submit assets.transferKeepAlive(id, account, ...) => The transaction should be included in a block, but the transfer should fail as the account is blocked. However, it seems to just hang in the transaction pool until eventually being dropped for mortality reasons.
4. Submit assets.transferKeepAlive(id, newAccount, ...) => It works, just ensuring that the sending account can still transact.

@bakhtin to submit node trace logs.

Logs

[juangirini]

Can this bug be replicated on other parachains? Or is it a Westmint particular bug?

[joepetrowski]

AFAIK, Westmint is the only parachain that has the Assets pallet with the latest block functionality.

[bkchr]

@joepetrowski please provide some script to reproduce this.

[NachoPal]

@joepetrowski please provide some script to reproduce this.

I couldn't reproduce it locally (it works as expected), but it is true it is happening in production.

I prepared the right environment for you @bkchr to be able to reproduce it. You just need to submit this extrinsic:
https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fwnd-rpc.stakeworld.io%2Fwestmint#/extrinsics/decode/0x320865010042f82a8f5ae2368b1b9dfb8f5578d658bb420e4f07d7890a5b93a6785b0fa44402093d00

I'll share the signer seed in a DM.

[bkchr]

@NachoPal does this depends on the rpc node or can it be every node? And locally it doesn't work? Maybe related to the version of the on chain runtime? Did you tried to use the same one?

[NachoPal]

@NachoPal does this depends on the rpc node or can it be every node? And locally it doesn't work? Maybe related to the version of the on chain runtime? Did you tried to use the same one?

I tried in different RPC nodes (Dwellir, Parity, Stakeworld) and the transaction gets stuck in the pool in all of them.

Locally it worked, I used this branch https://github.com/paritytech/cumulus/tree/westmint-9425 for the runtime and client, which is the one used in production (not for the client) and the only one that has the new pallet_asset dispatchables. I compared also client differences between my local and production (0.9.42 or 0.9.40) and I did not find any: paritytech/substrate@polkadot-v0.9.42...westmint-9425

Worth to mention that it works if the call is wrapped in a batch call: https://westmint.statescan.io/#/extrinsics/4652255-2

[MathCryptoDoc]

As requested by @bkchr, let me mention here a similar issue on Kusama as well.

Certain batch calls (with undelegate, unvote, claiming staking rewards) are very slow to be included lately. This happens on many RPC nodes (public and local) and polkadot versions (tested just now on 0.9.38 and 0.9.42). They worked fine a few weeks ago (I am not certain of the exact moment).

The easiest way to reproduce this is to undelegate all the OpenGov tracks. Here is a direct link to P.JS that you can submit from any Kusama account that does not participate in OpenGov. (The calls in the batch will fail but that is not the point.)

Lowering the amount of calls in the batch (less tracks) makes it work instantly: for example this link.

On P.JS, the first call will beachball for 5 minutes and then give a red error (probably a timeout). Called from the Python SubstrateInterface, inclusion times go from 3 to 25 and even 40 minutes.

Quite annoyingly, if you terminate the extrinsic and try again, you get the error {'code': 1014, 'message': 'Priority is too low: (1 vs 1)', 'data': 'The transaction has too low priority to replace another transaction already in the pool.'} Maybe that indicates it is still in the pool but not being executed.