Bug: Frequent transaction hash collision on mainnet

[alexprut]

Currently on the mainnet there are multiple cases of the same transaction hash included in different block numbers.
The content of the transaction does not appear to be the same in all the clashing cases, as an example consider:

• Both at block 10712420 and 10712421 is included the same transaction 7dcdcdb558028c7703faddee96e1aa48e9bbe7cd3a07cd0a74513a3599185b40, in this case the content of the transaction seems to be the same.
• Also at block 24301274 and 24301275 the transaction 0bd119c4fea6a1147b0eb6cfab1363c1c83b12eaab466fae35b74906d4cf69bc, in this case the content of the transaction seems to be different.

[rajathchain]

I would like to add onto that:
There are currently multiple transaction hash collisions in consecutive blocks in EOS as noted below:

1. 7dcdcdb558028c7703faddee96e1aa48e9bbe7cd3a07cd0a74513a3599185b40 at block 10712420 and 10712421
2. 23361432428f9fdf3f6c2d567b1742e365016d44866f41fceac41962855788ef at block 12587093 and 12587094
3. 12e1ff77c2353e6c6aedbfdbc553d5ef37795060bc950ba5ae3184309d3f8008 at block 13669127 and 13669128
4. 58fd855c2351c07daefb947fd8996381ae10b788812bea903ceffa668a451112 at block 13669235 and 13669236
5. 1d3c6d1db78b1c76b5aedd75c5825fac2a2a81dddfaa29966fd4f6a1afa56fd1 at block 15717258 and 15717259
6. 1109ca2314426f745f378d3ae7384c859e9a60b8b254ab87e0ad6c41b444d8ed at block 19109823 and 19109824
7. 2fc89e92e8e5230d18e3611ed37bf628db94463d9ad62be1e763111bf2ad1cc6 at block 19656619 and 19656620
8. 6a94a62c6467de6392bb7a6b2106d84414c1fcaaa8e1bc47b053e8213f422f73 at block 22037546 and 22037547

Using the Trace API, we have found in the case of the above 8 collisions, the transactions to be exactly the same in the consecutive blocks with the only difference being CPU and NET usage.

However, in the following two transaction hash collisions in consecutive blocks, the transactions are not exactly the same and have different set of eosio.token transfers associated with them:

1. 0bd119c4fea6a1147b0eb6cfab1363c1c83b12eaab466fae35b74906d4cf69bc at block 24301274 and 24301275
2. f0da5255676a1789b617fad1cd1940eba308d89013d03b963ecf6bbda5931288 at block 24305245 and 24305246

We're sure we'll find more collisions as we progress through the chain.

Are these transaction hash collisions a feature or a bug? And are they resolved later in the chain? And what is the feature/bug that is causing them?

[alexprut]

Pinging @johndebord @larryk85 @lparisc @heifner, any thoughts on this? 🙏

[jafri]

@alexprut @rajathchain those are deferred TXs. The scheduling appears in one block and the execution in the later block

[aclark-b1]

This should be fixed by the patch in issue #6115. As far as we know this fix hasn't been deployed on the EOS network, but if that's not the case please let me know.

[alexprut]

@jafri @aclark-b1 thanks for looking into this, a few follow-up questions:

1. Is there any extensive documentation that explains this type of Tx behaviour (without looking into the code)? I was looking at https://developers.eos.io/manuals/eosio.cdt/v1.7/best-practices/deferred_transactions but there is not much.
2. According to https://developers.eos.io/welcome/latest/protocol/transactions_protocol/#transaction_receipt-schema, shouldn't the transaction status be delayed?
3. Currently there is a pattern, i.e. all the clashing tx hashes are included in the immediate successive block, is this done intentionally by design or it's just a coincidence?

[b1bart]

@alexprut

1. Is there any extensive documentation that explains this type of Tx behaviour (without looking into the code)? I was looking at https://developers.eos.io/manuals/eosio.cdt/v1.7/best-practices/deferred_transactions but there is not much.

Not at this time, we can look into documenting it further in the future. While this behavior is allowed by the protocol prior to deployment of NO_DUPLICATE_DEFERRED_ID released in v1.8.0, the conditions for its appearance were also mitigated in the v1.8.0 release. Seeing recent incidences implies that there is a block-signing node running custom code or code prior to v1.8.0 (which is rare due to other activated protocol features in most networks).

2. According to https://developers.eos.io/welcome/latest/protocol/transactions_protocol/#transaction_receipt-schema, shouldn't the transaction status be delayed?

No, delayed transactions are distinct from deferred transactions and this only occurs with deferred transactions. Delayed transactions are explicitly constructed and signed by a user but contain a delay to meet authorization requirements. Deferred transactions are implicitly created as a result of contract execution.

I will file an issue to get the ambiguity in those documents corrected.

It is worth noting that seeing these transactions with same IDs and a status of executed does mean that the contents of the transaction were executed each time. Even though the IDs are the same, each of these executions was individually authorized and scheduled by the source contract so there is no accidental duplication of actions.

3. Currently there is a pattern, i.e. all the clashing tx hashes are included in the immediate successive block, is this done intentionally by design or it's just a coincidence?

This is not intentional however, it is a hallmark of the conditions that create these transactions with the protocol AND the officially released code prior to v1.8.0. Technically the protocol allows any number of duplicate IDs in the same block using this code path followed by at-most 1 duplicate ID in the following block. However, the code prior to v1.8.0 limited this to exactly 1 and 1 outside of the protocol.

Again, after v1.8.0 this should be functionally impossible to achieve and with the optional deployment of NO_DUPLICATE_DEFERRED_ID it will be impossible at the protocol layer.

[alexprut]

@b1bart this is super helpful, thanks for answering! At this point, should we leave the issue open or it makes sense to close it?

[lparisc]

@alexprut Let's keep it open until EOSIO/eosio.cdt#971 is addressed. @iamveritas