Receive packet fails chain proof verification due to duplicate send_packet events

[ancazamfir]

Summary of Bug

[Thanks to @mturkia for the help with debugging]

This issue is similar to #2453
Was found when debugging on the injective - nois testnet.

For nois-testnet-005:channel-16 --> injective-888:channel-51 the packet commitment for sequence 3 is present on nois and not received by injective. When searching for the send_packet event on nois, two Tx-es are returned by the /tx_search query which looks like this:

http://135.181.252.92:26657/tx_search?query="send_packet.packet_src_channel='channel-16' AND send_packet.packet_sequence='3' AND send_packet.packet_src_port='wasm.nois1xwde9rzqk5u36fke0r9ddmtwvh43n4fv53c5vc462wz8xlnqjhls6d90xc' AND send_packet.packet_dst_channel='channel-51' AND send_packet.packet_dst_port='wasm.inj1492g0dmg4l7dm6nppsy5zhuwh8q350j2wfrxrd'"

The send_packet event is present in both Tx-es for the same channel and sequence number and can be checked with

hermes query tx events --chain nois-testnet-005 --hash..

hermes selects the event from the first Tx. However in this first Tx there are multiple messages, one is the send_packet with sequence 3 and the after this there is a write_acknowledgment for the packet with sequence 2. In the events for the write_acknowledgment there is an error:

                    EventAttribute {
                        key: "packet_ack",
                        value: "{\"error\":\"ABCI code: 5: error handling packet: see events for details\"}",
                        index: true,
                    },

In this case (afaik) all messages are dropped, meaning that the nextSequenceSend is not incremented on chain, the commitment is not stored, etc. But it seems that the send_packet event is still emitted and stored by tendermint event indexer. So when hermes clears the packet it will find it there.

We should get feedback from the ibc-go team to confirm and better understand the issue [cc @crodriguezvega].

Until then I think we should fix this by getting the event from highest height (for both /tx_search and /block_search cases)

Note: For #2453 we introduced the --packet-data-query-height. I tentatively tried this:

hermes --config config_inj.toml tx packet-recv --dst-chain injective-888 --src-chain nois-testnet-005 --src-port wasm.nois1xwde9rzqk5u36fke0r9ddmtwvh43n4fv53c5vc462wz8xlnqjhls6d90xc --src-channel channel-16 --packet-data-query-height 930204

where 930204 is the height of the block that included the last Tx. But the problem is that the application state has been pruned and the proof returns empty.

Detailed logs and debug.

Version

Steps to Reproduce

Acceptance Criteria

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[ancazamfir]

In this case (afaik) all messages are dropped, meaning that the nextSequenceSend is not incremented on chain, the commitment is not stored, etc. But it seems that the send_packet event is still emitted and stored by tendermint event indexer. So when hermes clears the packet it will find it there.

We should get feedback from the ibc-go team to confirm and better understand the issue [cc @crodriguezvega].

In fact the Tx has not failed but rather that message is a NACK, thanks @colin-axner for pointing that out.