block height >= packet timeout height LOOP

[jayjay-crypto]

Summary of Bug

We relay between stargaze and osmosis with hermes. Many times we are forced to reboot hermes because we have an error that turns in loop :

avril 13 20:01:40 hermes[12649]: 2022-04-13T18:01:40.263825Z  INFO ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}:relay{odata=5xBxjpHkjo ->Destination @1-3985853; len=1}: [Async~>stargaze-1] response(s): 1; Err(22):C26ECF62EC0A6CAB8CAB10E58B90919F892FAC398AF919A63A31E959211C8315
avril 13 20:01:40 hermes[12649]: 2022-04-13T18:01:40.263847Z  INFO ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}:relay{odata=5xBxjpHkjo ->Destination @1-3985853; len=1}: success
avril 13 20:01:40 hermes[12649]: 2022-04-13T18:01:40.935410Z  INFO ThreadId(269) packet_cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-181 dst_chain=mainnet-3}:relay{odata=AthsMsMJaP ->Destination @1-3985983; len=1}: assembled batch of 2 message(s)
avril 13 20:01:41 hermes[12649]: 2022-04-13T18:01:41.038184Z  INFO ThreadId(269) packet_cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-181 dst_chain=mainnet-3}:relay{odata=AthsMsMJaP ->Destination @1-3985983; len=1}: [Async~>mainnet-3] response(s): 1; Ok:8188C52D51C67C90999D8F96764D48E6235364BCF83F598D7A1787F8C8DDF81D
avril 13 20:01:41 hermes[12649]: 2022-04-13T18:01:41.039023Z  INFO ThreadId(269) packet_cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-181 dst_chain=mainnet-3}:relay{odata=AthsMsMJaP ->Destination @1-3985983; len=1}: success
avril 13 20:01:42 hermes[12649]: 2022-04-13T18:01:42.267500Z ERROR ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}: timed out while confirming TxHashes: count=1; D06297095E223D54C5B63F105E0C3793F08D0F6375DC5EAB898DB49932815841
avril 13 20:01:42 hermes[12649]: 2022-04-13T18:01:42.273014Z  WARN ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}:relay{odata=71HkzHSCsZ ->Destination @1-3985847; len=3}: [osmosis-1 -> stargaze-1:07-tendermint-0] resolving trusted height from the full list of consensus state heights for target height 1-3985848; this may take a while
avril 13 20:01:42 hermes[12649]: 2022-04-13T18:01:42.705979Z  INFO ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}:relay{odata=71HkzHSCsZ ->Destination @1-3985847; len=3}: assembled batch of 4 message(s)
avril 13 20:01:42 hermes[12649]: 2022-04-13T18:01:42.711803Z ERROR ThreadId(14) send_tx{id=stargaze-1}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 3: receive packet verification failed: block height >= packet timeout height (1-2425097 >= 1-2425049): packet timeout", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2425097"} }
avril 13 20:01:42 hermes[12649]: 2022-04-13T18:01:42.712143Z ERROR ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}: error processing pending events in destination chain: failed with underlying error: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 3: receive packet verification failed: block height >= packet timeout height (1-2425097 >= 1-2425049): packet timeout", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2425097"} }
avril 13 20:01:43 hermes[12649]: 2022-04-13T18:01:43.714551Z ERROR ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}: timed out while confirming TxHashes: count=1; D06297095E223D54C5B63F105E0C3793F08D0F6375DC5EAB898DB49932815841
avril 13 20:01:43 hermes[12649]: 2022-04-13T18:01:43.723829Z  WARN ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}:relay{odata=71HkzHSCsZ ->Destination @1-3985847; len=3}: [osmosis-1 -> stargaze-1:07-tendermint-0] resolving trusted height from the full list of consensus state heights for target height 1-3985848; this may take a while
avril 13 20:01:44 hermes[12649]: 2022-04-13T18:01:44.117353Z  INFO ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}:relay{odata=71HkzHSCsZ ->Destination @1-3985847; len=3}: assembled batch of 4 message(s)
avril 13 20:01:44 hermes[12649]: 2022-04-13T18:01:44.122934Z ERROR ThreadId(14) send_tx{id=stargaze-1}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 3: receive packet verification failed: block height >= packet timeout height (1-2425097 >= 1-2425049): packet timeout", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2425097"} }
avril 13 20:01:44 hermes[12649]: 2022-04-13T18:01:44.123154Z ERROR ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}: error processing pending events in destination chain: failed with underlying error: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 3: receive packet verification failed: block height >= packet timeout height (1-2425097 >= 1-2425049): packet timeout", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2425097"} }
avril 13 20:01:45 hermes[12649]: 2022-04-13T18:01:45.127088Z ERROR ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}: timed out while confirming TxHashes: count=1; D06297095E223D54C5B63F105E0C3793F08D0F6375DC5EAB898DB49932815841
avril 13 20:01:45 hermes[12649]: 2022-04-13T18:01:45.137298Z  WARN ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}:relay{odata=71HkzHSCsZ ->Destination @1-3985847; len=3}: [osmosis-1 -> stargaze-1:07-tendermint-0] resolving trusted height from the full list of consensus state heights for target height 1-3985848; this may take a while
avril 13 20:01:45 hermes[12649]: 2022-04-13T18:01:45.482784Z  INFO ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}:relay{odata=71HkzHSCsZ ->Destination @1-3985847; len=3}: assembled batch of 4 message(s)
avril 13 20:01:45 hermes[12649]: 2022-04-13T18:01:45.488558Z ERROR ThreadId(14) send_tx{id=stargaze-1}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 3: receive packet verification failed: block height >= packet timeout height (1-2425097 >= 1-2425049): packet timeout", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2425097"} }
avril 13 20:01:45 hermes[12649]: 2022-04-13T18:01:45.488759Z ERROR ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}: error processing pending events in destination chain: failed with underlying error: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 3: receive packet verification failed: block height >= packet timeout height (1-2425097 >= 1-2425049): packet timeout", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2425097"} }
avril 13 20:01:46 hermes[12649]: 2022-04-13T18:01:46.491364Z ERROR ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}: timed out while confirming TxHashes: count=1; D06297095E223D54C5B63F105E0C3793F08D0F6375DC5EAB898DB49932815841
avril 13 20:01:46 hermes[12649]: 2022-04-13T18:01:46.496519Z  WARN ThreadId(106) packet{src_chain=osmosis-1 src_port=transfer src_channel=channel-75 dst_chain=stargaze-1}:relay{odata=71HkzHSCsZ ->Destination @1-3985847; len=3}: [osmosis-1 -> stargaze-1:07-tendermint-0] resolving trusted height from the full list of consensus state heights for target height 1-3985848; this may take a while

Version

hermes 0.13.0+2375a069

Steps to Reproduce

Acceptance Criteria

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[adizere]

Hi,
many thanks for opening this issue. This is a big priority for us, being tracked in #1792. We have a preliminary solution being tested & finalized in #1951. I will go ahead and close this issue as duplicate, please refer to PR #1951 for seeing our progress.

[jayjay-crypto]

Hi, thanks for coming back, and sorry for duplicate issue.