Support for disabling tx confirmation mechanism

[adizere]

Crate

ibc-relayer

Problem Definition

The tx confirmation feature has no specific use beyond its function to retry submitting certain transactions that have not been confirmed after a certain predefined timeout. An additional nice benefit of this feature is that a relayer operator can traverse the logs and obtain more thorough information about when certain transaction hashes have been confirmed.

https://github.com/informalsystems/ibc-rs/blob/33303e17e4d2ec69c15c41dd989192adfcf3311f/relayer/src/link/relay_path.rs#L1238

If the clear_pending_packets mechanism is enabled, then the retrying as part of transaction confirmation is actually redundant and may be creating unnecessary load on the querying interface of the full node that Hermes is using. Since the full node's (g)RPC APIs are fragile and pose a performance bottleneck, we'd like to add support for disabling the tx confirmation feature.

Additionally, from production operation we noticed that the usefulness of this feature is altogether not clear, since it may sometimes take many hours (!!) before Hermes finds confirmations, indicating an overwhelmed full node that is unable to respond to queries. Following is a log line showing that Hermes found a confirmation after ~6K seconds (1h40 mins).

Sep 19 10:55:03 hermes hermes-osmosis[2426]: Sep 19 14:55:03.973 DEBUG [osmosis-1:transfer/channel-0 -> cosmoshub-4] confirmed after 6026.068368672s: TxHashes: count=1; 4422BAEF7D35A4DCEB4148B269AB0027C1F4B93180D3F3FAA68C55A63885BD5B

Proposal & Acceptance Criteria

• add a config.toml option to permit disabling of the tx confirmation feature

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned