Hermes issues related to Tendermint-Go & SDK

[adizere]

Summary

This is a meta-issue capturing various problems we encountered in Hermes. Solving them requires some tendermint-go familiarity (or even other dependency), requires further context, and may be potentially blocked until an upstream fix appears in tm-go.

Sub-issues

The issues below are relatively high importance (we don't document here long-standing problems or wishes). Most of them should be fixed within 1-2 months ideally.

• Sequence number caching in Hermes is unreliable #1264

The full context for this issue is as follows:

• Our initial attempt at a solution is here Retry send_tx on account sequence mismatch #1349
  • This PR attempt to solve the problem by this logic:
    1. parsing the error code which the chain replies to Hermes whenever the relayer uses the wrong sequence number, ref here;
    2. upon encountering the predefined error code, Hermes re-fetches the sequence number by contacting the full node
• This PR, however, does not take into account a more severe problem w.r.t. to the sequence number, which can be reproduced cf. this comment.
• To solve the more severe problem, it seems we cannot rely on the information which the full node provides, i.e., step (2) in the logic above is not an appropriate fix.
• Different solutions, such as avoiding caching altogether and always querying the full node for the seq. nr, also also not appropriate. The full node cannot be trusted, because one of Hermes' transactions remains stuck in that node's mempool, thus the node mistakenly believes Hermes should use a higher sequence number.
• The only fix we know about is as follows: the relayer operator must restart the full node & explicitly flush the mempool, then Hermes has to reconnect to the full node

To summarize, we cannot solve this problem by making Hermes query the full node for the up-to-date seq. number, since the full node provides us with an incorrect value. Hermes cannot rely either on the deliverTx error message (which reflects the seq. number as the rest of the network sees it), because the full node which Hermes talk to will refuse that sequence number. For a principled fix, we need to get a deeper understanding of the mempool, and potentially submit a new feature request to tm-go mempool maintainers. This comment comprises some further ideas, but they are incomplete.

---

• Websocket size constraints #1299

Status:

• We solved (part of) the problem on the tendermint-rs side
  • cf. Add WebSocketClient::new_with_config to specify the WebSocket connection settings tendermint-rs#975
• We are still aware of another websocket-related problem, which is awaiting a fix on tendermint-go side
  • Tendermint emits events over WebSocket faster than any clients can pull them if tx includes many events tendermint/tendermint#6729
• Thane proposed to partially address 6729 with this PR rpc: Add experimental config params to allow for subscription buffer size control (tm v0.34.x) tendermint/tendermint#7230 (comment), which will be available on tm-go 0.34.x (SDK 0.44.x line); this has the potential of solving our base problem highlighted in Websocket size constraints #1299

---

• Unhelpful fee-related error messages #1400

---

• Account sequence errors from connecting to partially synced chains #1462

---

• RPC client fails to decode block_results with non-nil validator_updates tendermint-rs#1021

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[adizere]

Closing this issue to reduce admin burden. There is only one last outstanding item (#1400) that we're handling individually.