Support non IBC upgrade genesis restarts

[colin-axner]

Crate

relayer

Summary

When a chain undergoes a genesis restart which does not utilize an IBC upgrade proposal, the chain history will be reset. Hermes will attempt to get the trusted header after the restart, but it was erased during the genesis restart:

Jul 06 10:48:19.529 ERROR failed during a client operation: error raised while updating client: failed building header with error: Light client error for RPC address network1-1: Internal error: height 26 is not available, lowest height is 1101 (code: -32603):

Problem Definition

While using in process upgrades are ideal, chains will still opt for genesis restarts from time to time. Not all of these restarts will require an IBC upgrade. This is a bit of an edge case scenario, but if it does occur then it'd be a pain to try to fix live since it would need to be fixed before the client expires

Proposal

I'm not sure what the best solution is. Short term patches might include creating a genesis restart specific command to update a client by querying a node which hasn't been reset for the trusted header. Alternatively there could be a command to rely on historical entries, but these might be pruned after 24 hours. The best solution would likely rely on a cache of old headers. I think the trusted header must have been referenced at some point in the past to update the client. Maybe keeping this header around could be the solution, but I imagine this might be a larger change.

I mostly wanted to open this issue as food for thought in case there are other ongoing structural changes that might allow for this to be handled smoothly

Acceptance Criteria

Hermes can update clients after non IBC upgrade genesis restarts

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[sainoe]

Hi, I have encountered this same issue in the context of building a testnet for Interchain Security.

The steps to reproduce the issue:

1. Clone the IS-testnet repo

git clone https://github.com/sainoe/IS-testnet
cd ./is-testnet/docker/

2. Run the container and the testnet

docker build --tag is-testnet .
make run-testnet

3. Restart the provider chain

make restart-provider

4. Delegate tokens (send a IBC packet through CCV)

make delegate

5. Check the error message in the relayer logs

make relayer-logs

Note that sometime the relayer has to be restarted using make restart-relayer to observe the issue.

[romac]

@colin-axner @sainoe @adizere Should you ever run into this again, we now added two new flags to update client which would let you update a client across the restart by using an archive node and specifying its RPC address as well as the restart height (ie. the lowest block available on the node that underwent the restart):

hermes update client --host-chain consumer --client 07-tendermint-0 --archive-addr http://some.archive.node --restart-height 1101

Hermes will then use the archive node to fetch blocks below 1101 and the live node for the blocks at and above 1101, which enables it to update the client across the restart height.

[colin-axner]

Thank you @romac! This is so cool! Thanks y'all for your work 🙏