Reduce initialization time for Hermes start CLI

[adizere]

Crate

ibc-relayer and ibc-relayer-cli

Summary

Make start faster, by directly starting the relevant packet & client workers, instead of scanning all of the IBC states of all chains.

Problem Definition

Relayer operators report that Hermes can take several good minutes (10 or more in production), to start relaying. This slow start problem is mostly caused by the fact that Hermes iterates on all possible clients of a chain during startup, then it does the same with connections and channels, which takes a long time, especially given the misbehavior detection overhead.

https://github.com/informalsystems/ibc-rs/blob/8212fc8ebaa0a51fb45bc04cc656f42eab1cee68/relayer/src/supervisor/spawn.rs#L145

There is a potential to improve startup times, because in production use-cases Hermes is mostly operating on a predefined set of channels per chain, using the [chains.packet_filter] policies. One potential way to fix the slow-start problem is to leverage the fact that we already know which channels Hermes is expected to operate on, so we could avoid scanning clients/conns/channels.

Why do we need this feature?

To improve user experience, and allow operators quicker feedback upon start-ing Hermes in production use-cases.

What problems may be addressed by introducing this feature?

The problems of:

1. packets accumulate in a backlog if Hermes takes a long time to start relaying packets
2. slow feedback cycle: operators take a long time to understand if Hermes is relaying correctly or not

What benefits does IBC-rs stand to gain by including this feature?

The benefit of improving relayer operator use-case(s) and providing operators more confidence in our software.

Are there any disadvantages of including this feature?
There is a potential overlap or interference with feature requested in #1518.

Proposal

In cases when channels are defined using [chains.packet_filter] policies, we should adapt Hermes so that it will directly start the relevant packet & client workers, instead of scanning all of the IBC state of a chain.

Acceptance Criteria

• reduce the start time from O(minutes) to O(seconds) when channel packet allow relaying policies are defined

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[romac]

Should this just be the default if a chain is configured with an allow policy? We'd then fall back on scanning the chain for deny policies and if no policy is specified.