Workers should retry indefinitely

[greg-szabo]

Crate

hermes

Summary of Bug

sentry node restart took too long and it killed hermes worker:

Aug 16 13:34:34 hermes hermes[25520]: Aug 16 13:34:34.641 ERROR [cosmoshub-4:transfer/channel-141 -> osmosis-1] worker: handling command encountered error: failed during query to chain id osmosis-1 with underlying error: Light client error for RPC address http://10.10.51.151:26657/: node at http://10.10.51.151:26657/ running chain osmosis-1 not caught up path=packet::channel-141/transfer:cosmoshub-4->osmosis-1
Aug 16 13:34:34 hermes hermes[25520]: Aug 16 13:34:34.642 ERROR [core-1:transfer/channel-6 -> osmosis-1] worker: handling command encountered error: failed during query to chain id osmosis-1 with underlying error: Light client error for RPC address http://10.10.51.151:26657/: node at http://10.10.51.151:26657/ running chain osmosis-1 not caught up path=packet::channel-6/transfer:core-1->osmosis-1
Aug 16 13:34:34 hermes hermes[25520]: Aug 16 13:34:34.876 ERROR [cosmoshub-4:transfer/channel-141 -> osmosis-1] worker: handling command encountered error: failed during query to chain id osmosis-1 with underlying error: Light client error for RPC address http://10.10.51.151:26657/: node at http://10.10.51.151:26657/ running chain osmosis-1 not caught up path=packet::channel-141/transfer:cosmoshub-4->osmosis-1
Aug 16 13:34:34 hermes hermes[25520]: Aug 16 13:34:34.876 ERROR [packet::channel-141/transfer:cosmoshub-4->osmosis-1#103] worker aborted with error: Packet worker failed after 7 retries

Version

0.6.1

Steps to Reproduce

Acceptance Criteria

Workers should retry infinitely

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[adizere]

Looking at logs from gravity dex/emeris, the same problem shows up there. We'll on this ASAP, thanks for reporting Greg!

[romac]

As discussed during today's call, there are multiple, mostly orthogonal solutions for this:

a) Retry indefinitely
b) Make retry limit/strategy configurable
c) Detect if node is in fast sync mode and backoff temporarily
d) Add backup nodes to configuration and automatically fall back

We eventually want (b) (c) and (d), with (b) enabling (a), but will start with (a) directly for now.

[greg-szabo]

Sounds good. I was going to say “all of the above”. 😃 Retry indefinitely is a good start and the more complex solutions are nice to have.