Modified packet worker to use stubborn strategy

[adizere]

Closes: #1290

Description

The following describes worker behavior after this PR:

• packet worker:
  • old behavior: used a default retry strategy that retried for 6 iterations and then gave up;
    • the individual retry backoff varied between 200ms and 500ms
    • the cumulative backoff time was 2 seconds
  • new behavior: using a stubborn strategy, which retries for infinite iterations and never gives up
    • the individual backoff time (between subsequent retries) is 1sec in the beginning, and increases steadily by 10ms at every step

The rationale for this choice is as follows. I expect past a certain duration (I choose in this PR 6 hours), Hermes might as well give up, as there is little chance that the connection can become healthy again without operator intervention, but not sure if my intuition is appropriate.

---

Unchanged behavior:

• client worker

  • this worker is in charge of regularly updating clients (unless there is already channel traffic, which will trigger client updates), at a frequency of 2/3 of trusting period
  • this worker is already stubborn: never quits unless the supervisor instructs it to do so, or if the client is expired or frozen

• channel and connection workers

  • these are in charge of finishing up initialized (but incomplete) connection handshakes or initialized channel handshakes
  • these workers don't need to retry stubbornly, because their activity is time-sensitive

---

For contributor use:

• Added a changelog entry, using unclog.
• If applicable: Unit tests written, added test to CI.
• Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
• Updated relevant documentation (docs/) and code comments.
• Re-reviewed Files changed in the Github PR explorer.

[adizere]

This test is almost identical to this test
https://github.com/informalsystems/ibc-rs/blob/85446586daf6dba652271eef31dae29f35da86c2/relayer/src/util/retry.rs#L153

but I only realized after writing it.

I still found it useful to keep it around, for the purpose of understanding exactly the details of the strategy we use in practice.

[hu55a1n1]

👌💯