Do not retry clearing with same height forever #2348
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
romac
reviewed
Jun 28, 2022
.changelog/unreleased/improvements/ibc-relayer/2155-remove-infinite-loop.md
Outdated
Show resolved
Hide resolved
romac
approved these changes
Jun 30, 2022
There was a problem hiding this comment.
Works just as expected, the relayer properly recovers and does not get stuck in a loop and will then attempt to clear the packets (which will also fail in the test scenario with pruning=everything but should otherwise work).
Thanks @ancazamfir!
5 tasks
hu55a1n1
pushed a commit
to hu55a1n1/hermes
that referenced
this pull request
Sep 13, 2022
## Description `spawn_packet_cmd_worker` runs `handle_packet_cmd` passing a "command" that can be `NewBlock`. The first thing it does is to try clearing packets. If this fails with ignorable error, the old code will retry immediately with same command and height. If the error persists (as in informalsystems#2155) then the worker enters an infinite loop and new IBC events (coming via `IbcEvent` commands) are never processed. I believe the infinite loop started with informalsystems#2238. Before that the worker would abort (due to `TaskError::Fatal(RunError::retry(e))`) The propose change is to move to the next command even if previous has failed. The reasons are: - it avoids the infinite loop - the failed events will still be processed at next clear interval with a fresh height - at different levels down in `handle_packet_cmd` there are retries mechanisms for MAX_RETRIES (current value hardcoded at 5). ## Commits * Do not retry clearing with same height forever * Undo one-chain script changes * Add changelog * Reword changelog entry * Remove dbg! statement * Formatting Co-authored-by: Romain Ruetschi <romain@informal.systems>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes: #2155
Description
spawn_packet_cmd_workerrunshandle_packet_cmdpassing a "command" that can beNewBlock.The first thing it does is to try clearing packets. If this fails with ignorable error, the old code will retry immediately with same command and height. If the error persists (as in #2155) then the worker enters an infinite loop and new IBC events (coming via
IbcEventcommands) are never processed.I believe the infinite loop started with #2238. Before that the worker would abort (due to
TaskError::Fatal(RunError::retry(e)))The propose change is to move to the next command even if previous has failed. The reasons are:
handle_packet_cmdthere are retries mechanisms for MAX_RETRIES (current value hardcoded at 5).PR author checklist:
unclog.docs/).Reviewer checklist:
Files changedin the GitHub PR explorer.