feat: orchestrator not failing and requeing failed nonces #631
Conversation
rach-id
added
enhancement
WalkthroughThe recent updates focus on refining the retry logic and enhancing debuggability. The retrier's parameters have been adjusted for more frequent attempts, and logging levels have been shifted to debug for less verbosity in standard operation. The orchestrator's methods now require a context passed in, promoting better control over goroutine lifecycles. Additionally, a new requeue mechanism has been introduced, allowing for more resilient error handling and potential reprocessing of tasks. Changes
TipsChat with CodeRabbit Bot (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review Status
Actionable comments generated: 1
Configuration used: CodeRabbit UI
Files selected for processing (4)
- cmd/blobstream/orchestrator/cmd.go (1 hunks)
- e2e/scripts/start_orchestrator_after_validator_created.sh (2 hunks)
- helpers/retrier.go (1 hunks)
- orchestrator/orchestrator.go (6 hunks)
Additional comments: 12
cmd/blobstream/orchestrator/cmd.go (1)
- 113-113: The change in the retrier instantiation parameters from 6 attempts with a 1-minute interval to 5 attempts with a 30-second interval is correctly implemented.
e2e/scripts/start_orchestrator_after_validator_created.sh (2)
54-58: The addition of
--log.level debugwill increase the verbosity of the logs, which is useful for debugging. Ensure that this level of detail is appropriate for the environment where the script will be run, as it may result in large log files.67-71: The same change has been applied to the else branch of the if-else block, adding
--log.level debug. As with the previous hunk, verify that the increased log verbosity is suitable for the environment.helpers/retrier.go (4)
40-40: The change in logging level from "Info" to "Debug" for retry attempts is consistent with the summary and will reduce log verbosity as intended.
43-43: The log message for a successful retry remains at the "Info" level, which is appropriate for signaling important events like recovery from errors.
46-46: The log message for a failed retry attempt is at the "Debug" level, aligning with the changes to make the logs less verbose.
37-49: The exponential backoff strategy in
NextTickcould lead to very long delays for high retry counts. Ensure this aligns with the desired behavior, especially given the changes to the retry mechanism in other parts of the system.orchestrator/orchestrator.go (5)
29-31: The addition of
RequeueWindowconstant is consistent with the summary and seems to be used correctly in theMaybeRequeuemethod.73-87: The
Startmethod correctly accepts a context as an argument, which is a change from the previous version where it created its own context. This is consistent with the summary.73-88: > Note: This review was outside the patches, so it was mapped to the patch with the greatest overlap. Original lines [73-111]
The
StartNewEventsListener,ProcessNonces, andEnqueueMissingEventsmethods correctly accept a context as an argument, which is a change from the previous version where they used a context with cancel. This is consistent with the summary.
- 254-282: > Note: This review was outside the patches, so it was mapped to the patch with the greatest overlap. Original lines [240-263]
The
ProcessNoncesmethod has been updated to callMaybeRequeueinstead of closingsignalChanand returning an error, which aligns with the summary and the goal of requeuing failed nonces.
- 265-278: The
MaybeRequeuemethod has been added and correctly implements the logic to determine whether a failed nonce should be requeued, which aligns with the summary.
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #631 +/- ##
==========================================
- Coverage 25.91% 25.49% -0.43%
==========================================
Files 29 29
Lines 3044 3095 +51
==========================================
Hits 789 789
- Misses 2160 2211 +51
Partials 95 95 ☔ View full report in Codecov by Sentry. |
Overview
Checklist
Summary by CodeRabbit
Refactor
New Features
Bug Fixes