attempt to fix export deadlocks #486
Conversation
Signed-off-by: Neil South <neil.south@answerdigital.com>
Codecov Report
@@ Coverage Diff @@
## develop #486 +/- ##
===================================================
- Coverage 78.87427% 78.84177% -0.03250%
===================================================
Files 390 390
Lines 24553 24572 +19
Branches 1098 1098
===================================================
+ Hits 19366 19373 +7
- Misses 4799 4809 +10
- Partials 388 390 +2
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 2 files with indirect coverage changes Continue to review full report in Codecov by Sentry.
|
| @@ -145,6 +145,7 @@ private async Task OnMessageReceivedCallback(MessageReceivedEventArgs eventArgs) | |||
| if (Interlocked.Read(ref _activeWorkers) >= Concurrency) | |||
| { | |||
| _logger.ExceededMaxmimumNumberOfWorkers(ServiceName, _activeWorkers); | |||
| await Task.Delay(200).ConfigureAwait(false); // small delay to stop instantly dead lettering the next message. | |||
There was a problem hiding this comment.
between acknowledge message and releasing the Interlocked.Decrement call, new message can be received in a tight loop, sending them to the deadletter queue.
| @@ -328,12 +347,12 @@ private void ReportingActionBlock(ExportRequestDataMessage exportRequestData) | |||
| _configuration.Export.Retries.RetryDelays, | |||
| (exception, timeSpan, retryCount, context) => | |||
| { | |||
| _logger.ErrorAcknowledgingMessageWithRetry(exception, timeSpan, retryCount); | |||
| _logger.ErrorPublishingExportCompleteEventWithRetry(exception, timeSpan, retryCount); | |||
There was a problem hiding this comment.
swapped these two around, so that the message is acknowledged last, again to help with the fact that the next message will be received before the Interlocked.Decrement is called
Description
We've seen instances where an export message with an incorrect name can halt exports and cause issues.
this adds some try/catch to the TransformBlock's on an exception it now sets the file upload to failed so the process can continue.
Status
Ready
Types of changes