Fix unhandled status responses resulting in bad requests being sent repeatedly

[matux]

Description of the change

This PR fixes an issue where unhandled server responses were being set for immediate reprocessing and being queued for retry instead of being dropped.

The logic is excessively complicated and it's not easy to explain, so 🐻 with me.

Depending on the server response, we decide either to requeue an occurrence to be sent 60s later (eg. 503), or immediately drop it (eg. 400).

The timing aspect (do it after 60s or immediately), and the decision (drop or queue) are done separately.

• A first, we check the http status response and decide whether we want to reprocess in 60s, or immediately. (yeah)
• B then, later on, in an entirely different place, we check the http status response and decide whether we want to drop the request or queue it.

The way we were processing unhandled server responses was inverted in first and second. So, during A, we'd instruct the request to be reprocessed immediately (cause it has to be dropped), but during B, we'd instruct the request to be queued instead of dropped.

The result was a failed request that was being resent repeatedly and immediately, forever and ever (because it wasn't being remove from the db either).

Type of change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Maintenance
• New release

Related issues

• Fix SC-128837

Checklists

Development

• Lint rules pass locally
• The code changed/added as part of this pull request has been covered with tests
• All tests related to the changed code pass in development

Code review

• This pull request has a descriptive title and information useful to a reviewer. There may be a screenshot or screencast attached
• "Ready for review" label attached to the PR and reviewers assigned
• Issue from task tracker has a link to this pull request
• Changes have been reviewed by at least one other engineer

[shortcut-integration]

This pull request has been linked to Shortcut Story #128837: Unhandled http status responses lead to bad requests being sent repeatedly, forever.