Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issue with stalled requests if shutdown #3075

Merged
merged 1 commit into from
Mar 20, 2024

Conversation

kbirk
Copy link
Contributor

@kbirk kbirk commented Mar 20, 2024

I believe this to be the cause of the stalled requests that @mwdchang experienced.

If the hmi-server was shutdown between writing the initial TaskStatus.QUEUED state of the task to redis but before it actually sent the request to rabbitmq, then any subsequent request would simply wait for a non-existent task to finish.

This is consistent with the logs as no request was sent, but it did timeout waiting for the response.

This switches the logic to write out the TaskStatus.QUEUED state after it is dispatched. If the server is shutdown after it dispatches to redis but before that state is set, then the subsequent requests will simply re-send it.

Another small issue is that failed task id's were never re-written in the task SHA -> task id lookup. So after a task failed subsequent attempts, if successful, were not cached.

@kbirk kbirk requested review from YohannParis and mwdchang March 20, 2024 16:06
@kbirk kbirk requested review from dgauldie and dvince2 as code owners March 20, 2024 16:06
@kbirk kbirk merged commit 03365d2 into main Mar 20, 2024
6 checks passed
@kbirk kbirk deleted the taskrunner-shutdown-condition-fix branch March 20, 2024 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants