VCF annotation via asynchronous request-response pattern #108
Conversation
… restapi and worker Defend against null kwargs in task result Change worker AnyVar app init scheme to work with different Celery pool types
|
Hey @ehclark , apologies for the delay. I have been busy with grant writing. I can review this next week. |
|
Thanks @korikuzma. No rush. I am going to need to make some additional tweaks anyway and will be out next week. I have reverted it to a draft. But if you want to take a quick look next week just to make sure it looks directionally correct that would be helpful. |
|
OK this is ready for review. What I thought was a bigger problem was the result of switching to a different storage class for seqrepo data in our Kubernetes cluster. Which of course was easy to fix. |
|
@korikuzma @jsstevenson @theferrit32 I am going to go ahead and merge this on Monday if there are no other comments/reviews. If you want or need more time to look at it let me know. |
|
👍 looks good to me |
See also #107
The annotation of a large VCF file with VRS IDs can run for a long time. A synchronous request-response pattern where the client is waiting for a response can lead to instability and dropped connections in many environments. For long running processes, a better option is an asynchronous request-response pattern. The client submits a request and receives a 202 response and polling location. The client polls at some interval, receiving a 202 if the request is still pending, a 200 and the result if the request is completed, or a 500 if the request failed.
AnyVar uses the Celery distributed task queue for task management and result caching. The asynchronous feature is optional and the changes for this PR are backwards compatible.
See the README-async.md file for detailed documentation of the async feature and how to enable it.
Closes #107