You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Given a job is completed on a remote system without a shared file system.
The output files of the job must be copied back to the file system which can be seen by the bartender service.
In #17 the copying of output files is done during handling of GET /api/job/<jobid> or GET /api/job HTTP requests.
This works fine for small amounts (<1Mb) of bytes to copy, but for big amounts (~40Gb) this will cause timeouts on the HTTP requests.
A new job state should be introduced like staging_out to signal that copying of files is happening for a job.
The staging_out job state can also be used to prevent copying multiple times This is not a big issue as files should not change after job completion.
For uploading input files to a remote file system the background task is already used.
The text was updated successfully, but these errors were encountered:
Given a job is completed on a remote system without a shared file system.
The output files of the job must be copied back to the file system which can be seen by the bartender service.
In #17 the copying of output files is done during handling of
GET /api/job/<jobid>
orGET /api/job
HTTP requests.This works fine for small amounts (<1Mb) of bytes to copy, but for big amounts (~40Gb) this will cause timeouts on the HTTP requests.
It would be nice if we could use https://fastapi.tiangolo.com/tutorial/background-tasks or use some file copying queue.
A new job state should be introduced like
staging_out
to signal that copying of files is happening for a job.The
staging_out
job state can also be used to prevent copying multiple times This is not a big issue as files should not change after job completion.For uploading input files to a remote file system the background task is already used.
The text was updated successfully, but these errors were encountered: