-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor Job manager routes #296
base: main
Are you sure you want to change the base?
Conversation
6240cda
to
0f71084
Compare
…routes for jobs router
Co-authored-by: Chris Burr <chrisburr@users.noreply.github.com>
634dd67
to
22449b1
Compare
…schedule_job, eventually will extend to set_job_statuses)
65da423 is the start of using bulk insert/update statements in the job manager, and are supposed to be significantly faster than lots of single insert/update statements. I started with rescheduling and job status updates. |
|
||
# Update JDL (Should we be doing this here?) | ||
result = await job_db.setJobJDL(job_id, jobJDL) | ||
# DATABASE OPERATION | ||
await job_db.setJobJDLsBulk(jdl_changes) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I should check which rows were successfully updated in the previous operation and exclude unsuccessful updates from the JDL update statement.
@fstagni do you have any view on things to watch out for with bulk inserts and updates, based on previous experience in DIRAC? I can foresee some risks of things becoming inconsistent, which could be mitigated by reducing the batch sizes or carefully checking the input and results of each statement execution. |
job_manager
tojobs
TaskGroup
orForgivingTaskGroup
overasyncio.gather
, etc where appropriatereschedule_job
to respect the job state machineFor #22
Needs DIRACGrid/DIRAC#7794
TODO:
config
object to pull outMaxRescheduling
from the CS instead of making a property in JobDB