hardening the cancelation functionality #675
Conversation
openeo/extra/job_management.py
Outdated
| try: | ||
| running_start_time_str = row.get("running_start_time") | ||
| if not running_start_time_str or pd.isna(running_start_time_str): | ||
| _log.warning(f"Job {job.job_id} does not have a valid running start time. Cancellation skipped.") |
There was a problem hiding this comment.
This warning might be a bit too alarming. It will be shown every minute on each job that has no recorded start time, so this could be quite spammy.
"cancellation skipped" might also give wrong impression that job manager still thinks that job should be cancelled for some reason, but it's won't actually do it.
some possible improvements:
- only show this once per job, or for the whole job tracking run
- if running start time is missing, fill it in with timestamp of first observation that it's missing, to have a fallback value, so that the auto-cancel feature can still work
There was a problem hiding this comment.
Is the underlying issue then in:
` if previous_status in {"created", "queued"} and new_status == "running":
stats["job started running"] += 1
active.loc[i, "running_start_time"] = rfc3339.utcnow()
if new_status == "canceled":
stats["job canceled"] += 1
self.on_job_cancel(the_job, active.loc[i])
if self._cancel_running_job_after and new_status == "running":
self._cancel_prolonged_job(the_job, active.loc[i])`
The problem would be removed if I also only run the cancel prolonged job if the previous state was created or queued. Then we know for sure that a starting time has been set?
There was a problem hiding this comment.
that won't work in practice: you want cancelling to happen long after the state changed to "running", so both previous state and current state will be "running" when you typically want to cancel.
what you could do is changing the setting of "running_start_time", to something like (pseudo-code):
if running_start_time is not set and new_status == "running":
active.loc[i, "running_start_time"] = rfc3339.utcnow()
then running_start_time further degrades to a best effort guess of the actual start time, but at least you have something to work with
No description provided.