extend expiration (time limit) of running job

[grondo]

The expiration set in R is currently static, i.e. there is no way to extend the expiration of a running job.

@garlick proposed the idea of allowing updates to the expiration via the job eventlog. Interested parties (e.g. job-exec module and resource module of a child job) could watch the eventlog and update timers or their own expiration as required.

@ryanday36, what is relative priority of this feature? I don't think we've captured it before, but perhaps it falls under the general umbrella of #2799. (Though there is a practical difference between changing the duration before the job has started and the expiration after.)

Tasks

Beta Give feedback

1. Keep a copy of R in job-manager for use in jobtap plugin callbacks #5471
   +0
2. Allow expiration of running jobs to be adjusted via sched.expiration RPC flux-sched#1079
   +0
   
3. mock sched.expiration RPC in sched-simple

[ryanday36]

Yeah, I think it falls under the general rubric of #2799, although priority-wise I'd put it under 'general compute' or maybe even 'nice to have eventually' on the RM Operational Team Interlock. It'll definitely need to be aware of the work on job limits too.

[garlick]

@garlick proposed the idea of allowing updates to the expiration via the job eventlog.

I don't actually remember what I proposed here but given that this is essentially a resource update, it seems like we might have been thinking of something like an R-update event similar to jobspec-update? This would have the advantage of recording the change for the job record, as well as going to the job manager journal so that job-list could update its data.

When the job is a flux instance, we'd need some way to update the resource module's "inventory" (reflected in kvs resource.R) and we would also need a way to inform the scheduler of the change to R through the resource acquisition protocol.