compute prerequisites for tasks outside n=0

[oliver-sanders]

See #3938 (comment)

Tasks that are not active are listed as having no prerequisites. This is especially unhelpful as we are not especially interested in the prerequisites of an active task.

Suggest computing prerequisites for the entire scheduler window (n=1 by default).

[hjoliver]

Need to test current situation again.

Can we interrogate the hidden pool task proxies (used for tracking prerequisites). Further ahead, we need compute the prerequisites (none of which will be satisfied).

[hjoliver]

Easiest way to do this: temporarily instantiate the target task proxy and grab its cycle point-specific prerequisites.

[hjoliver]

The datastore should have these anyway, if the node is in the window. DS to check.

[hjoliver]

Need to check partially satisfied prerequisites.

[hjoliver]

Currently, the scheduler and DB record partially satisfied prerequisites. (And fully satisfied n=0 tasks).

Historical tasks can be assumed to have satisfied prerequisites, and future tasks unsatisfied.

However, if the scheduler is down the UIS can't ask it to instantiate a transient task to get its prerequisites. And, future tasks are not in the DB.

Maybe the scheduler could dump relevant task information to file so that the UIS can use that to compute prerequisites on the fly for any task in the graph, if the scheduler is down?

[hjoliver]

Summary as of #4581:

Historical task data in the n-window is now loaded from the DB, but that does not include prerequisite satisfaction status except for partially satisfied prerequisites, which are stored in the DB as needed for restarts - which we want to infer from task status rather than add to permanent DB bloat.

So currently:

• window expansion (i.e. increase n) at run time will cause some succeeded tasks to re-appear that don't show prerequisites satisfied
• we can't yet request to see historical tasks in the UI (without triggering a reflow) ... but if we could, they would currently have the wrong prerequisite status (but correct data otherwise)
• if you trigger a reflow back in the graph, window tasks behind the reflow will not show prereqs satisfied, and those ahead of won't either - but in this case that's correct for the reflow
• all of a task's job are shown, regardless of flow

(From @dwsutherland: It loads it if present in the DB, so for partially satisfied prereqs at n=1 .. Or infact any future task shows the correct prerequisite satisfaction (along with past tasks that stayed in the window since being active))

[hjoliver]

#4581 is merged

Bumping this Issue to rc2.

Historical task data in the n-window is now loaded from the DB, but that does not include prerequisite satisfaction status (except for partially satisfied prerequisites, which are stored in the DB for restarts).

To close this Issue, we need to infer other prerequisites (fully satisfied or fully unsatisfied) from task status rather than add to permanent DB bloat.

(Note 4581 originally closed this issue, by storing all prereqs in the DB, but we backed that out in favor of the above)

[hjoliver]

@dwsutherland - if you haven't started on this, I'll have a crack at it myself (just taking a look at the code now...)

[hjoliver]

We can close this by reporting no prerequisites (or at least not the satisfaction state of them) for past tasks. See #4849 (comment)

[hjoliver]

(relabeling: reporting past task prerequisites as unsatisfied is a bug)

Moved to rc3 as I thought I had a quick fix ... bug maybe not.

[hjoliver]

UPDATE on the status of this issue:

Currently the datastore:

• correctly shows the status of n=0 task prerequisites, and
• correctly shows future tasks have unsatisfied prerequisites

However, it wrongly shows past tasks loaded from the DB as having unsatisfied prerequisites.

To see this:

[scheduler]
    allow implicit tasks = True
[scheduling]
    [[graph]]
        R1 = "a => b => c => d => e"
[runtime]
    [[c]]
        script = "sleep 1000"

Then do cylc stop --now while c is running, followed by a restart so that 1/b in n=1 has to be loaded from the DB.

Finally, look at the prerequisites reported by cylc show for 1/d (future), 1/c (n=0), and 1/b (past).

As discussed in the last meeting, rather than create DB bloat by retaining prerequisite history for a very niche query (in a conditionally triggered past task, which prerequisites exactly were satisfied to make it trigger?) we will essentially say that past tasks do not have prerequisites - which I think is reasonable.

[hjoliver]

I thought I had quick fix for this, but getting the datastore to ignore prerequisites of past tasks (status > "running") on loading from the DB. However, that was a bit too simple, as the prerequisites then stay ignored (in the datastore) even if the affected tasks are brought back to life by re-triggering.