Intelligent Host Select.

[wxtim]

These changes partially address #3827
@datamel should be credited somewhere since we pair programmed the Remote Init and Filinstall logic.

Summary of logic:

• Added a new kwarg, badhosts: set to get_host_from_platform. This is then subtracted from the platforms list of hosts and the selection is carried out on the leftover hosts.
• Added TaskRemoteMgr.badhosts = set(), allowing us to store a list of bad hosts.
• Modified several cases of get_host_from_platform in remote.py to use badhosts if they can.
• Used this infrastructure to allow retrying of remote tasks if they fail with a 255.

Caveats

• I'm leaving platform from platform host as a follow up.
• Current testing is static - i.e. there is no checking what happens if hostA dies, then comes back online then hostB dies. All testing is done with platforms where lists of hosts contain deliberately fictional hosts, where ssh will fail causing a retry.

WIP Checklist

• Host within Platform
  • Create a persistent object (TaskRemoteMgr.badhosts) in which to store hosts which aren't accessible.
  • Created a mainloop plugin clearing TaskRemoteMgr.badhosts from time to time.
  • Remote init working.
  • ... and tested
  • Remote fileinstall working.
  • ... and tested
  • Job submission working.
  • ... and tested
  • Remote tidy working.
  • ... and tested
  • Job log retrieval working.
  • ... and tested
  • Polling Working
  • ... and tested
  • Kill working
  • ...and tested
• Check existing tests work = Now Passing locally.

Standard Checklist

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Appropriate tests are included (unit and/or functional).
  • Test workflow included which may become the basis for future functional tests.
• Appropriate change log entry included.
• (master branch) I have opened a documentation PR at cylc/cylc-doc/pull/XXXX.
• No dependency changes.

[wxtim]

Copied from a comment for @oliver-sanders

Intelligent Host Selection - Job Submission

The idea is Cylc tries to submit the job to one host, if that fails 255 it tries the next one, if it runs out of hosts it puts the job into a submit-failed state.

This means the job submission system has to remember the hosts it has previously tried. At the moment on Tim's branch this state is centralised which makes sense. This way Cylc doesn't try to use hosts which is knows are down. This is good for efficiency and also prevents flooding the system with SSH commands that are sitting around before they hit their timeout.

Problems:

We would have to wipe this global state at some point otherwise hosts which have come back will not be considered for any operation.
If there is a transient network problem all hosts could be added to the blacklist and the workflow would sit around doing nothing until the state is wiped.

Best solution I can come up with is:

Reset this state periodically (use cylc.flow.main_loop.periodical)
AND also reset the state whenever a submit-failed task is retriggered either manually or by a scheduled automatic retry.

[wxtim]

I've tagged multiple people in this review because I am aware that the half term hols are coming up. Feel free to poke about with this.
All tests are passing locally.

[MetRonnie]

11/31 files viewed

[MetRonnie]

21/32 files viewed

[MetRonnie]

33/33 files viewed

[oliver-sanders]

we might not want to lose the info from the logging of a flow run in a less verbose mode

I don't think this message provides us with any new information?

We should log:

• When a new "bad host" is detected.
• When the "bad hosts" set has been cleared.

No need to log the bad hosts with every single host-selection.

[wxtim]

we might not want to lose the info from the logging of a flow run in a less verbose mode

I don't think this message provides us with any new information?

We should log:

* When a new "bad host" is detected.

* When the "bad hosts" set has been cleared.

No need to log the bad hosts with every single host-selection.

Sorry, I've fixed this now - I honestly thought that I had already...

[oliver-sanders]

Couple of small comments on the rsync fallback:

[oliver-sanders]

Still testing, no new issues.

[wxtim]

Processes touched

• Task init
• Task file install
• Task Poll
• Remote Tidy
• Retrieve Job Logs

[oliver-sanders]

Still testing, made it most the way through ... 😓.

There is a potential bug in the rsync:fail => ssh logic if platform is not defined. I guess we could fallback to a plain ssh command although that makes it easy for the platform logic to be silently ignored. Here's a diff anyway:

diff --git a/cylc/flow/subprocpool.py b/cylc/flow/subprocpool.py
index 9c533dc3a..31e925f41 100644
--- a/cylc/flow/subprocpool.py
+++ b/cylc/flow/subprocpool.py
@@ -542,15 +542,21 @@ class SubProcPool:
         may cause different rsync failures depending on version, and some of
         the failures may be caused by other problems.
         """
-        if platform is not None:
-            ssh_cmd = platform['ssh command']
-
         rsync_255_fail = False
         if (
             ctx.cmd[0] == 'rsync'
             and ctx.ret_code not in [0, 255]
             and is_remote_host(ctx.host)
         ):
+            ssh_cmd = (
+                platform['ssh command']
+                if platform is not None
+                else 'ssh'
+            )
+            ssh_test_cmd = shlex.split(f'{ssh_cmd} {ctx.host} true')
+            LOG.info(
+                f'testing connectivity for {ctx.host} using {ssh_test_cmd}'
+            )
             ssh_test = run(
                 shlex.split(f'{ssh_cmd} {ctx.host} true'),
                 capture_output=True

[oliver-sanders]

Save the one small issue above and the one small conflict, all good 🎉!

[wxtim]

@oliver-sanders I think that I've responded with fff1c7b.

Sorry about the close/reopen. I've going to restart my VM. It's very laggy. :(

[oliver-sanders]

Going in!

This was a tricky one, good work!