Improve comm #2582

matthewrmshin · 2018-02-21T12:30:41Z

cylc client: new command to call network API client.

When SSH messaging is already set up for a task remote, other client
commands will pick up the set up automatically without having to use
--use-ssh and related options. Client logic will run locally, with
only network logic done via cylc client --use-ssh.

cylc message:

Expect suite name and task job (cycle/task/submit) as arguments.
Can now send multiple messages at different severities.
API method put_messages (replaces put_message).
- Multiple messages in one call.
- Event time as separate argument.
- Task job (cycle/task/submit) instead of ID (task.cycle).
Backward compatible if called with 1 or 2 arguments.
- Assume new interface when being called with 3 or more arguments.

Server program will ignore task messages from old submits.

Reduce job failed message to just say failed/SIGNAL, etc.

Easier to parse, and more similar to started and succeeded messages.

cylc remote-init: now handles SSH comm set up. Job files no longer
contain SSH comm settings.

Rationalise cylc.task_message and cylc.network.httpclient. Location
of network logic and retry can now be found in the latter.

Create a suite UUID on start up of a new run. This is stored in the
suite runtime database file and written to the contact file. Restart will
load and use the same UUID. The UUID is exported to the environment
of a job script. In a job environment, the client will only attempt a
connection to the suite if the UUID value in the environment matches
with that of the contact file.

Close #439. Close #2214. Close #2502. Close #2528. Tick final box in and close #2302.

matthewrmshin · 2018-02-21T12:52:38Z

~~(May close #2505 as well. Need some testing.)~~

Edited: not really. This change will only deal with the cylc message part of the issue raised in #2505. More work is required to solve that issue fully.

matthewrmshin · 2018-02-22T15:38:33Z

(Trying to figure out how to write an automated test for the suite UUID change. Otherwise, this PR is good for review.)

matthewrmshin · 2018-02-28T09:20:34Z

(Re-based.)

matthewrmshin · 2018-02-28T21:20:24Z

(Re-based. Codacy failure is nonsense.)

matthewrmshin · 2018-03-01T10:22:50Z

(Codacy failure is nonsense.)

oliver-sanders

Still working my way through this but looks pretty good so far.

oliver-sanders · 2018-02-28T11:12:15Z

lib/cylc/network/httpclient.py

+                if func_name == 'put_messages':
+                    results = self._call_server(func_name, payload=payload)
+                elif func_name == 'put_message':  # API 1, 7.5.0 compat
+                    cycle, name = payload['task_job'].split('/')[0:2]


Would it be easier to provide these as separate dictionary keys to avoid having to split the value in multiple places.

The task job or task id is used here for consistency with the other API methods that deal with tasks. We should definitely improve this in the future.

oliver-sanders · 2018-02-28T11:22:57Z

lib/cylc/network/httpserver.py

+    @cherrypy.expose
+    @cherrypy.tools.json_in()
+    @cherrypy.tools.json_out()
+    def put_messages(self, task_job=None, event_time=None, messages=None):


Should these be args rather than kwargs (else we could end up passing 'None' strings into the scheduler or does it make little difference anyway?

It makes no difference here, because the data actually come in via the payload JSON, not really via the arguments.

oliver-sanders · 2018-03-01T10:46:29Z

lib/cylc/remote.py

        command.append(user_at_host)

+        # Pass cylc version through.
+        command += ['env', quote(r'CYLC_VERSION=%s' % CYLC_VERSION)]
+        if 'CYLC_UTC' in os.environ:


Is this a change in behaviour?

This is functionality ported from the old SSH functionality of cylc.task_message. It is beneficial to other commands as well.

matthewrmshin · 2018-03-02T11:08:50Z

(Sorry to keep re-basing here. Still trying to get Travis CI to pass.)

matthewrmshin · 2018-03-02T12:58:00Z

(Travis CI has finally passed, but still reporting as in progress for some reason.)

oliver-sanders · 2018-03-06T10:47:12Z

bin/cylc-client

+        suite, options.owner, options.host, options.port,
+        options.comms_timeout, my_uuid=options.set_uuid,
+        print_uuid=options.print_uuid, auth=auth)
+    kwargs = json.load(sys.stdin)


Whilst cylc client is an internal command it would be nice if we could call it without providing stdin if it is not required i.e.:

cylc client get_suite_state_summary <reg>

Rather than:

cylc client get_suite_state_summary <reg> <<< '{}'

json.load raises a ValueError on an empty file. Although the error message can be used to deduce an empty file, the message may change in the future and break out logic here.

I have now introduced a -n option for those API functions that do not expect extra keyword arguments. It is not as clever, but is good enough.

This solution might work https://stackoverflow.com/a/3763257.

Only as a solution between interactive versus batch usage. Not quite the intention here.

oliver-sanders · 2018-03-06T10:48:47Z

bin/cylc-client

+        print_uuid=options.print_uuid, auth=auth)
+    kwargs = json.load(sys.stdin)
+    sys.stdin.close()
+    sys.stdout.write(json.dumps(getattr(pclient, func)(**kwargs), indent=4))


+1 for pretty printing!

matthewrmshin · 2018-03-20T13:59:25Z

(Re-based and resolved conflicts.)

hjoliver

(started reviewing; I'll try to finish tomorrow.)

hjoliver · 2018-03-26T09:41:21Z

bin/cylc-help

@@ -270,6 +270,8 @@ control_commands['set-verbosity'] = ['set-verbosity']
 control_commands['broadcast'] = ['broadcast', 'bcast']
 control_commands['ext-trigger'] = ['ext-trigger', 'external-trigger']
 control_commands['checkpoint'] = ['checkpoint']
+control_commands['message'] = ['message', 'task-message']


'message' is already categorized as a task_command

matthewrmshin · 2018-03-26T14:09:10Z

I have added some doc strings to the method where Codacy is reporting the security issues related to usage of subprocess.Popen - to justify that there should be no issue. (When #2503 is done, we should move this new Popen call from cylc.network.httpclient to use the new remote_cylc_cmd function in cylc.remote?)

matthewrmshin · 2018-03-26T14:50:01Z

(Re-based + Resolved Conflicts.)

hjoliver · 2018-03-26T21:22:51Z

move this new Popen call from cylc.network.httpclient to use the new remote_cylc_cmd function in cylc.remote?

yep

hjoliver

Looks really good, no problems found. I still need to do a bit of testing...

hjoliver · 2018-03-27T03:49:47Z

lib/cylc/job.sh

-    typeset run_err_script="$3"
-    shift 3
+    typeset run_err_script="$2"
+    shift 2


Comments documenting these arguments needs updating (in the comments just above).

hjoliver · 2018-03-28T04:34:23Z

lib/cylc/remote.py

+                self.owner = arg.replace('--user=', '')
+            elif arg.startswith('--host='):
+                self.host = arg.replace('--host=', '')
+            elif arg == '--ssh-cylc=':


The ssh command generated for ssh task messaging is screwed up. This line is the problem - it should use startswith() not ==.

hjoliver

Looks good to me now.

Generate new suite UUID on normal run. Load previous value on restart. In a task job environment, check that the environment value of the suite UUID matches the one in the suite contact file. Don't send message on mismatch.

cylc client: new command to call network API client. * Replace usage of `curl` in tests with this command. When SSH messaging is already set up for a task remote, other client commands will pick up the set up automatically without having to use `--use-ssh` and related options. Client logic will run locally, with only network logic done via `cylc client --use-ssh`. cylc message: * Expect suite name and task job (cycle/task/submit) as arguments. * Can now send multiple messages at different severities. * API method `put_messages`. * Multiple messages in one call. * Event time as separate argument. * Task job (cycle/task/submit) instead of ID (task.cycle). Server program will ignore task messages from old submits. Reduce job failed message to just say `failed/SIGNAL`. cylc remote-init: now handles SSH comm set up. Job files no longer contain SSH comm settings. Rationalise `cylc.task_message` and `cylc.network.httpclient`. Location of network logic and retry can now be found in the latter.

matthewrmshin · 2018-04-17T13:32:17Z

Branch re-based. Conflicts resolved. Now use new logic of #2503 for SSH cylc client call.

sadielbartholomew

Review in progress - on priority so will complete tomorrow. General proposal as I have understood it so far seems very sensible. Reporting a few possible minor issues I have spotted at this stage.

sadielbartholomew · 2018-04-17T16:30:19Z

bin/cylc-client

+#!/usr/bin/env python
+
+# THIS FILE IS PART OF THE CYLC SUITE ENGINE.
+# Copyright (C) 2008-2016 NIWA


Happy New Year x2: ~~2016~~ 2018

sadielbartholomew · 2018-04-17T16:35:35Z

bin/cylc-client

+    if remrun():
+        sys.exit(0)
+
+import cylc.flags


Unused import (assuming the standardcylc.flags.debug catch in the main() function is not necessary/advised as this command is only for internal use)?

sadielbartholomew · 2018-04-17T16:58:50Z

bin/cylc-message

+failure. Applications run by task jobs can use this command to report messages
+and to report registered task outputs.
+
+Messages can be specified as arguments. A '-' indicates that messages the


Something is amiss with the second sentence: '...that ~~messages~~ the...' seems likely.

sadielbartholomew · 2018-04-17T17:38:40Z

lib/cylc/network/httpclient.py

+                # * Exhausted number of tries.
+                # * Contact info file not found, suite probably not running.
+                #   Don't bother with retry, suite restart will poll any way.
+                if i >= max_tries or isinstance(exc, ClientInfoError):


ClientInfoError is raised separately prior to it's base exception class ClientError on 287:288, so I believe the second condition will never be True as it would have already have been caught?

Fix copyright year. Fix doc for new commands. Fix bad exception handling. Remove pointless warning message.

matthewrmshin · 2018-04-17T19:40:46Z

Minor issues and comments addressed.

sadielbartholomew

Review complete. Test battery passes locally for me, with new tests suitable. Feedback from first sub-review addressed appropriately. I have several new comments, though they are largely minor so can be addressed separately once this has been merged if this would help ease the PR 'logjam', hence approving.

sadielbartholomew · 2018-04-18T09:40:16Z

lib/cylc/network/httpclient.py

+        """Call server via "cylc client --use-ssh".
+
+        Call "cylc client --use-ssh" using `subprocess.Popen`. Payload and
+        arguments of the API method are serialized as JSON and is written to a


A few tiny docstring issues that we could ignore as it still makes sense but ideally we could change: '...is are written...' here & then 'even gets a chance' on line 534.

sadielbartholomew · 2018-04-18T09:51:45Z

lib/cylc/network/httpclient.py

+            json.dump(kwargs, stdin)
+            stdin.seek(0)
+        else:
+            stdin = None


stdin = open(os.devnull) instead unless there is a reason that it is unnecessary or wrong in this case?

This is now handled by remote_cylc_cmd, which will set this to open(os.devnull) and add -n to the SSH command line.

I'll add some extra comments.

Ah I see, sorry - my bad! By all means add some comments if you think others might not see this but having had a second look it seems fairly obvious.

sadielbartholomew · 2018-04-18T10:02:20Z

lib/cylc/network/httpclient.py

+            # Check mismatch suite UUID
+            env_suite = os.getenv(self.srv_files_mgr.KEY_NAME)
+            env_uuid = os.getenv(self.srv_files_mgr.KEY_UUID)
+            if (self.suite and env_suite and env_uuid and


It would I think be easier to comprehend the list of conditions if env_suite == self.suite came before the env_uuid truth condition so that line 628 is a related group (or an re-ordering along a similar line).

Interesting. It proves that our brains are wired differently. I'll make the change to keep you happy.

No need, it's your code & obviously an inconsequential change. It seemed a much more natural grouping to me personally but I can only use my own brain - I can only guess how others might regard things :)

sadielbartholomew · 2018-04-18T10:25:31Z

lib/cylc/network/httpserver.py

+    @cherrypy.tools.json_in()
+    @cherrypy.tools.json_out()
+    def put_messages(self, task_job=None, event_time=None, messages=None):
+        """Put task messages.


This description should be improved with a bit more context.

sadielbartholomew · 2018-04-18T10:49:14Z

lib/cylc/remote.py

@@ -81,21 +85,24 @@ def remote_cylc_cmd(cmd, user=None, host=None, capture=False,
    #   The command is read from the site/user global config file, but we check
    #   above that it ends in 'cylc', and in any case the user could execute
    #   any such command directly via ssh.
-    proc = Popen(command, stdout=stdout, stdin=open(os.devnull))
+    if stdin is None:
+        stdin = open(os.devnull)


Could just set this at the end of the block on line 54 (unless it is particularly important that the code there deals entirely with the commands & the stdin/out in the subsequent code).

sadielbartholomew · 2018-04-18T15:03:15Z

lib/cylc/task_message.py

+        job_log_name = os.path.join(
+            glbl_cfg().get_derived_host_item(suite, 'suite job log directory'),
+            'job')
+    job_status_file = None


Is this necessary? And if so shouldn't it get set to an empty file & not None so that if not caught as an IOError the write() method isn't called on a non-file object on lines 111 etc?

sadielbartholomew · 2018-04-18T15:21:40Z

lib/cylc/task_message.py

+            if job_id > 1:
+                # If os.getppid() returns 1, the original job process
+                # is likely killed already
+                job_status_file.write('%s=%s\n' % (CYLC_JOB_PID, job_id))


Seems quite unwieldy to keep applying the string formatting template '%s=%s\n' so could we assign it as a regex to a constant to apply?

I think it is actually clearer in the context here to use a string literal for format strings instead of introducing a constant.

Fair enough! I was mainly thinking it would make it much simpler if we ever wanted to adjust the 'template' but would probably make the code a little less clear, yes.

sadielbartholomew · 2018-04-18T15:35:12Z

lib/cylc/task_message.py

+                # is likely killed already
+                job_status_file.write('%s=%s\n' % (CYLC_JOB_PID, job_id))
+            job_status_file.write('%s=%s\n' % (CYLC_JOB_INIT_TIME, event_time))
+        elif message == TASK_OUTPUT_SUCCEEDED:


This elif statements use the same format & variables to write to job_status_file as the following two, with only the 'message' changing, so 112:127 could definitely be consolidated (at some point in the future at least).

Not worth doing as part of this change, given that this is not really new code (- just re-indented), and it is unlikely that we can reduce the line count by consolidation.

Good point.

sadielbartholomew · 2018-04-18T15:47:53Z

lib/cylc/task_remote_cmd.py

@@ -56,6 +58,11 @@ def remote_init(uuid_str, rund):
        tarhandle.close()
    finally:
        os.chdir(oldcwd)
+    if indirect_comm:


if indirect_comm is not None given default is None?

It is a boolean, so can be set to False as well.

sadielbartholomew · 2018-04-18T16:04:37Z

tests/api-suite-info/05-get-latest-state-1.t

-    env no_proxy=* curl -A "${AGENT}" -v --cacert "${SRV_D}/ssl.cert" \
-    --digest -u "cylc:$(<"${SRV_D}/passphrase")" \
-    "https://${HOST}:${PORT}/get_latest_state"
+    cylc client -n --set-uuid="${UUID}" 'get_latest_state' "${SUITE_NAME}"


Could (eventually) create a function for a run_ok test of incremental name calling this exact command (as is done throughout this test file) after sleep 1 or other activity to make this test a bit more intuitive & tidy .

Agreed especially if we end up doing more of these.

And removal of some unnecessary logic.

matthewrmshin · 2018-04-19T10:29:00Z

All review issues fixed or addressed.

sadielbartholomew · 2018-04-19T11:21:47Z

Re-approving after looking at the one response commit. Advised to merge given trivial nature of changes in the commits subsequent to review from @hjoliver who is now on short-term leave.

Fix docstring typo in #2582

matthewrmshin added this to the next release milestone Feb 21, 2018

matthewrmshin self-assigned this Feb 21, 2018

matthewrmshin force-pushed the comm-ssh branch from e491e2a to c425e52 Compare February 21, 2018 12:31

matthewrmshin mentioned this pull request Feb 22, 2018

cylc message severity levels #2505

Closed

matthewrmshin requested review from hjoliver and oliver-sanders February 22, 2018 15:40

matthewrmshin force-pushed the comm-ssh branch from 56b0de5 to 60012c7 Compare February 28, 2018 09:19

matthewrmshin force-pushed the comm-ssh branch from 60012c7 to 14e8f63 Compare February 28, 2018 20:27

matthewrmshin force-pushed the comm-ssh branch 2 times, most recently from afadadb to 881762a Compare March 1, 2018 10:17

matthewrmshin force-pushed the comm-ssh branch from 881762a to f8e20b5 Compare March 1, 2018 21:47

oliver-sanders reviewed Mar 2, 2018

View reviewed changes

matthewrmshin force-pushed the comm-ssh branch from 7aaff6b to b256c30 Compare March 2, 2018 10:54

oliver-sanders reviewed Mar 6, 2018

View reviewed changes

matthewrmshin force-pushed the comm-ssh branch from b256c30 to 35e4373 Compare March 7, 2018 13:08

matthewrmshin force-pushed the comm-ssh branch from 2ad0161 to 11354f2 Compare March 20, 2018 13:58

matthewrmshin force-pushed the comm-ssh branch from 4003f05 to b1d5098 Compare March 21, 2018 12:54

hjoliver reviewed Mar 26, 2018

View reviewed changes

matthewrmshin force-pushed the comm-ssh branch from b1d5098 to 80b17bb Compare March 26, 2018 11:59

hjoliver reviewed Mar 27, 2018

View reviewed changes

hjoliver requested changes Mar 28, 2018

View reviewed changes

matthewrmshin mentioned this pull request Mar 28, 2018

cat-log: remote cylc sub-command #2503

Merged

hjoliver approved these changes Apr 3, 2018

View reviewed changes

matthewrmshin mentioned this pull request Apr 10, 2018

Use cylc subcommand for remote functionality #2302

Closed

5 tasks

matthewrmshin force-pushed the comm-ssh branch from 37a962e to 143670d Compare April 12, 2018 11:55

matthewrmshin added 5 commits April 17, 2018 08:59

Suite UUID

b37d403

Generate new suite UUID on normal run. Load previous value on restart. In a task job environment, check that the environment value of the suite UUID matches the one in the suite contact file. Don't send message on mismatch.

Update comment

6a3f0e4

Fix --ssh-cylc=/path/to/cylc/bin/cylc option parse

f3d6327

ssh comms: use remote_cylc_cmd to call cylc client

91f9b34

matthewrmshin force-pushed the comm-ssh branch from 143670d to 91f9b34 Compare April 17, 2018 13:29

matthewrmshin requested a review from sadielbartholomew April 17, 2018 13:30

sadielbartholomew mentioned this pull request Apr 17, 2018

Improve task polling and timeout handling #2593

Merged

sadielbartholomew reviewed Apr 17, 2018

View reviewed changes

Fix comment, bad and pointless exception handling

ff1561d

Fix copyright year. Fix doc for new commands. Fix bad exception handling. Remove pointless warning message.

sadielbartholomew approved these changes Apr 18, 2018

View reviewed changes

More comments and docs

4fef380

And removal of some unnecessary logic.

sadielbartholomew merged commit 732040b into cylc:master Apr 19, 2018

matthewrmshin deleted the comm-ssh branch April 19, 2018 11:22

sadielbartholomew mentioned this pull request Apr 19, 2018

Fix docstring typo in #2582 #2630

Merged

sadielbartholomew added a commit that referenced this pull request Apr 19, 2018

Merge pull request #2630 from sadielbartholomew/fix-docstring-typo

799207f

Fix docstring typo in #2582

matthewrmshin mentioned this pull request Apr 24, 2018

Improve contact2 locate logic #2637

Merged

matthewrmshin mentioned this pull request Oct 9, 2018

cylc > 7.5.0 cherrypy server.socket_host problems #2785

Closed

hjoliver mentioned this pull request Mar 19, 2020

Tidy Up Task Communications Methods #3511

Closed

Improve comm #2582

Improve comm #2582

Conversation

matthewrmshin commented Feb 21, 2018 • edited Loading

matthewrmshin commented Feb 21, 2018 • edited Loading

matthewrmshin commented Feb 22, 2018

matthewrmshin commented Feb 28, 2018

matthewrmshin commented Feb 28, 2018

matthewrmshin commented Mar 1, 2018

oliver-sanders left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewrmshin commented Mar 2, 2018

matthewrmshin commented Mar 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewrmshin commented Mar 20, 2018

hjoliver left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewrmshin commented Mar 26, 2018

matthewrmshin commented Mar 26, 2018

hjoliver commented Mar 26, 2018

hjoliver left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hjoliver left a comment

Choose a reason for hiding this comment

matthewrmshin commented Apr 17, 2018

sadielbartholomew left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewrmshin commented Apr 17, 2018

sadielbartholomew left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sadielbartholomew Apr 18, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewrmshin commented Apr 19, 2018

sadielbartholomew commented Apr 19, 2018

matthewrmshin commented Feb 21, 2018 •

edited

Loading

matthewrmshin commented Feb 21, 2018 •

edited

Loading

sadielbartholomew Apr 18, 2018 •

edited

Loading