WS-UIS data and GraphQL integration

[dwsutherland]

The sibling pull-request to the UI Server graphql protobuf feed:
cylc/cylc-flow#3122
And the Cylc8 UI Server component implementation of the GraphQL PoC.

There will be many more improvements/additions to be incorporated both in this and subsequent pull requests, however, for the sake of UI development and review this needs to be exposed.

The extra dependencies are:

    'pyzmq',
    'protobuf',
    'python-jose',
    'colorama',

(colorama due to the cylc scan CL call, which will be deprecated as workflow discovery is already integrated)

The instructions on installation are included in the README.md, and for this change cylc workflow library needs to be put into the python path (or venv site packages), I just source:

source ~/.envs/cylc-uis/bin/activate
source ~/.npm_env
export PYTHONPATH="${PYTHONPATH}:/home/sutherlander/repos/cylc8/lib"

With .npm_env being my local node.js environment:

# Node environment
export NPM_PACKAGES="${HOME}/.npm-packages"
export NODE_PATH="$NPM_PACKAGES/lib/node_modules:$NODE_PATH"
export PATH="$NPM_PACKAGES/bin:$PATH"

unset MANPATH
export MANPATH="$NPM_PACKAGES/share/man:$(manpath)"

There are three main files in this change:
schema.py - GraphQL data structure and types definition (uses graphene)
workflow_services_mgr.py - Workflow discovery, REQ client established (and passed to data manager), and spawning (to come).
data_mgr.py - Uses established client/workflows to retrieve and process data structure (decode protobuf), and contains access & filtering methods used by the resolvers.

In order to pass in the uiserver/data_manager as context to the resolver's I had to subclass the graphene-tornado GraphQL handler:

class UIServerGraphQLHandler(TornadoGraphQLHandler):

    #Extra attributes
    uiserver = None

    def initialize(self, **kwargs):
        super(TornadoGraphQLHandler, self).initialize()
        for key, value in kwargs.items():
            if hasattr(self, key):
                if key == 'middleware' and value is not None:
                    setattr(self, key,
                        list(self.instantiate_middleware(value)))
                else:
                    setattr(self, key, value)

    @property
    def context(self):
        wider_context = {
            'uiserver': self.uiserver,
            'request': self.request,
        }
        return wider_context

The data update method is very crude at the moment; it pulls the full workflow data msg every 5 secs (which should be faster than CLI $ cylc client --no-input baz pb_entire_workflow as the client is already established by the WS manager)... This message is decoded/parsed as a protobuf object (class instance) and stored in a dictionary with "owner/workflow" as a key.
One of the good features of the UI Server holding/maintaining a data structure is, without needing multiple requests, we can query and filter across workflows, for example:
query

query workflows($tIds: [ID]){
  workflows(ids: ["baz", "jin"]){
    id
    name
    status
    stateTotals {
      runahead
      waiting
      held
      queued
      expired
      ready
      submitFailed
      submitRetrying
      submitted
      retrying
      running
      failed
      succeeded
    }
    tasks(ids: $tIds){
      name
      meta {
        title
        description
        URL
        userDefined
      }
      proxies(ids: $tIds){
        cyclePoint
        state
      }
    }
  }
}

variables

{
  "tIds": ["foo", "baz"]
}

results

{
  "data": {
    "workflows": [
      {
        "id": "sutherlander/baz",
        "name": "baz",
        "status": "held",
        "stateTotals": {
          "runahead": 0,
          "waiting": 0,
          "held": 5,
          "queued": 0,
          "expired": 0,
          "ready": 0,
          "submitFailed": 0,
          "submitRetrying": 0,
          "submitted": 0,
          "retrying": 0,
          "running": 0,
          "failed": 0,
          "succeeded": 19
        },
        "tasks": [
          {
            "name": "foo",
            "meta": {
              "title": "Some Top family",
              "description": "some task foo",
              "URL": "https://github.com/dwsutherland/cylc",
              "userDefined": [
                "importance=Critical",
                "alerts=none"
              ]
            },
            "proxies": [
              {
                "cyclePoint": "20170301T0000+12",
                "state": "succeeded"
              },
              {
                "cyclePoint": "20170401T0000+12",
                "state": "succeeded"
              },
              {
                "cyclePoint": "20170501T0000+12",
                "state": "held"
              }
            ]
          }
        ]
      },
      {
        "id": "sutherlander/jin",
        "name": "jin",
        "status": "held",
        "stateTotals": {
          "runahead": 0,
          "waiting": 0,
          "held": 1,
          "queued": 0,
          "expired": 0,
          "ready": 0,
          "submitFailed": 0,
          "submitRetrying": 0,
          "submitted": 0,
          "retrying": 0,
          "running": 0,
          "failed": 0,
          "succeeded": 0
        },
        "tasks": [
          {
            "name": "baz",
            "meta": {
              "title": "",
              "description": "",
              "URL": "",
              "userDefined": []
            },
            "proxies": [
              {
                "cyclePoint": "1",
                "state": "held"
              }
            ]
          }
        ]
      }
    ]
  }
}

This applies to all fields (from root query), i.e. tasks, jobs, familes, edges... This could open up some possibilities for us.. (And obviously gives us the data for the "gscan" type web view)

A lot more to do, but this is a start 😃

This Pull Request

• Workflow services discovery, management, ZeroMQ REQ/RES client
• Add Data management (protobuf message request & processing) and access methods
• Write/Integrate PoC graphql schema to represent UIS context.
• Query types functioning.
• Mutations types functioning.
• ? Perhaps some of the below additional

Additional (to come or future PR?)

• Incremental updates to data structure using deltas and checksum skeleton.
• Deltas received via PUB/SUB from Workflow Services UI Server and checksum skeleton data management.
• Tornado WebSockets and Subscription server implementation
• Pagination
• Workflow services spawning
• Testing

[dwsutherland]

Travis will likely fail, as cylc/cylc-flow is not a requirement yet.

[kinow]

Travis will likely fail, as cylc/cylc-flow is not a requirement yet.

Indeed it failed with: ModuleNotFoundError: No module named 'cylc'

In this case I think once everybody is back from holidays, perhaps we could prioritize cylc/cylc-flow#2990. That one has become a blocker for a few issues now.

[kinow]

Came here to comment on setup.py but checking out the code in the IDE a few issues popped in the warnings panel 👍

[codecov-io]

Codecov Report

Merging #34 into master will decrease coverage by 7.32%.
The diff coverage is 40.62%.

@@            Coverage Diff             @@
##           master      #34      +/-   ##
==========================================
- Coverage   68.47%   61.15%   -7.33%     
==========================================
  Files           2        2              
  Lines          92      121      +29     
  Branches        6        9       +3     
==========================================
+ Hits           63       74      +11     
- Misses         29       47      +18

Impacted Files	Coverage Δ
handlers.py	57.53% <27.27%> (-11.7%)	⬇️
cylc_singleuser.py	66.66% <70%> (-0.84%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 80b4027...3575acb. Read the comment docs.

[kinow]

@dwsutherland see my last commit with the temporary fix for this build. Now Travis is happy again.

[matthewrmshin]

Had a quick look by 👀 and 👓. Code looks OK in general. A few comments on style. For readability, I prefer explicit for loops to list comprehensions when the logic does not fit in a line - but that is a personal preference that you can feel free to ignore.

[sadielbartholomew]

I have not yet had a chance to try this out, as it is not trivial to set-up the environment given difficulty in getting certain installation requirements e.g. npm & jupyterhub from inside the Met Office, but I've had a scan through the code.

It looks good in general, but I have a few minor observations & questions.

[oliver-sanders]

This is pretty exciting! Need to get it set up and running. Request structure looks good!

I'd kinda envisaged the GraphQL layer being in cylc-flow itself rather than in the UI-Server, this way could work fine too but we will need to have a quick think about:

• How the UI-Server will work when connecting to workflows of different versions.
• How to make this work with (API-On-The-Fly)[https://github.com/Suite API - On-The-Fly cylc-flow#2123].

I guess here we have a GraphQL-ZMQ adapter layer so it's about ensuring that the adapter isn't brittle to changes.

[dwsutherland]

@oliver-sanders

I'd kinda envisaged the GraphQL layer being in cylc-flow itself rather than in the UI-Server,

Well we could have both, the data structure is present at both. They would be different though, because the cylc-flow one wouldn't be aware of other workflows. If we only had one GraphQL layer at the cylc-flow end then:

• There would be no point in maintaining a data structure at the UI-Server (which we kinda agreed on, and your ZeroMQ prototype addressed the update of this).
• We would lose the resolver ability to filter over data elements (flows, tasks, families, jobs, edges ...etc) from multiple workflows (without a huge mass of ZMQ requests routed according to the query args).
• Apart from the type checking on assignment, protobuf would be pointless.
• The demand of the web-client will directly effect the workflows.
• GraphQL Subscription haven't been developed for ZeroMQ PUB/SUB, not sure how much work this would be.
• ?

this way could work fine too but we will need to have a quick think about:

• How the UI-Server will work when connecting to workflows of different versions.

Potential conflicts when:

• Protobuf message definition changes. Although the fields are optional and additions etc are fine. If other cylc-flow code changes, then protobuf field assignment and/or type checking will raise an exception.
• The cylc-flow API changes in some way between versions..

I guess here we have a GraphQL-ZMQ adapter layer so it's about ensuring that the adapter isn't brittle to changes.

Exactly, protobuf helps alot (it was designed with these issues in mind!). But I can imagine similar issues even with only a cylc-flow GraphQL, some of which would be passed on to the web-client instead.

• How to make this work with (API-On-The-Fly)[https://github.com/cylc/cylc-flow/issues/2123].

Will need to read up again, but GraphQL has introspection which can return all the information about the schema (including descriptions, which I need to do more of!), as shown in the graphiql interface.

[oliver-sanders]

I'm struggling quite a bit with this one trying to see how it can fit into my past visions of decoupling Cylc components. I've tried and kinda failed to outline some of it here, I might make more sense of things later...

The issue I'm trying to get my head around is this:

• Cylc-UI is tightly coupled to Cylc-UIServer by the GraphQL interface
• Cylc-UIServer is tightly coupled to Cylc-Flow by the ZMQ interface
• Therefore the Cylc-UI is tightly coupled to Cylc-Flow.

I don't know what the best way to decouple things is, it was all very clear to me once but things have evolved so far since then I can't see it any more. Back when things were simpler and we were talking about one workflow talking to one GUI (in Electron) it was easy to see how decoupling the Cylc Flow and Cylc UI versions might work. The GUI can just request the API from the workflow and present it in a nice form for the user (e.g. the stop suite panel) and GraphQL can handle most upgrade requirements simply enough.

Now we have one UIServer talking to multiple workflows things are more difficult as by defining the GraphQL schema at the UIServer we are defining the interface for all suites irrespective of their version. If we make changes to a Cylc suite (which we have form of) we will have to put a lot of adapter logic into the UIServer. GraphQL (and presumably ProtoBuff) can protect us from the most basic examples (fields added / removed) but the rest...

I'm struggling to see a way forward at the moment, @dwsutherland could you help me out by outlining how you envisage us handling some example API changes e.g. (adding fields, changing arguments in ZMQ endpoints, etc)?

Further thoughts on upgrading Cylc components: (a quick observation with no point to make really)

Cylc-UI is instantaneously upgradeable, that is to say you can change versions at the press of a button or just by refreshing the page. Handling change here is easy.

The Cylc-UIServer is much harder to upgrade, we have to wait for users to close their browser windows for the UIServer to become inactive for it to auto-shutdown (or have some sort of force upgrade system).

So we probably want the dynamic components isolated in Cylc-UI where they are easy to change?

[matthewrmshin]

Is it possible to define multiple workflow API data models under a single GraphQL schema? (With luck, we'll only have to back support a small number of workflow server APIs for each UI/UI-Server version. So even if we have to spell out all the data models explicitly for the supported workflow API data models, it should not be a big deal. But the top level data model must be able to handle this.)

For upgrading a ui-server... I suppose it should be possible for use to configure the hub to tell all ui-servers running outdated software to shutdown and restart with the current software version. Browsers sessions can reconnect automatically. This does not sound any worse than what Outlook (web) does to my sessions everyday.

[dwsutherland]

@oliver-sanders: Thanks for bring this up, yes there will be ways to break this PR data model..
I was a little bit taken back, not that it matters, as we had discussed this design before I implemented it (obviously not in enough detail!).

I guess before we posit any solutions, we need to describe problems that are unique to the current proposed data model...

The issue I'm trying to get my head around is this:

• Cylc-UI is tightly coupled to Cylc-UIServer by the GraphQL interface
• Cylc-UIServer is tightly coupled to Cylc-Flow by the ZMQ interface
• Therefore the Cylc-UI is tightly coupled to Cylc-Flow.

Firstly I'm not exactly sure what you mean by tightly, because I don't really see how

Back when things were simpler and we were talking about one workflow talking to one GUI (in Electron) it was easy to see how decoupling the Cylc Flow and Cylc UI versions might work. The GUI can just request the API from the workflow and present it in a nice form for the user (e.g. the stop suite panel) and GraphQL can handle most upgrade requirements simply enough.

doesn't apply to UIServer => GUI? ... To the GUI it's just a GraphQL endpoint in both designs, and I would have thought that in both cases the GUI would potentially be different repositories/versions than the API they connect to.

Cylc-UI is instantaneously upgradeable, that is to say you can change versions at the press of a button or just by refreshing the page. Handling change here is easy.

The UIServer, of course, can ensure the version of GUI/cylc-ui is compatible on start up
as it has to serve the HTML via Tornado, so those versions are always in sync and like you said users can just refresh the page...

The Cylc-UIServer is much harder to upgrade, we have to wait for users to close their browser windows for the UIServer to become inactive for it to auto-shutdown (or have some sort of force upgrade system).

Maybe it's just Dev/Debug mode.. But during development you can have the UIServer running and change the source code, Tornado must reload or something because I never have to shut my browser down.
Even if we did have to shut the UIServer down and/or restart JupyterHub, refreshing the browser usually just restarts the UIServer and/or reloads the page..

Now we have one UIServer talking to multiple workflows things are more difficult as by defining the GraphQL schema at the UIServer we are defining the interface for all suites irrespective of their version.
If we make changes to a Cylc suite (which we have form of) we will have to put a lot of adapter logic into the UIServer.

This setup is arguably how it should be! Why should the frontend have to deal with Workflow-GraphQL version inconsistencies in the backend?
Let me explain...

The UIServer will always be the one talking to all the user's workflows, because (and correct me if I'm wrong):

• JupyterHub always redirects the signed in user to the UIServer
• The UIServer will monitor and have the ability to spawn the workflows.
• ZeroMQ doesn't serve HTML and setup URLs, so there needs to be some translation between browser-proxy-jupyterhub and the workflow

We could still have graphql at the workflow, however:

• Even though queries/mutations could be sent as strings through ZeroMQ, there is no support/tools for it (i.e. no GraphiQL interface, we would have to cannibalize the graphene-tornado code for the interface)
• How do GraphQL Subscription Types work in ZeroMQ? We would have to write our own tool for that (not the same as ZeroMQ PUB/SUB even if it rides it).
• I think the ApolloGraphQL client expects to hook straight into a GraphQL endpoint for introspection (and possibly more...)
• How do you traverse across workflows in a single query? You couldn't really do it without routing your queries to every workflow
• We still have the issue with running multiple workflow versions of different graphql APIs (perhaps the queries differ, or input args or w/e), and all this does is pass a backend problem to the frontend developer...

BTW - GraphQL is designed/often-used to take multiple input sources of different types (SQLDB, ProtoBuf, REST, MongoDB ...etc) and provide a coherent data access point.
(also, I think protobuf is arguably a better data format than JSON for transport over ZeroMQ)

So, I think we can narrow down the question to: How do we envisage Cylc-Flow <=> Cylc-UIServer handling some example API changes e.g. (adding fields, changing arguments in ZMQ endpoints, etc)?

GraphQL (and presumably ProtoBuff) can protect us from the most basic examples (fields added / removed) but the rest...

Yes, ProtoBuf can easily handle adding more and more fields and I've made all the fields in the .proto definition optional (which is default and recommended) so you can remove them (within reason) as you please..
In fact the fields are only sent if they are populated, which means if we wanted then with incremental updates we could just populate the same protobuf objects with only the updated fields (i.e taskproxy status ...etc)...
This also means I don't do any filtering at the workflow, only the full protobuf objects are requested. This means we can design ZeroMQ endpoints with very few arguments, and those with them should have defaults (i.e. an argument might be a list of TaskProxy IDs found missing on checksum comparison, with the default sending all TaskProxy protobuf messages).

Leaving you with the question: How the hell will the UIServer handle this?
I set defaults in the fields of the GraphQL schema, so if they are missing in the input data it returns that.. And in fact; if no default is set and required is not set, then graphql will return null.

If there are other breaking changes, then either the workflows or uiserver will have to contain back compatibility
As @matthewrmshin suggested; we could have the UIServer restart the workflow (by default or, with option in .cylc/global.rc, ignore suites with incompatible API version), and require the identity endpoint to always be there and always return the API version..

Just as a side note, we had the same issue with the REST Cherrypy API, where we had to duplicate the URLs with /id (or something) as a prefix for back compatibility... So there's always going to be something

Also important to remember; it's only differing APIs that may cause issues, workflows can differ in version all they want otherwise..

Please call me out if I'm missing something or plain wrong!

[oliver-sanders]

OK, it looks like I'm the only one who's concerned so should probably push ahead whatever. Perhaps discuss at the next video meet.

Something else to think about which we came up with at a UK team meet was that we want out CLI client to have the ability to route via the UI server. I guess the UI server could just act as a TCP-Websocket buffer?

[dwsutherland]

Something else to think about which we came up with at a UK team meet was that we want out CLI client to have the ability to route via the UI server. I guess the UI server could just act as a TCP-Websocket buffer?

Hmm... You need both, so how would the CLI discover a running UIServer?

At the moment, I import the workflow runtime client (from network/client.py ) into the UIServer and instantiate for every discovered workflow (on startup and refresh every 19sec (arbitrary))..

Perhaps for uniformity we could put a graphql schema the workflow, and import some of this to the graphql schema from there into the UIServer... But it would still have to be designed to handle multiple workflows (defaulting to itself when running in a single workflow)...

The UIServer is running with Tornado (which will be TCP-Websocket at some point, but for browser interaction), perharps we could start up a ZeroMQ stub in addition....
(I did see the use of Tornado in the pyzmq example folder)

[kinow]

Hey @dwsutherland , any chance of including unit tests for the non-web related code? i.e. no need to bother with the Tornado handlers being tested.

Resolved, re-requested.

[kinow]

@dwsutherland with the following query

query {
  tasks {
    id, name, meta {
      title
      description
      URL
    }, meanElapsedTime, namespace, proxies {
      id
      task {
        id
      }
      state
      cyclePoint
      spawned
      depth
      jobSubmits
      latestMessage
      outputs
      prerequisites {
        expression
        satisfied
      }
      jobs {
        id
      }
      parents {
        id
      }
    }, depth
  }
}

am getting the error

  "errors": [
    {
      "message": "Unknown server error"
    }
  ]
}

The console running jupyterhub with cylc-uiserver has:

ERROR:tornado.application:Error: Object of type RepeatedScalarContainer is not JSON serializable Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/graphene_tornado/tornado_graphql_handler.py", line 95, in post
    yield self.run('post')
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
    value = future.result()
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/gen.py", line 736, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/graphene_tornado/tornado_graphql_handler.py", line 112, in run
    result, status_code = yield self.get_response(data, method, show_graphiql)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
    value = future.result()
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.send(value)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/graphene_tornado/tornado_graphql_handler.py", line 208, in get_response
    result = self.json_encode(response, pretty=self.pretty or show_graphiql)
  File "/home/kinow/Development/python/workspace/cylc-uiserver/venv/lib/python3.7/site-packages/graphene_tornado/tornado_graphql_handler.py", line 272, in json_encode
    return json.dumps(d, separators=(',', ':'))
  File "/home/kinow/anaconda3/lib/python3.7/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/home/kinow/anaconda3/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/kinow/anaconda3/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/kinow/anaconda3/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type RepeatedScalarContainer is not JSON serializable

ERROR:tornado.access:500 POST /user/kinow/graphql/graphiql? (127.0.0.1) 13.41ms

Is it something you are aware that could happen? I built the query in the browser, using GraphiQL's auto complete feature.

Thanks!

[kinow]

Appears to be the outputs field that is causing the error on the server side.

[kinow]

Another dummy user support question @dwsutherland (sorry!). At the moment we have 2 REST calls to populate the Suite Tasks.

One to retrieve the current Suite, then one more to retrieve the Tasks in the suite. But as we are using GraphQL, I decided to try and use a single query now too.

query ($name: String!) {
  workflows (name: $name) {
    id 
    name 
    owner 
    host 
    port 
    tasks {
      id, name, meta {
        title
        description
        URL
      }, meanElapsedTime, namespace, proxies {
        id
        task {
          id
        }
        state
        cyclePoint
        spawned
        depth
        jobSubmits
        latestMessage
      }, depth
    }
  }
}

However, I am not able to pass the name variable. It fails with

{
  "errors": [
    {
      "message": "Unknown argument \"name\" on field \"workflows\" of type \"Query\".",
      "locations": [
        {
          "line": 2,
          "column": 14
        }
      ]
    }
  ]
}

Just wondering if there is anything special that needs to be done on the Resolver to accept variables? Or am I able to use a variable for any field in the Query?

Thanks!

[dwsutherland]

@kinow

query ($name: String!) {
  workflows (name: $name) {
    id 
    name 
    owner 
    host 
    port 
    tasks {
      id, name, meta {
        title
        description
        URL
      }, meanElapsedTime, namespace, proxies {
        id
        task {
          id
        }
        state
        cyclePoint
        spawned
        depth
        jobSubmits
        latestMessage
      }, depth
    }
  }
}

Two things:

a) Like the error suggests. there is no argument name, you could use workflows (ids: [$name]).. as the ID owner/name:state argument will work for globs or just name. (this is the cause of the error)
b) workflows{tasks{proxies{task}}}} is circular, so no point in getting task from within tasks.

With cylc-ui use /graphql not /graphql/graphiql

I'll try the other error you presented.. I often get browser cache issues when restarting the ui-server..

[dwsutherland]

outputs bug fixed.

[kinow]

Thanks a lot @dwsutherland ! Will update the code with your suggestions and rebase the code to try it again.

[dwsutherland]

Mutations added last week. Some examples:

---------------------------------------------------

mutation {
  stopWorkflow(
      command: "set_stop_after_clock_time",
      workflows: ["*/baz"],
      args: {datetimeString: "2019-05-30T21:24:06+12:00"}) {
    result
  }
}

---------------------------------------------------

mutation{
  holdWorkflow(
      command: "hold_after_point_string",
      workflows: ["baz"],
      pointString: "20171201T0000+12") {
    result
  }
}

mutation {
  releaseWorkflow(workflows: ["baz"]){
    result
  }
}

---------------------------------------------------

mutation{
  putMessages(
      workflows: ["baz"],
      ids: ["20170101T0000+12/foo/01"],
      messages: [["CRITICAL", "HELLO, IS THERE ANYONE OUT THERE?"]]) {
    result
  }
}

---------------------------------------------------

mutation {
  taskActions(
      command: "insert_tasks",
      workflows: ["baz"],
      ids: ["20180501T0000+12/foo"]) {
    result
  }
}

mutation {
  taskActions(
      command: "trigger_tasks",
      workflows: ["baz"],
      ids: ["2017*/foo:failed"]) {
    result
  }
}

---------------------------------------------------

mutation{
  putBroadcast(
    workflows: ["baz"],
    namespaces: ["foo"],
    settings: [{environment: {DANGER: "Dr Robertson"}}]) {
    result
  }
}

mutation{
  clearBroadcast(
      workflows: ["baz"],
      namespaces: ["foo"],
      cancelSettings: [{environment: {DANGER: "Dr Robertson"}}]) {
    result
  }
}

[matthewrmshin]

HI @dwsutherland We are going to try to crowd review and merge this and cylc/cylc-flow#3122 while @hjoliver and @kinow are in UK. Please resolve the conflict of this branch when you have the time to do so.

[dwsutherland]

HI @dwsutherland We are going to try to crowd review and merge this and cylc/cylc-flow#3122 while @hjoliver and @kinow are in UK. Please resolve the conflict of this branch when you have the time to do so.

Done.

I've got a few important changes inbound for both PR, will try and get them in before.

[dwsutherland]

Travis failing with:

Collecting cylc-flow==8.* (from cylc-uiserver==1.0)
  Could not find a version that satisfies the requirement cylc-flow==8.* (from cylc-uiserver==1.0) (from versions: 8.0a0)
No matching distribution found for cylc-flow==8.* (from cylc-uiserver==1.0)
The command "pip install .[all]" failed and exited with 1 during .

@kinow confirmed it will be like this until both PRs are merged.. (right?)

[dwsutherland]

Ok pushed, however, Travis may still break as it imports cylc-flow modules not yet in master..

[hjoliver]

MERGING! 💥 thanks @dwsutherland 💐 We'll put up a bunch of Issues for urgent follow-up...

[dwsutherland]

🍻 Thanks everyone for your input and critique!

[kinow]

👏 👏 👏 Here's to you @dwsutherland 🍻 ! Well done!