One GUI using technology for web applications

[hjoliver]

Depends on #1872 DONE.

[hjoliver]

[meeting]

• discuss this after the new API is in
• (various kinds of "web based GUI"- can refer to the comms under the hood, thru to in-browser...)

[matthewrmshin]

@cylc/core I believe we have agreed that web based GUI is the way forward. In which case, this issue should supersede #112, #570 and #885.

[matthewrmshin]

We should design the new GUI to be a pluggable container for the functionality of gscan + #1302, gcylc, Rose Bush, #1683, etc.

[matthewrmshin]

To avoid confusion, the main idea currently is to write a new GUI using new technology (in particular technology for building web applications). It does not necessarily mean that we want the new GUI to be a web application.

[oliver-sanders]

To attach a few related matters discussed but not yet captured...

Graph View For Large Suites

For suites with large numbers of tasks the graph view becomes unusable. One way to get around this would be to have the graph view display a configurable number of levels from the currently selected task. I.E. if a user clicked on a task in the treeview the graphview would display that task and n levels of connections to it.

This would supersede #1086.

[update: see also #2566, which expresses this problem slightly differently]

Suite List

With a new plugable architecture gscan can be used as a sidebar to gcylc allowing it to replace the current open another suite feature. Along with changes planed for gscan this would mean that gcylc would display all registered suites with running suites at the top. This should alleviate most of the requirement for having multiple gcylc windows open as one can quickly change from viewing one suite to another. The gui would be isolated from most changes in cylc versions by #2123.

Portability

The gui should not be deeply written into a windowing framework which prevents if from being rendered in a browser so as to leave scope for future extensions in functionality.

Data Driven

Using a data driven approach (e.g. d3.js) would simplify things greatly and allow a single updater to drive all of the GUI components with minimal fuss.

The use of graphql or a similar framework would offer better flexibility in suite communication enabling us to dynamically adjust what information is requested to meet the requirements of GUI components whilst minimising http requests, see cylc/cylc-flow#2123 (comment).

[matthewrmshin]

https://electron.atom.io/ looks like the most attractive framework right now.

[hjoliver]

Graph

layout

• visjs - http://visjs.org/network_examples.html
• d3.js - force-directed graphs: https://github.com/d3/d3/wiki/Gallery
• dagre - https://github.com/dagrejs - see its github wiki page
• cytoscape - http://js.cytoscape.org/

other:

• graph construction: http://bl.ocks.org/rkirsling/5001347

[hjoliver]

(A minor feature request that I've punted to the new GUI): We should be able to annotate tasks with warning indicators (and maybe other things) - perhaps via a column in the tree view and badges in the graph view.

[hjoliver]

WebSocket?
https://en.wikipedia.org/wiki/WebSocket

The WebSocket protocol enables interaction between a web client (such as a browser) and a web server with lower overheads, facilitating real-time data transfer from and to the server. This is made possible by providing a standardized way for the server to send content to the client without being first requested by the client, and allowing messages to be passed back and forth while keeping the connection open. In this way, a two-way ongoing conversation can take place between the client and the server.

[hjoliver]

Hybrid apps:
https://slack.engineering/interops-labyrinth-sharing-code-between-web-electron-apps-f9474d62eccc

[amckinstry]

Hi, Is there a timeline / milestone for this issue ? pygtk is being removed from Debian/Ubuntu, which means as it stands cylc will ship without a GUI.
The current timeline is release of Ubuntu ~October (freeze Sept) and Debian 10 freeze ~Feb 2019.

Note that if its agreed what new frameworks will be used (electron?), then that would be good to know, as I can make sure dependencies are present before then.

[hjoliver]

@amckinstry - yikes, that's tight, thanks for the heads-up. The core developers are to meet in person in 1-2 weeks time, we'll see if we can chart the way forward...

[oliver-sanders]

Comments on the benefits of using GraphQL or an equivalent system: cylc/cylc-flow#2563 (comment)

[oliver-sanders]

Finally got around to mocking up most of what we discussed in June.

According to tradition the following [rough] sketches are in aqua, windowing systems may vary!

The One Cylc GUI

A combination of gcylc and gscan with suites listed down the left and suite visualisations on the right.

• Nested suites (suites with / chars in their reg) displayed hierarchically to facilitate the management of large numbers of suites.
• Stopped suites displayed beneath running ones.
• GUI "view" switch at the top of the screen (this should be able to follow the context of the selected task where possible e.g. view => review).
  • Configure = rose edit
  • View = cylc gui
  • Review = rose bush
  • Other possibilities:
    • Preview = cylc graph
    • Analyse (the long anticipated GUI providing gant charts, timings, shortest path to result analysis, etc)
• A mobile-style search bar located at the top of the gscan component for filtering suites by name or [meta] would probably be a good idea, especially if we are going to display stopped suites.

Mobile Friendly Callouts

Context menus provided on "tap" or "left click" to leave open the potential for mobile friendliness.

Sub-contexts provided as "tokens" where possible (or at lest most commonly used sub-contexts) to avoid nested contexts.

GUI Tools Displayed In Context

Tools specific to a particular view displayed in a "hover-over" transparency on "capable" platforms rather than an obstructive extra toolbar.

Rightmost icon is a depth selector enabled automatically for large graphs, it determines the depth of nodes / edges to display as measured from the selected node, enabling responsive display of large graphs.

Alternative Node Styling

Web-tech opens up a lot of opportunities as regards node styling (gradients, patterns, etc).

One simple option for denoting node stays without the use of colour displayed below (task failure could be represented with a cross in a circle). This leaves colour free to be defined by the user (using the current visualisation settings) or for displaying suites as collections of sub-systems al la (https://github.com/cylc/cylc/wiki/Possible-Cylc-Futures#sub-suites).

In the above example bool, pub and nix all belong to one sub-system (or family) and the remaining tasks to another.

[hjoliver]

Suite Discovery (for running suites) no longer needed under the new web architecture?

If all suite server programs know (via global config) where the reverse proxy is, they can initiate communications with the proxy instead of the reverse. Subsequently, the reverse proxy knows where each suite is via existing persistent WebSocket connections and/or from host:port values stored from previous connections (which can then be verified or dropped after attempting to connect again).

I suppose we might still want suites to store host:port in their .service/contact files (as info for users) but the reverse proxy will not need read them from the filesystem (if the previous paragraph is correct).

[hjoliver]

The reverse proxy will still need a discovery service for suites that are not running, to allow uses to start them up via the web interface.

[UPDATE] - actually, now that we need a "central" system component (the reverse proxy) we might as well modify cylc register to update a central registration DB (via the proxy, or not) with installed suite locations, so that the proxy does not need to scan the filesystem for not-running suites.

In which case, suite discovery just got a whole lot easier.

[hjoliver]

Sticky force layout - good for manual exploration of workflow structure? https://bl.ocks.org/mbostock/3750558

[oliver-sanders]

I've not had the time to investigate graphing properly, I've set up a trial with a couple of graphing systems using our current global suite as a performance test. Probably after Brussels I'll plug in some of the other systems and do some profiling (with comparisons to graphviz).

Quick conclusions thus far:

• The difficult bit is the layout, rendering thousands of nodes is quick, arranging them in a coherent hierarchy takes much longer.
• Not all hierarchies are as coherent as what we currently have in Cylc (when edges span multiple levels graphviz moves intermediate nodes to intermediate levels, one system I've looked at doesn't do this).
• The initial rendering might be slow but once performed interaction is fast. Hopefully adding / removing nodes won't require a complete redraw.
• Force directed layouts probably aren't much use as there is no axis of time/progress through the workflow?

[oliver-sanders]

Latest brain dump. Some ideas on how to separate tasks and jobs, rough sketches not polished proposals.

Plenty more to follow...

[hjoliver]

Force directed layouts probably aren't much use as there is no axis of time/progress through the workflow?

You're probably right, but maybe we could add an additional linear "gravity" that increases with cycle point, to impose some structure.

[hjoliver]

@oliver-sanders - regarding graph layout performance, yes I doubt any system can do very large graphs quickly enough for the control GUI, so we will need to aim for the restricted view (as previously discussed) n-levels out from user-selected tasks of interest. We can play with a wider variety of graphing tools for static workflow visualisation though.

[oliver-sanders]

we could add an additional linear "gravity" that increases with cycle point, to impose some structure.

ooh, interesting

[oliver-sanders]

Sharing selcted nodes beteen views & adjacency matrix (#2753)

On second thoughts this looks horrible, there is a much nicer way to represent this I'll come back to it later...

Ok, slightly nicer, still a work in progress. The cycles make it more complicted than it needs to be.

[oliver-sanders]

Also introducing the gant chart:

• In live mode view projcted workflow timings.
• In anlysis mode view averaged timings for resource planning.

would probably need to run left to right and collapse nodes for real suites.

[sadielbartholomew]

I've tried to collate thoughts on considerations for the UI into a few mind maps. Posting them here for ease of reference in discussions this week. Apologies for going low-tech!

[sadielbartholomew]

Here are photos of the schematics drawn up today, as further reference material (unfortunately at a slight angle but should be mostly legible by zooming in if necessary).

[hjoliver]

d3 force-directed graph layout Vue component:

• demo https://emiliorizzo.github.io/vue-d3-network/
  • (options to turn on node and edge labels etc.)
• repo https://github.com/emiliorizzo/vue-d3-network

[hjoliver]

Vue collapsing/expanding tree view from a nested tree data structure: https://vuejs.org/v2/examples/tree-view.html

[hjoliver]

A Vue component to draw directed acyclic graphs in the browser leveraging dagre-d3v4 (https://github.com/smbolton/dagre-d3v4) - https://github.com/vdidonato/vue-dagre-d3v4 - seems to be broken.

[oliver-sanders]

I'm currently looking at Javascript libraries for graphing, I'll update this issue with conclusions as I investigate. I'll probably put up my experiments as a GitHub repo when more complete.

Graphing consists of two components:

The Layout Engine

The bit which arranges nodes / edges spatially. See the layout engines supported by cytoscape for examples.

• Directed hierarchical graphs are hard
  • Layout is slow, there isn't a whole lot of choice when it comes to layout engines. Dagre is the standout but is a fairly stagnant project.
• More arbitrary layouts are easy
  • We might want to automatically swap to a less ideal layout engine for larger graphs.

Requirements

1. Directed & hierarchical
2. Must handle edges spanning multiple levels
3. Width of node considered in graph (i.e. nodes cannot overlap)
4. Speed!

Results

Engine	1	2	3	4	Supported Engines
Dagre	🆗	🆗	🆗	4.1s	cytoscape, d3
Klay	🆗	⛅	🚫	6.4s	cytoscape, d3
D3-DAG	🆗	🆗	🚫 [2]	>> 60s [3]	None? [4]

[1] Benchmark using the first cycle of MO global opp suite
[2] Possibly using the separation function though I can't get this to work
[3] Tested "Vert", "Greedy" and "Spread" layout methods
[4] Visualisation constructed by user provided functions, i.e. user provides the engine

The Graph Engine

The bit which renders the graph, provides an interface for manipulating and interacting with it. This is the component we will end up working with / extending to form the graph view.

Requirements

1. Mobile friendliness
2. Hackability of the graph:
   • Job status indicators (require more than just text labels)
   • Sub-workflows?
   • Advanced collapsing (e.g. parametrisations using an ellipsis symbol)?
3. Not a massive RAM hog
4. Support for the desired layout engine(s)
5. Node / family collapsing

Results

Engine	1	2	3	4	5
cytoscape	🆗	⛔ ?		Dagre, Klay	🆗
d3				Dagre, Klay
vis.js
arbor

Hybrids

Systems where the graph engine and layout engine are integrated:

Results

Engine	L1	L2	L3	L4	G1	G2	G3	G4	G5
viz.js	🆗	🆗	🆗					🆗

[hjoliver]

Interactive arbitrary grouping: select a bunch of nodes (individually, or by drawing a rectangle around them) and group them into a single node.

[hjoliver]

Layout algorithms for visualizing directed acyclic graphs - https://github.com/erikbrinkman/d3-dag - new-ish? demo https://jsfiddle.net/ye2xanf9/12/

[kinow]

Digressing a bit, but maybe we could choose more than one library for visualizing the graphs in Cylc?

I believe the new Web GUI will have authentication, and user management. If we add to the list of requirements, to have system settings and/or user settings (for things like language, locale, etc), then we could an option to choose the graph visualisation algorithm/library.

With that, if one implementation works very well for small-mid graphs and provides a lot of useful features, but perform poorly with large graphs, maybe users would still be able to use that implementation if they only had small/mid graphs.

Furthermore, that way we could also release new versions with different implementations for testing, and get feedback of users without having to break their current environments.

[oliver-sanders]

Two pieces of user feedback on UI:

• Make the different modes of "cylc stop" clearer from the GUI. I think the "cylc stop" options in the current "gscan" are a start, we could do with some lightweight documentation to re-enforce this.
• Ensure the page title is easily identifiable with the Cylc GUI to help avoid tabs getting buried. A favicon would help though would easily become confused with the Cylc GitHub pages site so we would want to use a variant to make this clear.

[matthewrmshin]

Make the different modes of "cylc stop" clearer from the GUI. I think the "cylc stop" options in the current "gscan" are a start, we could do with some lightweight documentation to re-enforce this.

If the suite is unable to stop intermediately, the UI should display some information on what the suite is waiting for. (It may be waiting for submitted/running tasks, kill/poll commands, event handlers, etc.)

[oliver-sanders]

From conversations in #2842:

• Support "themeing" for the new GUI
• Provide small number of built-in themes?
  • Light
  • Dark
  • Accessible
• Try to integrate accessibility checks into the GUI test battery (see https://www.w3.org/WAI/ER/tools/)
• Store user preferences in:
  • Browser cookies (obviously not portable across browsers)
  • Some centralised "preferences service" (not portable across cylc suite "services" e.g. local "out of the box", research suite service, production suite service)
  • Other?

[wxtim]

We have a disability & accessibility network within our organization. Perhaps such groups might be worth talking to and/or getting to test gui propositions?

[wxtim]

Whilst trying to understand someone else's Javascript I was wondering if there are docstring standards and/or docstring parsing tools. Casting around the internet I have found a number of tools:

• JSdoc (github) - looks to be the standard tool, insofar as most other tools look to borrow its syntax
• Sphinx JS seems like a good idea, in that it's integrated with Sphinx
• ESDOC
• Docco
• Mr-Doc
• YUIdoc

None of the ones which I've looked at on GitHub look vastly active, although if they work that's not necessarily an issue. This blog post has some interesting comments on how well a number of systems work, although frustratingly not SphinxJS, which I like the sound of since it is an extension to software we already use.

The good news is that across these platforms JS appears to have a fairly standard syntax for docstrings. Is it worth including docstringing in any definition of done?

[kinow]

@wxtim good question. I think in the long run they will converge to whatever standard is the one more adopted by TypeScript developers.

Maybe https://github.com/Microsoft/tsdoc? Which looks refreshingly - for me - similar to Java's javadoc (for me Sphinx/reStructuredText Python does too). Example from their readme:

export class Statistics {
  /**
   * Returns the average of two numbers.
   *
   * @remarks
   * This method is part of the {@link core-library#Statistics | Statistics subsystem}.
   *
   * @param x - The first input number
   * @param y - The second input number
   * @returns The arithmetic mean of `x` and `y`
   *
   * @beta
   */
  public static getAverage(x: number, y: number): number {
    return (x + y) / 2.0;
  }
}

[hjoliver]

@amckinstry - a belated update to your question (in case the answer to it was lost in the noise here and elsewhere):

Hi, Is there a timeline / milestone for this issue ? pygtk is being removed from Debian/Ubuntu, which means as it stands cylc will ship without a GUI.
The current timeline is release of Ubuntu ~October (freeze Sept) and Debian 10 freeze ~Feb 2019.

Note that if its agreed what new frameworks will be used (electron?), then that would be good to know, as I can make sure dependencies are present before then.

We're not going to port the GUI to PyGObject, and not going to make a drop-in GUI replacement with Electron, because there is a strong need for an in-browser web GUI that integrates with site identity management and supports remote workflow monitoring, etc. We have settled on an architecture very much like that of Jupyter Hub. It involves: Python 3, proper packaging (pip and conda), Tornado, ZeroMQ, GraphQL, Vue.js, and other technologies. Target first release mid to late 2019.

[kinow]

This issue has discussions around Graph, Design, and several other topics - some covered by other issues.

I wonder if we would be able to close this one and move whatever is not documented in other issues to new, and more specific issues.

[matthewrmshin]

I wonder if we would be able to close this one and move whatever is not documented in other issues to new, and more specific issues.

Makes sense.

[kinow]

I finished re-reading this ticket and the comments. And looks to me like we are still in the right direction.

Most comment had features like Warning icon (already implemented in a component by Oliver), GraphQL (done), sets of icons for different statuses (done but still being discussed in other items regarding design and separation of Job/Task), performance displaying graphs (being discussed in a separate issue though with lower priority than the Tree view), theming (discussed in a separate issue), accessibility (covered in design and other issues like the one for font size), I think the only things that could be copied to other issues (or the other issue could simply reference this one to add a link) are:

• Gantt chart sketches (One GUI using technology for web applications #81 (comment)), add to the item in the list here Implement Views #75
• Some comments from Hilary and Oliver about graphs and JS (One GUI using technology for web applications #81 (comment), One GUI using technology for web applications #81 (comment) add to the issue about graph component

Looks like most of the topics here are already being covered in separated issues 🎉 Any objection to just adding the two links above, and then closing this issue?

And great to see that from 2017 to now, we are making good progress and in a steady trajectory 🚢

[hjoliver]

Looks like most of the topics here are already being covered in separated issues tada Any objection to just adding the two links above, and then closing this issue?

Seems fine to me - I just quickly re-read all the comments above.

[kinow]

Comments copied to new issues. Closing this one now 👍

[hjoliver]

Thanks @kinow 👍