static graph visualization

[hjoliver]

We probably want to replace cylc graph with web technology too, not just the control GUI graph view.

[hjoliver]

This graph demo (found by @MartinRyan ) allows manual node repositioning (as many do, I think): https://cytoscape.org/cytoscape.js-euler/. Could be useful when automatic layout fails us.

[MartinRyan]

Cytoscape looks great, it has a dagre extension, and a navigator 'bird's eye view and pan control' extension (mini-map). It consumes JSON though there are convertors for graph objects.

[hjoliver]

Also from @MartinRyan py2cytoscape might be useful for static vis. (Cytoscape itself might be useful for the live control GUI as well)

[oliver-sanders]

This is an interesting challenge because:

• Users typically run cylc graph to visualise a suite whilst developing the working copy.
• The UI-Server is concerned with the installed copy (in the cylc-run directory).

The UI-Server is only concerned with the installed copy and is not necessarily able to see the working copy.

It is OK to require users to register a suite for cylc graph functionality (as the suite wouldn't appear in the web UI otherwise) but we wouldn't want users to have to repeatedly re-install a suite in order to visualise it.

[hjoliver]

Well, yes and no.

If you're working on the working copy then you are able to install it repeatedly pretty easily, and if the payoff is kick-ass web base viz that seems a small price to pay, And besides you currently have to do that anyway if your suite.rc takes Jinja2 inputs from rose-suite.conf (it won't validate or graph without being installed first).

Also what's the alternative? We don't want to keep a GTK-based static graph GUI. An electron app might allow us to kill both birds with one stone, I guess.

[matthewrmshin]

Assuming that we have done enough logical migration from rose suite-run to Cylc in the coming months, then we should have enough logic in Cylc to deal with the rose-suite.conf stuffs. The web UI can have an local file system browser so users should be able to open a suite from its source location. Otherwise, we can also build support for the UI to search for named suites from various configurable locations on the file system. (E.g. If a suite cannot be found under ~/cylc-run/SUITE/ then look at ~/roses/SUITE/.)

[oliver-sanders]

If you're working on the working copy then you are able to install it repeatedly pretty easily

But what if the suite is running?

Also this is the situation we have at the moment, I advise people to do this:

$ rose suite-run -l; cylc graph mysuite

Unfortunately this causes confusion and is a barrier to usage. As a result very few people here use essential Cylc utilities such as:

• cylc graph
• cylc get-config
• cylc view --jinja2

I wrote this page to try and break down the barrier but sadly it has made little difference.

I still regularly see users trying is running the entire suite to test changes.

Also what's the alternative?

No nice solution:

• An electron app if it's quick and simple.
• The UI Server could attempt to access the working copy (via ~/cylc-run/<suite>/.service/source)
  • If the working copy is not available it could fallback to the installed copy and present a warning.
• Require the user repeatably re-installs the suite during development.

[hjoliver]

Fair point on barriers to easy use. So let's do better than that if we can. We do need a non-PyGTK static vis tool with cylc-8 though, one way or another.

I still regularly see users trying is running the entire suite to test changes.

Same, and even worse, running an entire beast of a suite repeatedly to test how some cylc feature works, when a 5-line 1-task test suite would do. You can lead a horse to water, but you can't make it change its spots. Or something.

[kinow]

See comments from (now closed) #81 about graphs in Cylc 8:

@hjoliver

A Vue component to draw directed acyclic graphs in the browser leveraging dagre-d3v4 (https://github.com/smbolton/dagre-d3v4) - https://github.com/vdidonato/vue-dagre-d3v4 - seems to be broken.

#81 (comment)

@oliver-sanders

I'm currently looking at Javascript libraries for graphing, I'll update this issue with conclusions as I investigate. I'll probably put up my experiments as a GitHub repo when more complete.

Graphing consists of two components:

The Layout Engine

The bit which arranges nodes / edges spatially. See the layout engines supported by cytoscape for examples.

• Directed hierarchical graphs are hard
• Layout is slow, there isn't a whole lot of choice when it comes to layout engines. Dagre is the standout but is a fairly stagnant project.
• More arbitrary layouts are easy
• We might want to automatically swap to a less ideal layout engine for larger graphs.

Requirements

1. Directed & hierarchical
2. Must handle edges spanning multiple levels
3. Width of node considered in graph (i.e. nodes cannot overlap)
4. Speed!

Results

Engine	1	2	3	4	Supported Engines
Dagre	🆗	🆗	🆗	4.1s	cytoscape, d3
Klay	🆗	⛅	🚫	6.4s	cytoscape, d3
D3-DAG	🆗	🆗	🚫 [2]	>> 60s [3]	None? [4]

[1] Benchmark using the first cycle of MO global opp suite
[2] Possibly using the separation function though I can't get this to work
[3] Tested "Vert", "Greedy" and "Spread" layout methods
[4] Visualisation constructed by user provided functions, i.e. user provides the engine

The Graph Engine

The bit which renders the graph, provides an interface for manipulating and interacting with it. This is the component we will end up working with / extending to form the graph view.

Requirements

1. Mobile friendliness
2. Hackability of the graph:
   • Job status indicators (require more than just text labels)
   • Sub-workflows?
   • Advanced collapsing (e.g. parametrisations using an ellipsis symbol)?
3. Not a massive RAM hog
4. Support for the desired layout engine(s)
5. Node / family collapsing

Results

Engine	1	2	3	4	5
cytoscape	🆗			Dagre, Klay
d3				Dagre, Klay
vis.js
arbor

Hybrids

Systems where the graph engine and layout engine are integrated:

Results

Engine	L1	L2	L3	L4	G1	G2	G3	G4	G5
viz.js	🆗	🆗	🆗					🆗

#81 (comment)

@hjoliver

Interactive arbitrary grouping: select a bunch of nodes (individually, or by drawing a rectangle around them) and group them into a single node.
Layout algorithms for visualizing directed acyclic graphs - https://github.com/erikbrinkman/d3-dag - new-ish? demo https://jsfiddle.net/ye2xanf9/12/

#81 (comment)
#81 (comment)