Tui 1.0 #5731

oliver-sanders · 2023-09-14T16:44:38Z

Remove the "Tui is experimental" notice 🎉

Move the data update into another process: Closes tui: improve performance #3527
- Tui used to contact workflows in the same process it used to run the application.
- This meant that the app was held in lockstep with the update. If the update took longer than the refresh time it would fall over.
- This PR puts the updater into another process to break this lockstep so that the application and updater refresh rates can differ.
Add multi-workflow capability: Closes tui: list all suites #3464
- Use the scan pipe to list workflows in Tui the same way we use it in the CylcUIServer.
- The workflow refresh rate is different to the scan refresh rate.
- Each workflow node has one child under it, the "spring" node. When you expand this "spring" node, Tui will connect to the workflow and will replace the "spring" node with the workflow tree once the data arrives.
- When you collapse the workflow node, Tui will disconnect from the workflow.
- You can expand multiple workflows.
Add visual regression testing framework: Closes tui: testing #3530
- Add a test fixture which launches Tui in a "headless" mode where it's not rendering to a terminal.
- The fixture has a method to provide user input allowing you to press keys as if there was a terminal running.
- The fixture has a method for generating HTML screenshots which are then used for visual regression.
- I.E. if anything changes in the output HTML, then test will fail.
- This allows us to test navigation, filtering, help screens and more.
- Tui coverage is now up to ~80%.
Add some new functionalities.
- Log view: support for cylc cat-log.
- Show view: support for cylc show.

tui2-large.mp4

In the process I spotted an issue with task filtering which looks like it might be a data store thing but requires more investigation, lower priority as filtering is not an issue for the GUI. Until then the task filtering won't work right: #5716

TODO:

Niceify the code.
Complete restart/reconnect test.
Do something better with the Tui header (which is now blank).
Investigate test flakyness (workflow states test quite flaky).

Tui integration test failures result in HTML files like this being written into the test directory in ~cylc-run/. When tests fail in CI, these HTML files will be bundled into the artefact so can be inspected locally to determine the cause of the failure:

Check List

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
Tests are included (or explain why tests are not needed).
CHANGES.md entry included if this is a change that can affect users
Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/663.
If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

oliver-sanders · 2023-09-14T16:51:46Z

tests/integration/tui/screenshots/test_navigation.cursor-at-bottom-of-screen.html

@@ -0,0 +1,31 @@
+<pre><span style="color:#000000;background:#e5e5e5">                                                                                </span>


I was really hoping GitHub would render this HTML which would make it much easier to review screenshot updates in PRs. Any ideas?

The best thing I can come up with is to use some Python lib to render them into png derived files.

hjoliver · 2023-09-14T23:59:18Z

Nice, looking forward to this!

oliver-sanders · 2023-09-26T14:22:39Z

cylc/flow/scripts/tui.py

@@ -15,34 +15,37 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.

-"""cylc tui WORKFLOW
+"""cylc tui [WORKFLOW]


If you give a workflow, tui will startup, pre-filtered to just that workflow. Press "E" to unset this filtering an view all workflows.

oliver-sanders · 2023-09-26T14:23:12Z

cylc/flow/scripts/tui.py

        # auto_add=False,  NOTE: at present auto_add can not be turned off
        color=False
    )

-    parser.add_option(
-        '--display',


Removed the HTML testing mode from the Tui script as this is now handled by an integration test fixture.

oliver-sanders · 2023-09-26T14:24:27Z

cylc/flow/tui/app.py

+    # the UI update interval
+    # NOTE: this is different from the data update interval
+    UPDATE_INTERVAL = 0.1


This means that the UI will look for updates every 0.1 seconds, but the frequency of these updates is determined by the updater which is decoupled and runs to its own clock.

oliver-sanders · 2023-09-26T14:25:59Z

cylc/flow/tui/data.py

-    # for now we will exit tui when the workflow is cleaned
-    # this will change when tui supports multiple workflows
-    cli_cmd('clean', workflow)
-    sys.exit(0)


Tui used to exit when you cleaned the workflow, but now it can stay open.

oliver-sanders · 2023-09-26T14:29:30Z

cylc/flow/tui/updater.py

+    # the maximum time to wait for a workflow update
+    CLIENT_TIMEOUT = 2


If the workflow takes more than 2 seconds to send its response, then it will time out. Tui will put a message under the workflow saying that it has timed out, but will retry with the next update cycle.

Previously both the application and updater used the same update period of 1 second. If it took longer to get the response and do the required processing, then Tui would crash.

oliver-sanders · 2023-09-26T14:30:17Z

cylc/flow/tui/updater.py

+    # the interval between workflow data updates
+    BASE_UPDATE_INTERVAL = 1


This is the desired update interval. If it takes longer than 1 second to update the workflow, no problem, the updates just come through slower, no more crashes.

oliver-sanders · 2023-09-28T15:59:49Z

With Tui now better tested, this brings the project coverage up above 90%!

We still can get more of Tui covered now there's a testing framework for it.

dwsutherland · 2023-10-05T04:51:18Z

Amazing work 🚀

oliver-sanders · 2023-10-06T15:26:39Z

(added some extra tests to fill some coverage holes)

wxtim · 2023-10-13T09:45:28Z

Sorry, I think that I found a bug on pressing p in the main view I get this error (sorry, my terminal isn't letting me copy/paste it:

oliver-sanders · 2023-10-13T09:49:22Z

Good spot, the tests cover toggling workflow states using W and filling out the form, but not using the shortcuts. Will fix.

oliver-sanders · 2023-10-13T10:13:16Z

Would clean-all-stopped be unreasonable to ask for?

Not unreasonable, I think that's just cylc clean '*:stopped' or since clean auto filters out running workflows, just cylc clean '*'.

It might be considered a little dangerous having a button like that kicking about, so we might want to put an "are you sure" check on that which Tui doesn't currently have an interface for.

oliver-sanders · 2023-10-13T10:29:06Z

Fixed the "p" shortcut and added it to the tests: https://github.com/cylc/cylc-flow/compare/8e371542150b9792a8593399cefc14f96be5c1df..54b7e609af8676c7fcb3b4e8723a50d92ad9c3a7

* Move the updater into another process. This removes the limitation that caused Tui to crash with active workflows. Closes cylc#3527 * Add multi-workflow capability. Closes cylc#3464 * Add visual regression testing framework. Closes cylc#3530

* Fixes an obscure issue where the applications main loop becomes suspended as the result of attempting to register an `on_press` callback which takes as an argument a Cylc client. * Even though this callback is not actually called at this point, just passing the reference to it appears to be enough to cause thread deadlocking because Cylc clients are async based and not thread safe.

oliver-sanders · 2023-10-13T10:57:29Z

Fixed an issue with "E" not working properly: https://github.com/cylc/cylc-flow/compare/54b7e609af8676c7fcb3b4e8723a50d92ad9c3a7..6f3f1701c1106ecd779b11b08ad4293186678da0

The "E" (reset all workflow filters) binding is the one thing in Tui which we cannot test. This is because in order to prevent Tui from seeing other workflows on the system (outside of the test), we have to use an ID filter. Pressing "E" would reset that filter getting Tui to view everything.

cylc/flow/scripts/tui.py

wxtim

Functional testing of PR.
Read function code.
Read the testing.

I really enjoyed working through the Raikura test framework. 🐦 !

tests/integration/tui/screenshots/test_auto_expansion.later-time.html

tests/integration/tui/test_app.py

* Post-refactor clean up of dangling code.

oliver-sanders · 2023-11-08T10:52:21Z

Training day came around again, added support for cat-log and show.

@wxtim, could you check the last three commits.

oliver-sanders · 2023-11-08T10:53:54Z

cylc/flow/data_store_mgr.py

@@ -1380,7 +1380,7 @@ def insert_job(self, name, cycle_point, status, job_conf):
            name=tproxy.name,
            cycle_point=tproxy.cycle_point,
            execution_time_limit=job_conf.get('execution_time_limit'),
-            platform=job_conf.get('platform')['name'],
+            platform=job_conf['platform']['name'],


Unrelated, spotted during testing.

The .get here wasn't protecting against anything as the default return value of None would cause the ['name'] bit to error anyway.

* Add a log view to tui and add it to the context menu for workflows, tasks and jobs. * Allow raikura sessions to see module-scoped fixtures when appropriate. * Fix a bug (which only surfaced in tests for some reason) where a colour was specified as an empty string which urwid doesn't appear to like.

* Add `cylc show` support for tasks in Tui.

wxtim

This is great. Please stop with the mission creep so that we can merge. Pretty Please.

oliver-sanders added efficiency For notable efficiency improvements could be better Not exactly a bug, but not ideal. labels Sep 14, 2023

oliver-sanders added this to the cylc-8.3.0 milestone Sep 14, 2023

oliver-sanders self-assigned this Sep 14, 2023

oliver-sanders force-pushed the tui++ branch from 25548da to 931d025 Compare September 14, 2023 16:49

oliver-sanders commented Sep 14, 2023

View reviewed changes

oliver-sanders force-pushed the tui++ branch from 931d025 to ad7cbb7 Compare September 15, 2023 08:42

oliver-sanders force-pushed the tui++ branch 2 times, most recently from 11926b2 to 5869bd9 Compare September 26, 2023 12:55

oliver-sanders commented Sep 26, 2023

View reviewed changes

oliver-sanders force-pushed the tui++ branch from c3133b5 to cb2ac47 Compare September 28, 2023 12:33

oliver-sanders marked this pull request as ready for review September 28, 2023 16:00

oliver-sanders force-pushed the tui++ branch 3 times, most recently from 2b969af to 54356aa Compare October 4, 2023 11:02

oliver-sanders force-pushed the tui++ branch from 54356aa to 5f221db Compare October 6, 2023 15:26

oliver-sanders force-pushed the tui++ branch from 5f221db to 8e37154 Compare October 12, 2023 09:28

oliver-sanders requested a review from wxtim October 12, 2023 10:12

This comment was marked as resolved.

Sign in to view

oliver-sanders force-pushed the tui++ branch from 8e37154 to 54b7e60 Compare October 13, 2023 10:28

oliver-sanders added 3 commits October 13, 2023 11:56

Tui 1.0

571b27f

* Move the updater into another process. This removes the limitation that caused Tui to crash with active workflows. Closes cylc#3527 * Add multi-workflow capability. Closes cylc#3464 * Add visual regression testing framework. Closes cylc#3530

tui: update screenshots

b175218

oliver-sanders force-pushed the tui++ branch from 54b7e60 to 6f3f170 Compare October 13, 2023 10:56

wxtim reviewed Oct 13, 2023

View reviewed changes

cylc/flow/scripts/tui.py Outdated Show resolved Hide resolved

wxtim reviewed Oct 13, 2023

View reviewed changes

wxtim mentioned this pull request Oct 20, 2023

Fix cylc show satisfied xtriggers info #5780

Merged

6 tasks

wxtim reviewed Oct 23, 2023

View reviewed changes

tests/integration/tui/screenshots/test_auto_expansion.later-time.html Show resolved Hide resolved

wxtim reviewed Oct 23, 2023

View reviewed changes

tests/integration/tui/test_app.py Show resolved Hide resolved

wxtim approved these changes Oct 23, 2023

View reviewed changes

tui: tidy dangling interfaces

7b2058c

* Post-refactor clean up of dangling code.

oliver-sanders commented Nov 8, 2023

View reviewed changes

oliver-sanders force-pushed the tui++ branch from cea784d to 45c975a Compare November 8, 2023 13:49

oliver-sanders added 2 commits November 8, 2023 15:24

tui: show view

c303aa9

* Add `cylc show` support for tasks in Tui.

oliver-sanders force-pushed the tui++ branch from 45c975a to c303aa9 Compare November 8, 2023 15:28

wxtim approved these changes Nov 9, 2023

View reviewed changes

oliver-sanders requested a review from hjoliver November 13, 2023 11:54

oliver-sanders requested review from markgrahamdawson and removed request for hjoliver November 22, 2023 15:22

markgrahamdawson approved these changes Nov 27, 2023

View reviewed changes

oliver-sanders merged commit ddeaa97 into cylc:master Nov 27, 2023
20 of 24 checks passed

oliver-sanders deleted the tui++ branch November 27, 2023 10:51

oliver-sanders mentioned this pull request Nov 28, 2023

tests/i: fix flaky tui/test_updater tests #5849

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tui 1.0 #5731

Tui 1.0 #5731

oliver-sanders commented Sep 14, 2023 •

edited

Loading

oliver-sanders Sep 14, 2023

oliver-sanders Sep 26, 2023

hjoliver commented Sep 14, 2023

oliver-sanders Sep 26, 2023

oliver-sanders Sep 26, 2023

oliver-sanders Sep 26, 2023

oliver-sanders Sep 26, 2023

oliver-sanders Sep 26, 2023

oliver-sanders Sep 26, 2023

oliver-sanders commented Sep 28, 2023

dwsutherland commented Oct 5, 2023

oliver-sanders commented Oct 6, 2023

wxtim commented Oct 13, 2023

oliver-sanders commented Oct 13, 2023

This comment was marked as resolved.

This comment was marked as resolved.

oliver-sanders commented Oct 13, 2023

oliver-sanders commented Oct 13, 2023

oliver-sanders commented Oct 13, 2023 •

edited

Loading

wxtim left a comment •

edited

Loading

oliver-sanders commented Nov 8, 2023

oliver-sanders Nov 8, 2023

wxtim left a comment

		@@ -0,0 +1,31 @@
		<pre><span style="color:#000000;background:#e5e5e5"> </span>

		# the maximum time to wait for a workflow update
		CLIENT_TIMEOUT = 2

		# the interval between workflow data updates
		BASE_UPDATE_INTERVAL = 1

Tui 1.0 #5731

Tui 1.0 #5731

Conversation

oliver-sanders commented Sep 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hjoliver commented Sep 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oliver-sanders commented Sep 28, 2023

dwsutherland commented Oct 5, 2023

oliver-sanders commented Oct 6, 2023

wxtim commented Oct 13, 2023

oliver-sanders commented Oct 13, 2023

This comment was marked as resolved.

This comment was marked as resolved.

oliver-sanders commented Oct 13, 2023

oliver-sanders commented Oct 13, 2023

oliver-sanders commented Oct 13, 2023 • edited Loading

wxtim left a comment • edited Loading

Choose a reason for hiding this comment

oliver-sanders commented Nov 8, 2023

Choose a reason for hiding this comment

wxtim left a comment

Choose a reason for hiding this comment

oliver-sanders commented Sep 14, 2023 •

edited

Loading

oliver-sanders commented Oct 13, 2023 •

edited

Loading

wxtim left a comment •

edited

Loading