More deterministic tests #3336

matthewrmshin · 2019-08-29T23:13:05Z

Remove many sleeps in favour of detecting events.

test_header: new function to install, validate and run reference test.

Fewer lines and flavours for many test files.

test_header: dedicated poll related functions to simplify usage patterns.

Also reversed the logic of the poll function, as most usage were doing poll ! ....

New function to poll suite log in cylc/flow/etc/job.sh. Will follow in #3346.

Remove unit tests duplicated in battery.

Move cylc.wallclock tests from battery to unit test.

These changes partially address #3046.

Requirements check-list

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).

Already covered by existing tests.

No change log entry required (why? e.g. invisible to users).

No documentation update required.

out

tests/cylc-cat-log/00-local.t

tests/cylc-cat-log/06-log-rotation.t

oliver-sanders · 2019-08-30T10:05:51Z

tests/reload/remove-add-alter-task/suite.rc

+    while test "$(grep -cF 'Reload completed' "${CYLC_SUITE_LOG_DIR}/log")" -ne "$1"
+    do
+        sleep 1
+    done


Might want to add some form of timeout to these tests prevent them from hanging.

Use the poll function from test_header - it now aborts after a minute if condition not met.

Yes, if there is an easy way to expose the test battery poll function in the job scripts.

Set PATH at the suite level?

No straightforward. It is a shell function in the test_header.

For the purpose of this change, we should have a way to share the shell function, because the pattern is repeated in many places. I wonder if we should just add the function to cylc/flow/etc/job.sh? Maybe users can make use of a poll function in their task jobs as well?

There are multiple test functions it might be useful to run from within a suite. How about moving the functions out of the test header into another file which can easily be sourced in a pre-script or something like that.

Yes, a library of shell functions is definitely useful, but probably outside the scope of this PR.

New issue for this: #3346

I have added a function to cylc/flow/etc/job.sh to allow job scripts that have the same poll suite log pattern to simple call a function. We can generalise this as part of #3346.

tests/triggering/fam-finish-any/suite.rc

matthewrmshin · 2019-09-03T09:58:08Z

This is not done yet. I have yet to do capture:

~~All the files that run a basic validate + reference test.~~
A few more unnecessary sleeps.

matthewrmshin · 2019-09-03T15:46:01Z

The tests are definitely more stable, but I am not really seeing any overall speed gain (although most of the heavily modified tests do run faster individually), so removing the faster bit from the title.

matthewrmshin · 2019-09-03T18:19:41Z

Tests very stable in my environment, but not so on Travis CI. I'll continue to monitor.

hjoliver · 2019-09-06T10:50:18Z

Is this ready to for a final review or are you still working on it?

matthewrmshin · 2019-09-06T10:51:36Z

This is ready. (More work can be done in follow-on PRs.)

matthewrmshin · 2019-09-06T10:59:50Z

cylc/flow/tests/test_wallclock.py

+    try:
+        assert get_unix_time_from_time_string(time_str) == time_sec
+    finally:
+        CALENDAR.set_mode(mode)


This change replaces tests/cylc.wallclock/00-single.t

hjoliver · 2019-09-08T00:57:57Z

tests/cyclers/38-unittests.t

-#-------------------------------------------------------------------------------
-run_ok "${TEST_NAME_BASE}-cycling-iso8601" python3 -m 'cylc.flow.cycling.iso8601'
-run_ok "${TEST_NAME_BASE}-cycling-integer" python3 -m 'cylc.flow.cycling.integer'
-run_ok "${TEST_NAME_BASE}-cycling" python3 -c 'import cylc.flow.cycling'


Huh, I guess this used to actually run some primitive unit tests in the source modules, and we forgot to delete it once we started doing unit tests separately.

hjoliver

Looks good, no problems found 👍

Remove many sleeps in favour of detecting events. test_header: new function to install, validate and run reference test.

Dedicated functions for the usage patterns. Positive poll logic.

This is a temporary measure to address the repeated pattern in various test suites where we have to repeatedly poll the suite log for events.

matthewrmshin · 2019-09-09T13:30:57Z

Re-based to pick up and test with latest changes.

matthewrmshin added this to the cylc-8.0a2 milestone Aug 29, 2019

matthewrmshin self-assigned this Aug 29, 2019

hjoliver mentioned this pull request Aug 30, 2019

Support symlinked alternate run directories (cylc 8). #2935

Merged

6 tasks

oliver-sanders reviewed Aug 30, 2019

View reviewed changes

out Outdated Show resolved Hide resolved

oliver-sanders reviewed Aug 30, 2019

View reviewed changes

matthewrmshin force-pushed the fix-tests-again branch from 5af74ca to 98968e0 Compare September 3, 2019 14:05

matthewrmshin changed the title ~~Faster and more deterministic tests~~ More deterministic tests Sep 3, 2019

matthewrmshin force-pushed the fix-tests-again branch from 98968e0 to fa44e30 Compare September 3, 2019 15:47

hjoliver mentioned this pull request Sep 3, 2019

Make some shell functions available to job scripts #3346

Open

matthewrmshin force-pushed the fix-tests-again branch 4 times, most recently from 1e71f28 to 58fe9aa Compare September 4, 2019 22:14

matthewrmshin marked this pull request as ready for review September 6, 2019 08:37

matthewrmshin commented Sep 6, 2019

View reviewed changes

hjoliver reviewed Sep 8, 2019

View reviewed changes

hjoliver approved these changes Sep 8, 2019

View reviewed changes

matthewrmshin added 3 commits September 9, 2019 14:29

More deterministic tests

9553d09

Remove many sleeps in favour of detecting events. test_header: new function to install, validate and run reference test.

Improve test battery poll functions

01f9b8d

Dedicated functions for the usage patterns. Positive poll logic.

Add poll suite log function to job.sh

e4943d8

This is a temporary measure to address the repeated pattern in various test suites where we have to repeatedly poll the suite log for events.

matthewrmshin force-pushed the fix-tests-again branch from 58fe9aa to e4943d8 Compare September 9, 2019 13:30

oliver-sanders approved these changes Sep 9, 2019

View reviewed changes

oliver-sanders merged commit 5899d66 into cylc:master Sep 9, 2019

matthewrmshin removed this from the cylc-8.0a2 milestone Sep 9, 2019

matthewrmshin added this to the cylc-8.0a1 milestone Sep 9, 2019

matthewrmshin deleted the fix-tests-again branch September 9, 2019 17:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More deterministic tests #3336

More deterministic tests #3336

matthewrmshin commented Aug 29, 2019 •

edited

Loading

oliver-sanders Aug 30, 2019

hjoliver Sep 3, 2019

matthewrmshin Sep 3, 2019

oliver-sanders Sep 3, 2019

matthewrmshin Sep 3, 2019

matthewrmshin Sep 3, 2019

oliver-sanders Sep 3, 2019

matthewrmshin Sep 3, 2019

hjoliver Sep 3, 2019

matthewrmshin Sep 6, 2019

matthewrmshin commented Sep 3, 2019 •

edited

Loading

matthewrmshin commented Sep 3, 2019 •

edited

Loading

matthewrmshin commented Sep 3, 2019

hjoliver commented Sep 6, 2019

matthewrmshin commented Sep 6, 2019

matthewrmshin Sep 6, 2019

hjoliver Sep 8, 2019

hjoliver left a comment

matthewrmshin commented Sep 9, 2019

More deterministic tests #3336

More deterministic tests #3336

Conversation

matthewrmshin commented Aug 29, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewrmshin commented Sep 3, 2019 • edited Loading

matthewrmshin commented Sep 3, 2019 • edited Loading

matthewrmshin commented Sep 3, 2019

hjoliver commented Sep 6, 2019

matthewrmshin commented Sep 6, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hjoliver left a comment

Choose a reason for hiding this comment

matthewrmshin commented Sep 9, 2019

matthewrmshin commented Aug 29, 2019 •

edited

Loading

matthewrmshin commented Sep 3, 2019 •

edited

Loading

matthewrmshin commented Sep 3, 2019 •

edited

Loading