Platform from platform group basic unit test #3832

wxtim · 2020-09-22T10:13:43Z

Precursor to #3827
Partial fix of issues outstanding from #3793

store platform group name

Probably required for #3827

refactor platforms tests

The file tests/unit/test_platforms.py had become large. I moved tests of the function get_platform to a separate file: test_platforms_get_platform.py

add test for random platform selection

The test for random platform selection tests/flakyfunctional/platform_aliases/00-simple.t makes a trade off between run time and probability of giving a false negative. For some reason I failed to use the Pytest unit testing framework at tests/unit/test_platforms.py. By using the speed of pytest I was able to increase the number of iterations of the function carried out in a reasonable time, and reduce the probability of flakiness to about 1.6x10^-30 - I think this might be overkill, but it's not slow, and effectively not flaky.

Question

Can/Should I delete tests/flakyfunctional/platform_aliases?

- store platform name - add test for random platform selection

hjoliver · 2020-09-23T05:23:24Z

and reduce the probability of flakiness to about 1.6x10^-30

🤣

hjoliver · 2020-09-23T05:26:20Z

Can/Should I delete tests/flakyfunctional/platform_aliases?

Do you mean move it out of the "flaky" category? (Not delete it outright). If so, yes - going by your description of the chance failure, it is now the least flaky functional test in the entire universe. [UPDATE: it's not entirely clear to me from your description if that applies to the functional test or the unit tests??]

wxtim · 2020-09-23T08:40:54Z

Can/Should I delete tests/flakyfunctional/platform_aliases?

Do you mean move it out of the "flaky" category? (Not delete it outright). If so, yes - going by your description of the chance failure, it is now the least flaky functional test in the entire universe. [UPDATE: it's not entirely clear to me from your description if that applies to the functional test or the unit tests??]

I mean delete tests/flakyfunctional/platform_aliases outright.
tests/unit/test_platforms_get_platform::test_get_platform_groups_basic unit test is a new test.
I think it covers the same ground as the functional test.
I'd like the reviewers to check that the functional test doesn't offer anything extra that we want.

datamel

I have checked out your branch, read the code and run the tests. I am happy with the change and I think it makes sense to move these tests to a new file.
With regards to the flaky functional test, I think, if I have understood things correctly, that test can be deleted - a flakiness number of 100 is preferential to 9, and I think the unit tests now have this covered.

tests/unit/test_platforms_get_platform.py

datamel · 2020-09-23T08:57:46Z

tests/unit/test_platforms_get_platform.py

+    names = []
+    # this number keeps the test fast but give about 1.6x10^-30 chance of
+    # a false negative.
+    flakiness_n = 100


Nice to be able to increase the flakiness number from 9 to 100.

Given the speed of pytest it could go a lot higher without being a problem, but 10^-30 seems sufficient. It's probably less likely than computer running the test bursting into flames whilst running the test.

Co-authored-by: Melanie Hall <37735232+datamel@users.noreply.github.com>

…platform::test_get_platform_from_group_basic

…lc into platforms.aliases.add_unit_test * 'platforms.aliases.add_unit_test' of github.com:wxtim/cylc: Update tests/unit/test_platforms_get_platform.py Update tests/unit/test_platforms_get_platform.py

oliver-sanders · 2020-09-23T11:58:48Z

tests/unit/test_platforms_get_platform.py

+    flakiness = 2 / 2 ** flakiness_n
+    for i in range(flakiness_n):
+        names.append(get_platform('hebrew_letters')['name'])
+        if 'aleph' in names and 'bet' in names:
+            break


With this test written in Python there might be a nicer solution, the Python random library is only psudo-random. You can control the series of random numbers it produces by setting the random seed.

random.seed(1234) assert [ get_platform('hebrew_letters')['name'] for _ in range(N) ] == [ # results should come out in the same order each time ]

This will, I think work rather well. Thanks for the point!

I tried it and it works. I am amused that this testing approach takes about 4x the time the earlier rather crude version did.

oliver-sanders · 2020-09-24T08:45:26Z

Can/Should I delete tests/flakyfunctional/platform_aliases?

Yes, I think we've gone to great lengths to prove that the random method does indeed work.

* master: (36 commits) Cleaned up logging a little (cylc#3836) Platform from platform group basic unit test (cylc#3832) added reqd test_header tid platform tests scan: improve colour output spelling error fix scan: fix hier suites test: old host/batch system configs don't conflict with dummy-local mode Fix task message validation for multiline messages Fix spelling error. removed cylc7 data structure removed unused endpoints Commands migrated to GraphQL endpoint cli: rename spawn -> set-outputs Update changelog Remove test_unicode_rules Do not allow any colons in task outputs Fix command help. Correct typo. Upgraded new test. ...

- refactor platforms tests

dadb0b6

- store platform name - add test for random platform selection

wxtim self-assigned this Sep 22, 2020

wxtim requested review from datamel and oliver-sanders September 22, 2020 10:15

wxtim added could be better Not exactly a bug, but not ideal. platforms-follow-up question Flag this as a question for the next Cylc project meeting. labels Sep 22, 2020

exit loop if test satisfied

558f352

datamel approved these changes Sep 23, 2020

View reviewed changes

wxtim and others added 4 commits September 23, 2020 11:00

Update tests/unit/test_platforms_get_platform.py

9245d65

Co-authored-by: Melanie Hall <37735232+datamel@users.noreply.github.com>

Update tests/unit/test_platforms_get_platform.py

d6ac88c

Co-authored-by: Melanie Hall <37735232+datamel@users.noreply.github.com>

remove flakyfunctional test replaced by tests/unit/test_platform_get_…

f9b22be

…platform::test_get_platform_from_group_basic

Merge branch 'platforms.aliases.add_unit_test' of github.com:wxtim/cy…

2a21898

…lc into platforms.aliases.add_unit_test * 'platforms.aliases.add_unit_test' of github.com:wxtim/cylc: Update tests/unit/test_platforms_get_platform.py Update tests/unit/test_platforms_get_platform.py

oliver-sanders reviewed Sep 23, 2020

View reviewed changes

made platform_group random selecta test unflaky

b6d9520

wxtim requested a review from oliver-sanders September 25, 2020 06:55

oliver-sanders approved these changes Sep 25, 2020

View reviewed changes

oliver-sanders merged commit 340b8ae into cylc:master Sep 25, 2020

wxtim deleted the platforms.aliases.add_unit_test branch March 22, 2022 17:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Platform from platform group basic unit test #3832

Platform from platform group basic unit test #3832

wxtim commented Sep 22, 2020

hjoliver commented Sep 23, 2020

hjoliver commented Sep 23, 2020 •

edited

Loading

wxtim commented Sep 23, 2020 •

edited

Loading

datamel left a comment

datamel Sep 23, 2020

wxtim Sep 23, 2020 •

edited

Loading

oliver-sanders Sep 23, 2020

wxtim Sep 23, 2020 •

edited

Loading

oliver-sanders commented Sep 24, 2020

Platform from platform group basic unit test #3832

Platform from platform group basic unit test #3832

Conversation

wxtim commented Sep 22, 2020

store platform group name

refactor platforms tests

add test for random platform selection

Question

hjoliver commented Sep 23, 2020

hjoliver commented Sep 23, 2020 • edited Loading

wxtim commented Sep 23, 2020 • edited Loading

datamel left a comment

Choose a reason for hiding this comment

datamel Sep 23, 2020

Choose a reason for hiding this comment

wxtim Sep 23, 2020 • edited Loading

Choose a reason for hiding this comment

oliver-sanders Sep 23, 2020

Choose a reason for hiding this comment

wxtim Sep 23, 2020 • edited Loading

Choose a reason for hiding this comment

oliver-sanders commented Sep 24, 2020

hjoliver commented Sep 23, 2020 •

edited

Loading

wxtim commented Sep 23, 2020 •

edited

Loading

wxtim Sep 23, 2020 •

edited

Loading

wxtim Sep 23, 2020 •

edited

Loading