Platforms: Upgrade Cylc 7 Host Syntax #3772

wxtim · 2020-08-13T08:54:30Z

Summary

As per the deprecation method section of the platforms proposal suite tasks where Cylc 7 syntax such as

[runtime]
  [[MY_TASK]]
    [[[job]]]
       batch system = some batch system
    [[[remote]]]
      host = myhost

should select a sensible batch system, or return a reasonable error if unable to do so.

This PR attempts to address this.

Work done

Removed cylc.flow.cfgspec.suite.host_to_platform_upgrader and replaced it with an upgrade warner function to be called on workflow validation. This can't catch every possible problem, but will hopefully fail loudly if something obviously wrong is put in a config.
Added the logic to a get_platform interface to the platform module: Replaced instances of platform_from_name with get_platform.
Added upgrader logic to get_platform.
Refactored some functional tests which I missed in the original platforms work, but now fail.
Added Functional test tests/functional/validate/04-host-platform-upgrade.t - this is not meant to comprehensively test the upgrader function, merely to test that it has been plugged into the cylkc
Added s unit test tests/unit/cfgspec/test_suite.py which is meant to provide comprehensive coverage of the warning function, and a unit test for get_platform to tests/unit/test_platforms.py.

MetRonnie · 2020-08-17T11:34:38Z

tests/functional/test_config_upgrader.py

@@ -221,9 +221,13 @@ def test_upgrader_where_host_is_function(tmp_path, caplog):
    caplog.set_level(logging.DEBUG)


I don't understand why this file has been moved to functional/?

I think that at one point I began to believe that this was more functional than unit, but it isn't.

So what aspect of this test is functional?

The "line in the sand" that I have in my head is:

Integration - Involves the interaction of more than one significant piece of logic.

Functional - Involves interaction with the shell, browser or other systems.

This is why the integration test suites are started in held mode and don't involve anything that goes via the CLI.

I think this has been moved back again or removed?

tests/unit/test_suite_config_upgrader.py

cylc/flow/cfgspec/suite.py

cylc/flow/config.py

MetRonnie

Need to go through and update *.rc -> .cylc

hjoliver · 2020-08-19T00:25:23Z

(A few conflicts and test failures now).

cylc/flow/config.py

replace platform_from_name with get_platform where args=None replace platform_from_name with get_platform where args!=None fix unit tests with new syntax

allow for platform = localhost where to settings at cylc 7 or 8 have been set removed unwanted tests for deleted method finessing the platform lookup

…rt.t tests: tests/functional/cylc-get-config/00-simple/section2.stdout tests: tests/f/job-submission/10-at-shell.t tests: tests/functional/directives/01-at.t tests: tests/functional/job-submission/00-user.t

… that

wxtim · 2020-08-25T10:20:09Z

Can I poke @oliver-sanders and @MetRonnie to have a final double check that I've answered outstanding issues?

You said that I couldn't do the upgrade before inheritance, but that I didn't want to warn for every failure after inheritance. So I've replaced the upgrade-on-config-loading function with a warn (fail if is deffo wrong) at config-load, then added a the upgrade logic at task job sumbit.

cylc/flow/cfgspec/suite.py

cylc/flow/platforms.py

Co-authored-by: Oliver Sanders <oliver.sanders@metoffice.gov.uk>

oliver-sanders · 2020-08-27T13:25:00Z

tests/unit/test_platforms.py

+# ----------------------------------------------------------------------------
+# Tests of platform_from_name
+# ----------------------------------------------------------------------------


Not a blocker but we shouldn't have file-dividers in code. If the file is getting so big that you feel the need to add markers to divide related logic into separate sections, consider splitting the file.

I've always hated them in the bash test scripts too 🤮 (no offense @wxtim !)

None Taken.

tests/unit/test_host_select.py

hjoliver

The new upgrade logic seems a pretty good way to do this. 👍

… function

oliver-sanders · 2020-09-01T14:56:35Z

I think all outstanding comments are resolved, looks ok, merging with two approvals.

wxtim added this to the cylc-8.0a3 milestone Aug 13, 2020

wxtim self-assigned this Aug 13, 2020

wxtim marked this pull request as draft August 13, 2020 08:54

wxtim force-pushed the plumbing.reverse-lookup-unit-tests branch 3 times, most recently from 9c89175 to bf70a1c Compare August 17, 2020 10:41

wxtim requested a review from MetRonnie August 17, 2020 10:54

wxtim marked this pull request as ready for review August 17, 2020 10:56

wxtim requested a review from hjoliver August 17, 2020 11:33

MetRonnie reviewed Aug 17, 2020

View reviewed changes

tests/unit/test_suite_config_upgrader.py Outdated Show resolved Hide resolved

MetRonnie approved these changes Aug 17, 2020

View reviewed changes

hjoliver previously requested changes Aug 17, 2020

View reviewed changes

cylc/flow/cfgspec/suite.py Outdated Show resolved Hide resolved

cylc/flow/cfgspec/suite.py Outdated Show resolved Hide resolved

cylc/flow/config.py Outdated Show resolved Hide resolved

wxtim force-pushed the plumbing.reverse-lookup-unit-tests branch 2 times, most recently from f51a918 to ee35376 Compare August 18, 2020 10:20

MetRonnie reviewed Aug 18, 2020

View reviewed changes

wxtim force-pushed the plumbing.reverse-lookup-unit-tests branch 2 times, most recently from 9daf8aa to e817122 Compare August 19, 2020 09:33

wxtim marked this pull request as draft August 20, 2020 10:21

wxtim force-pushed the plumbing.reverse-lookup-unit-tests branch from 804b815 to ea51631 Compare August 20, 2020 12:04

wxtim commented Aug 21, 2020

View reviewed changes

cylc/flow/config.py Show resolved Hide resolved

wxtim added 3 commits August 21, 2020 07:21

replace usage of platform_from_name with get_platform

4644648

replace platform_from_name with get_platform where args=None replace platform_from_name with get_platform where args!=None fix unit tests with new syntax

add upgrader logic to get_platform method

c4aefdd

allow for platform = localhost where to settings at cylc 7 or 8 have been set removed unwanted tests for deleted method finessing the platform lookup

tests: Platformized tests/flakyfunctional/hold-release/13-ready-resta…

64d0265

…rt.t tests: tests/functional/cylc-get-config/00-simple/section2.stdout tests: tests/f/job-submission/10-at-shell.t tests: tests/functional/directives/01-at.t tests: tests/functional/job-submission/00-user.t

wxtim force-pushed the plumbing.reverse-lookup-unit-tests branch from dced080 to 64d0265 Compare August 21, 2020 07:21

wxtim added 4 commits August 21, 2020 07:23

moved platforms tests to file test_<name_of_file in main repo>

a955b30

added host => platform upgrade warner to suite.py

fee5098

remove unwanted shebang line and add copyright notice to file missing…

88d8f7b

… that

functional test for upgrader added

b72dcf7

changes to warning suggested by @osanders

326f3fe

wxtim force-pushed the plumbing.reverse-lookup-unit-tests branch from aaa3f93 to 326f3fe Compare August 24, 2020 16:38

fixed error message

c8b8af4

MetRonnie approved these changes Aug 25, 2020

View reviewed changes

wxtim mentioned this pull request Aug 26, 2020

Platforms.fix tests #3790

Merged

wxtim requested a review from hjoliver August 26, 2020 13:37

wxtim mentioned this pull request Aug 26, 2020

Platforms.eval platform cmd #3791

Merged

2 tasks

made platforms errors more helpful.

5ab88e9

wxtim requested a review from oliver-sanders August 27, 2020 08:06

MetRonnie approved these changes Aug 27, 2020

View reviewed changes

oliver-sanders reviewed Aug 27, 2020

View reviewed changes

cylc/flow/cfgspec/suite.py Outdated Show resolved Hide resolved

cylc/flow/cfgspec/suite.py Outdated Show resolved Hide resolved

cylc/flow/platforms.py Outdated Show resolved Hide resolved

cylc/flow/platforms.py Outdated Show resolved Hide resolved

wxtim and others added 3 commits August 27, 2020 14:43

Update cylc/flow/cfgspec/suite.py

a0e30dc

Co-authored-by: Oliver Sanders <oliver.sanders@metoffice.gov.uk>

Update cylc/flow/platforms.py

fa072e5

Co-authored-by: Oliver Sanders <oliver.sanders@metoffice.gov.uk>

fix issues caused by change in error message

5e987c9

oliver-sanders reviewed Aug 27, 2020

View reviewed changes

hjoliver approved these changes Aug 28, 2020

View reviewed changes

wxtim force-pushed the plumbing.reverse-lookup-unit-tests branch from a9551a4 to 4bb8e9c Compare August 28, 2020 07:53

Refactored failure caused by platform and host conflict into a single…

2122c51

… function

wxtim force-pushed the plumbing.reverse-lookup-unit-tests branch from 4bb8e9c to 2122c51 Compare August 28, 2020 07:55

centralized the platform warn logic

93522d0

wxtim force-pushed the plumbing.reverse-lookup-unit-tests branch from b52428e to 93522d0 Compare August 28, 2020 14:46

Merge branch 'master' into plumbing.reverse-lookup-unit-tests

60731b1

oliver-sanders merged commit 330735e into cylc:master Sep 1, 2020

wxtim deleted the plumbing.reverse-lookup-unit-tests branch September 3, 2020 08:44

oliver-sanders mentioned this pull request Sep 11, 2020

[New Config 2] Introduce [job platforms] #3421

Closed

2 tasks

hjoliver modified the milestones: cylc-8.0a3, cylc-8.0b0 Feb 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Platforms: Upgrade Cylc 7 Host Syntax #3772

Platforms: Upgrade Cylc 7 Host Syntax #3772

wxtim commented Aug 13, 2020 •

edited

Loading

MetRonnie Aug 17, 2020

wxtim Aug 17, 2020

oliver-sanders Aug 18, 2020 •

edited

Loading

oliver-sanders Sep 1, 2020 •

edited

Loading

MetRonnie left a comment •

edited

Loading

hjoliver commented Aug 19, 2020

wxtim commented Aug 25, 2020

oliver-sanders Aug 27, 2020

hjoliver Aug 27, 2020

wxtim Sep 1, 2020

hjoliver left a comment

oliver-sanders commented Sep 1, 2020

		@@ -221,9 +221,13 @@ def test_upgrader_where_host_is_function(tmp_path, caplog):
		caplog.set_level(logging.DEBUG)

Platforms: Upgrade Cylc 7 Host Syntax #3772

Platforms: Upgrade Cylc 7 Host Syntax #3772

Conversation

wxtim commented Aug 13, 2020 • edited Loading

Summary

Work done

MetRonnie Aug 17, 2020

Choose a reason for hiding this comment

wxtim Aug 17, 2020

Choose a reason for hiding this comment

oliver-sanders Aug 18, 2020 • edited Loading

Choose a reason for hiding this comment

oliver-sanders Sep 1, 2020 • edited Loading

Choose a reason for hiding this comment

MetRonnie left a comment • edited Loading

Choose a reason for hiding this comment

hjoliver commented Aug 19, 2020

wxtim commented Aug 25, 2020

oliver-sanders Aug 27, 2020

Choose a reason for hiding this comment

hjoliver Aug 27, 2020

Choose a reason for hiding this comment

wxtim Sep 1, 2020

Choose a reason for hiding this comment

hjoliver left a comment

Choose a reason for hiding this comment

oliver-sanders commented Sep 1, 2020

wxtim commented Aug 13, 2020 •

edited

Loading

oliver-sanders Aug 18, 2020 •

edited

Loading

oliver-sanders Sep 1, 2020 •

edited

Loading

MetRonnie left a comment •

edited

Loading