Cylc install #4000

datamel · 2020-12-16T10:44:40Z

Cylc Install command added. These changes close #3889

Cylc can now install workflows from from arbitrary locations into the cylc-run directory.

This PR includes:

Removing cylc register command.
New install log added in log directory: log/install/<timestamp>-install.log

Requirements check-list

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Appropriate tests are included (unit and/or functional).
Appropriate change log entry included. - TODO before ready for review
(master branch) I have opened a documentation PR at cylc/cylc-doc/pull/XXXX. _TODO
No dependency changes.

hjoliver · 2020-12-16T19:25:41Z

Looks impressive @datamel 🎉

oliver-sanders · 2020-12-16T22:36:20Z

Dammit I totally forgot we were approaching PR 4000!

wxtim · 2020-12-17T09:46:38Z

Dammit I totally forgot we were approaching PR 4000!

IMO this is why Mel waited for a few days to put up the draft PR!

cylc/flow/pathutil.py

oliver-sanders · 2020-12-29T10:08:35Z

The bash docker test failures are nothing to worry about, those tests pass locally, likely that the test configuration requires a bit of a poke.

Looking at the GH actions cylc-run directory it looks like the flow.cylc files are not in place suggesting that that installation failed somehow. Perhaps there is some useful error that is being swallowed by 2>/dev/null?

--- a/tests/functional/lib/bash/test_header
+++ b/tests/functional/lib/bash/test_header
@@ -430,7 +430,7 @@ install_suite() {
     SUITE_RUN_DIR="${RUN_DIR}/${SUITE_NAME}"
     mkdir -p "${TEST_DIR}/${SUITE_NAME}/" # make source dir
     cp -r "${TEST_SOURCE_DIR}/${TEST_SOURCE_BASE}/"* "${TEST_DIR}/${SUITE_NAME}/"
-    cylc install --no-run-name --flow-name="${SUITE_NAME}" --directory="${TEST_DIR}/${SUITE_NAME}" 2>'/dev/null'
+    cylc install --no-run-name --flow-name="${SUITE_NAME}" --directory="${TEST_DIR}/${SUITE_NAME}"
     cd "${TEST_DIR}/${SUITE_NAME}/"
 }

It is possible that is might be installing the files in the wrong place? We run as root inside the docker images.

oliver-sanders · 2020-12-30T09:30:25Z

The cylc-rose stuff has been merged now, should be straight forward to merge the two cylc install commands, just need to ensure the plugins get called at the right point.

pre_configure - Before the flow configuration is read.
post_install - Last thing that cylc install does.

oliver-sanders · 2020-12-30T13:23:05Z

Took a crack, this seems to do the job:

diff --git a/cylc/flow/scripts/install.py b/cylc/flow/scripts/install.py
index 5011996d2..f773ffdc4 100755
--- a/cylc/flow/scripts/install.py
+++ b/cylc/flow/scripts/install.py
@@ -58,9 +58,13 @@ multiple workflow run directories that link to the same suite definition.
 
 """
 
+import os
+import pkg_resources
+
 from cylc.flow.option_parsers import CylcOptionParser as COP
-from cylc.flow.suite_files import install_workflow
 from cylc.flow.terminal import cli_function
+from cylc.flow.pathutil import get_workflow_run_dir
+from cylc.flow.suite_files import install_workflow
 
 
 def get_option_parser():
@@ -116,14 +120,30 @@ def get_option_parser():
 def main(parser, opts, flow_name=None, src=None):
     if opts.no_run_name and opts.run_name:
         parser.error(
-            """options --no-run-name and --run-name are mutually exclusive.
+            """opts --no-run-name and --run-name are mutually exclusive.
             Use one or the other""")
-    install_workflow(
+
+    for entry_point in pkg_resources.iter_entry_points(
+        'cylc.pre_configure'
+    ):
+        entry_point.resolve()(opts.source)
+
+    flow_name = install_workflow(
         flow_name=opts.flow_name,
         source=opts.source,
         run_name=opts.run_name,
         no_run_name=opts.no_run_name,
-        no_symlinks=opts.no_symlinks)
+        no_symlinks=opts.no_symlinks
+    )
+
+    for entry_point in pkg_resources.iter_entry_points(
+        'cylc.post_install'
+    ):
+        entry_point.resolve()(
+            dir_=opts.source or os.getcwd(),  # move the source dir logic into this script
+            opts=opts,
+            dest_root=get_workflow_run_dir(flow_name)
+        )
 
 
 if __name__ == "__main__":

oliver-sanders · 2021-01-06T11:22:57Z

This might be the answer to the bash test failures:

diff --git a/dockerfiles/bash/Dockerfile b/dockerfiles/bash/Dockerfile
index 8b9f6a2e1..2396924a1 100644
--- a/dockerfiles/bash/Dockerfile
+++ b/dockerfiles/bash/Dockerfile
@@ -13,6 +13,7 @@ RUN apt-get update && apt-get install -y \
     python3.7-dev \
     sqlite3 \
     wget \
+    rsync \
     && python3.7 -m pip install -U pip setuptools \
     && rm -rf /var/lib/apt/lists/*

cylc/flow/suite_files.py

oliver-sanders · 2021-01-07T10:32:59Z

cylc/flow/suite_files.py

+    return rsync_cmd
+
+
+def install_workflow(flow_name=None, source=None, run_name=None,


Can break this function down a bit, would make it possible to unit test bits of of (e.g. the runN bit).

oliver-sanders · 2021-01-13T09:54:54Z

Something seems to have angered the tests there. Lots of this diff in the logs:

-Parsing cylctb-20210112T182706Z/f/cylc-diff/00-basic-2 (/tmp/tmp.yw9DTu8vUR/cylctb-20210112T182706Z/f/cylc-diff/00-basic-2/flow.cylc)
250
+Parsing cylctb-20210112T182706Z/f/cylc-diff/00-basic-1 (/home/runner/cylc-run/cylctb-20210112T182706Z/f/cylc-diff/00-basic-1/flow.cylc)

datamel · 2021-01-13T10:11:27Z

Something seems to have angered the tests there. Lots of this diff in the logs:

-Parsing cylctb-20210112T182706Z/f/cylc-diff/00-basic-2 (/tmp/tmp.yw9DTu8vUR/cylctb-20210112T182706Z/f/cylc-diff/00-basic-2/flow.cylc)
250
+Parsing cylctb-20210112T182706Z/f/cylc-diff/00-basic-1 (/home/runner/cylc-run/cylctb-20210112T182706Z/f/cylc-diff/00-basic-1/flow.cylc)

Just looking into it now - angering the tests is my speciality!

oliver-sanders · 2021-01-19T13:30:10Z

Looking at the timestamps in the logs none of the four cancelled jobs from the last run produced any output in the minutes before being cancelled.

This suggests that some tests are hanging.

Unfortunately it can be quite hard to work out which of the tests are hanging (I have an idea on this I developed for the Rose tests, will try some time) but you can always try locally.

oliver-sanders

Given a test locally, works well.

I found one quirk where you can end up with a combination of custom and automatically named runs:

$ cylc install --run-name a
$ cylc install
$ tree ~/cylc-run -L 2
~/cylc-run/
└── testy
    ├── _cylc-install
    ├── b
    ├── run1
    └── runN -> ~/cylc-run/testy/run1

I think the correct behaviour should be that the second install would fail, we can tell that b is an installed flow not a run because ~/cylc-run/testy/b contains a flow.cylc file.

cylc/flow/suite_files.py

datamel · 2021-01-22T09:47:32Z

I think the correct behaviour should be that the second install would fail, we can tell that b is an installed flow not a run because ~/cylc-run/testy/b contains a flow.cylc file.

Are we happy for someone to do a named run after a named run? Is it only mix of named and run numbered installs we want to avoid?

e.g.

$ cylc install --run-name a
$ cylc install --run-name b
$ tree ~/cylc-run/testy -L 1

~/cylc-run/testy
|-- _cylc-install
|-- a
`-- b

oliver-sanders · 2021-01-22T09:58:19Z

Yes, because the meaning of the runN symlink gets fuzzy.

.github/workflows/test_functional.yml

oliver-sanders

Nice tests!

GH actions is angry again, looks like all tests are failing for the same reason at least.

tests/functional/cylc-install/00-simple.t

tests/functional/authentication/01-remote-suite-same-name.t

oliver-sanders

Nearly there!

The argument of cylc install is a little funny with its flow/path/PWD opts thing, but we can sort that with the universal id changes later.

cylc/flow/suite_files.py

Cylc install, prohibit source in cylc-run

Add cylc-install to cylc-clean

Add functional tests for cylc-install

Co-authored-by: Oliver Sanders <oliver.sanders@metoffice.gov.uk>

Fix functional test for cylc-install

Fix get_flow_file Fix unit test

* abort if no flow is to be purged * only wait on lsof for a maximum of five tries

Update yml timeout

oliver-sanders

Looks good, a lot of nuances and Cylc history has been blasted aside here, nice.

There are some minor quirks I think we should skip over for this review and come back to later:

The argument of cylc install can be reg/path.
- Come back to this later (universal id stuff).
The flow.cylc is being created in the "source" dir and rsync'ed to the "run" dir.
- It may be better not to create the symlink in the "source" dir.
- This will silence future warnings.
- Users should just mv suite.rc flow.cylc.
The runN symlinks are absolute rather than relative.
- This blocks registration changes, but perhaps that's not the worst thing?

hjoliver · 2021-01-27T21:35:01Z

Looks good, a lot of nuances and Cylc history has been blasted aside here, nice.

yeah brilliant, thanks @datamel 🎉 I'll have a close look as soon as possible...

hjoliver

Looks great, no problems found 🎉 🎉

datamel self-assigned this Dec 16, 2020

wxtim reviewed Dec 17, 2020

View reviewed changes

cylc/flow/pathutil.py Outdated Show resolved Hide resolved

oliver-sanders mentioned this pull request Dec 17, 2020

Rose conf.read jinja2 #3913

Merged

4 tasks

datamel force-pushed the cylc-install branch 4 times, most recently from b3be897 to b66a765 Compare January 6, 2021 10:39

datamel marked this pull request as ready for review January 6, 2021 13:46

oliver-sanders reviewed Jan 7, 2021

View reviewed changes

MetRonnie mentioned this pull request Jan 8, 2021

get-config: unify commands #4009

Closed

datamel force-pushed the cylc-install branch from 315643c to a97cae2 Compare January 12, 2021 18:00

This was referenced Jan 15, 2021

rose suite-run --name=X from multiple source directories metomi/rose#1977

Closed

rose suite-run --new: prompt for confirmation metomi/rose#1536

Closed

datamel force-pushed the cylc-install branch 2 times, most recently from 06deb04 to ea288a5 Compare January 19, 2021 10:38

oliver-sanders mentioned this pull request Jan 19, 2021

Cli setting of rose opts #4023

Merged

3 tasks

datamel force-pushed the cylc-install branch from eb9c151 to 9a60de5 Compare January 20, 2021 13:23

oliver-sanders reviewed Jan 21, 2021

View reviewed changes

cylc/flow/suite_files.py Outdated Show resolved Hide resolved

oliver-sanders reviewed Jan 25, 2021

View reviewed changes

.github/workflows/test_functional.yml Outdated Show resolved Hide resolved

oliver-sanders reviewed Jan 25, 2021

View reviewed changes

tests/functional/cylc-install/00-simple.t Show resolved Hide resolved

datamel force-pushed the cylc-install branch from b2bbb0d to d78d790 Compare January 26, 2021 10:54

datamel requested a review from hjoliver January 26, 2021 10:54

oliver-sanders reviewed Jan 26, 2021

View reviewed changes

tests/functional/authentication/01-remote-suite-same-name.t Outdated Show resolved Hide resolved

oliver-sanders reviewed Jan 26, 2021

View reviewed changes

cylc/flow/suite_files.py Outdated Show resolved Hide resolved

cylc/flow/suite_files.py Show resolved Hide resolved

cylc/flow/suite_files.py Outdated Show resolved Hide resolved

datamel and others added 11 commits January 26, 2021 20:59

cylc install

5495920

Update functional tests with cylc install

629bd09

Install logging

a25cc3e

Cylc install, prohibit source in cylc-run

cylc install tests

ca20517

Add cylc-install to cylc-clean

Rose-cylc install fix

f57b5ae

Add functional tests for cylc-install

Update cylc/flow/suite_files.py

74f0d61

Co-authored-by: Oliver Sanders <oliver.sanders@metoffice.gov.uk>

Ammend directory structure for cylc-install

0f6f89b

Fix functional test for cylc-install

Cylc install - get suite_source_dir adapted, fix functional tests

ba860e4

Fix get_flow_file Fix unit test

tests: make purge safer

6401d78

* abort if no flow is to be purged * only wait on lsof for a maximum of five tries

Raise error with mix of cylc install options

f79b564

Update yml timeout

Refactor install, add cylc install bash test

6ce05cc

datamel force-pushed the cylc-install branch from 6789961 to 6ce05cc Compare January 26, 2021 21:00

datamel added 2 commits January 27, 2021 00:16

Add suite.rc deprecation tests

ba17a8a

Update tests/f/authentication/01

0d13918

oliver-sanders approved these changes Jan 27, 2021

View reviewed changes

hjoliver approved these changes Jan 31, 2021

View reviewed changes

hjoliver merged commit 288a6f2 into cylc:master Jan 31, 2021

datamel deleted the cylc-install branch February 1, 2021 09:12

datamel mentioned this pull request Feb 10, 2021

Cylc Reinstall #4071

Merged

7 tasks

dpmatthews added this to the cylc-8.0b0 milestone Apr 27, 2021

oliver-sanders mentioned this pull request Aug 12, 2021

Reloading a suite with files removed doesn't remove such files from the cylc run directory metomi/rose#2477

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cylc install #4000

Cylc install #4000

datamel commented Dec 16, 2020 •

edited

Loading

hjoliver commented Dec 16, 2020

oliver-sanders commented Dec 16, 2020

wxtim commented Dec 17, 2020

oliver-sanders commented Dec 29, 2020 •

edited

Loading

oliver-sanders commented Dec 30, 2020

oliver-sanders commented Dec 30, 2020 •

edited

Loading

oliver-sanders commented Jan 6, 2021

oliver-sanders Jan 7, 2021

oliver-sanders commented Jan 13, 2021

datamel commented Jan 13, 2021

oliver-sanders commented Jan 19, 2021 •

edited

Loading

oliver-sanders left a comment

datamel commented Jan 22, 2021

oliver-sanders commented Jan 22, 2021

oliver-sanders left a comment

oliver-sanders left a comment

oliver-sanders left a comment

hjoliver commented Jan 27, 2021

hjoliver left a comment

		return rsync_cmd


		def install_workflow(flow_name=None, source=None, run_name=None,

Cylc install #4000

Cylc install #4000

Conversation

datamel commented Dec 16, 2020 • edited Loading

hjoliver commented Dec 16, 2020

oliver-sanders commented Dec 16, 2020

wxtim commented Dec 17, 2020

oliver-sanders commented Dec 29, 2020 • edited Loading

oliver-sanders commented Dec 30, 2020

oliver-sanders commented Dec 30, 2020 • edited Loading

oliver-sanders commented Jan 6, 2021

oliver-sanders Jan 7, 2021

Choose a reason for hiding this comment

oliver-sanders commented Jan 13, 2021

datamel commented Jan 13, 2021

oliver-sanders commented Jan 19, 2021 • edited Loading

oliver-sanders left a comment

Choose a reason for hiding this comment

datamel commented Jan 22, 2021

oliver-sanders commented Jan 22, 2021

oliver-sanders left a comment

Choose a reason for hiding this comment

oliver-sanders left a comment

Choose a reason for hiding this comment

oliver-sanders left a comment

Choose a reason for hiding this comment

hjoliver commented Jan 27, 2021

hjoliver left a comment

Choose a reason for hiding this comment

datamel commented Dec 16, 2020 •

edited

Loading

oliver-sanders commented Dec 29, 2020 •

edited

Loading

oliver-sanders commented Dec 30, 2020 •

edited

Loading

oliver-sanders commented Jan 19, 2021 •

edited

Loading