Create cylc config save file (`opt/rose-suite-cylc-install.conf`) #20

wxtim · 2021-01-05T16:51:03Z

Summary

Partial response to cylc-flow:#3819
Required by a future PR to cylc-flow adding command line options similar to the following rose suite-run options:

--opt-conf-key
--defines
--suite-defines

Some auto tests will fail if Cylc-flow is master.

Checklist

Outstanding questions

Protected filenames:

Configuration would be written to an optional configuration called cylc-install. The cylc install command would exclude this file from the rsync in order to protect it.

cylc install proposal

Is this the rose-suite-cylc-install.conf file. Does this need doing in cylc/cylc-flow?

cylc/rose/rose.py

oliver-sanders · 2021-01-15T10:58:34Z

I think this should close #3819 wdyt?

wxtim · 2021-01-18T12:31:52Z

I didn't enjoy rebasing this - I think that there is scope for tidying up the testing a little before release, but if things are functional I should like to have this PR merged.

oliver-sanders · 2021-01-19T11:11:31Z

Is this the rose-suite-cylc-install.conf file. Does this need doing in cylc/cylc-flow?

Yes, and yes.

This needs adding to the cylc-install rsync exclude list.

oliver-sanders · 2021-01-19T11:43:34Z

cylc install is erroring because the opts kwarg is not always passed through to the plugin and the code doesn't check is opts before trying to access attributes of opts.

cylc/cylc-flow#4023 (comment)

tests/test_record_cylc_install_options.py

wxtim · 2021-01-19T16:26:17Z

@oliver-sanders and @datamel Ready for review. Sorry about all the messing about.

cylc/rose/entry_points.py

oliver-sanders · 2021-01-21T15:41:01Z

Any new “defines” should be written to a new optional configuration with a special protected name

✅ Tested -D and -S.
✅ The (cylc-install) config is added to the end of the list.

The cylc reinstall command should always move the cylc-install opt conf to the end of the list

⌛ Will test the reinstall part once we have a reinstall command.

The cylc reinstall command should have an option to delete the cylc-install opt conf

⌛ Saving this one for a future PR.

cylc install should record optional configs to the cylc-install optional config

❌ via -O CLI opt.
✅ via ROSE_SUITE_OPT_CONF_KEYS env var.
❌ cylc-install|opts are appended to rose-suite.conf|opts

my test suite:

$ find .  -type f -name '*.conf' -exec echo -e '\n=== {} ===' \; -exec cat '{}' \;

=== ./opt/rose-suite-foo.conf ===
[jinja2:suite.rc]
Y="f"

=== ./opt/rose-suite-bar.conf ===
[jinja2:suite.rc]
Y="b"

=== ./rose-suite.conf ===
opts=

[jinja2:suite.rc]
Y="base"

my test install:

$ ROSE_SUITE_OPT_CONF_KEYS=foo cylc install -O bar -D '[env]a=b' -S 'a="b"'
INSTALLED foo from ~/roses/foo -> ~/cylc-run/foo/run1

the result:

$ find ~/cylc-run/foo/runN/ -type f -name '*.conf' -exec echo -e '\n=== {} ===' \; -exec cat '{}' \;

=== ~/cylc-run/foo/runN//opt/rose-suite-foo.conf ===
[jinja2:suite.rc]
Y="f"

=== ~/cylc-run/foo/runN//opt/rose-suite-bar.conf ===
[jinja2:suite.rc]
Y="b"

=== ~/cylc-run/foo/runN//opt/rose-suite-cylc-install.conf ===
# This file records CLI Options.

!opts=foo

[env]
a=b

[jinja2:suite.rc]
a="b"

=== ~/cylc-run/foo/runN//rose-suite.conf ===
# Config Options '(cylc-install)' from CLI appended to options already in `rose-suite.conf`.
!opts=(cylc-install)

[jinja2:suite.rc]
Y="base"

the diff:

 === ~/cylc-run/foo/runN//opt/rose-suite-cylc-install.conf ===
 # This file records CLI Options.

-!opts=foo
+!opts=bar foo

 [env]
 a=b

 [jinja2:suite.rc]
 a="b"

 === ~/cylc-run/foo/runN//rose-suite.conf ===
 # Config Options '(cylc-install)' from CLI appended to options already in `rose-suite.conf`.
-!opts=(cylc-install)
+opts=bar foo (cylc-install)

 [jinja2:suite.rc]
 Y="base"

oliver-sanders · 2021-01-25T16:57:27Z

rose-suite.conf|opts setting is still ignored (which means it won't be loaded by metomi.rose.config_tree).

Co-authored-by: Oliver Sanders <oliver.sanders@metoffice.gov.uk>

cylc/rose/entry_points.py

oliver-sanders · 2021-02-02T09:26:41Z

$ cylc install
...
PluginError: Error in plugin cylc.post_install.rose_files
can only concatenate str (not "bytes") to str

oliver-sanders · 2021-02-03T10:17:10Z

cylc-flow: tim/cli-setting-of-rose-opts@d7f6cb18214776f63a336af1f478f584e1827192
cylc-rose: tim/create-cylc-config-save-file@6e7e0172bcaba225dc4961406f89e661d7c6853a

$ cylc install
...
PluginError: Error in plugin cylc.post_install.rose_files
can only concatenate str (not "bytes") to str

wxtim · 2021-02-04T10:48:33Z

cylc-flow: tim/cli-setting-of-rose-opts@d7f6cb18214776f63a336af1f478f584e1827192
cylc-rose: tim/create-cylc-config-save-file@6e7e0172bcaba225dc4961406f89e661d7c6853a
$ cylc install
...
PluginError: Error in plugin cylc.post_install.rose_files
can only concatenate str (not "bytes") to str

I just can't replicate this @oliver-sanders

oliver-sanders · 2021-02-04T10:56:48Z

Here's an example flow:

$ find . -type f -exec echo -e '\n# {}\n' \; -exec cat '{}' \;

# ./flow.cylc


# ./opt/rose-suite-x.conf

[env]
FILENAME=x

# ./rose-suite.conf

[env]
FILENAME=base

[file:myfile]
source=localhost:$PWD/$FILENAME

# ./x

x

# ./base

base

For the record it turns out this was caused by the localhost: prefix on the source. This works (or at least parses) in Rose2019, however, does not at Rose2. Not related to this PR, a Rose issue will follow.

oliver-sanders · 2021-02-04T13:01:36Z

Opt conf key stuff looks good, for interest here is my test:

$ lsd
# ./opt/rose-suite-y.conf
[env]
O=Y

# ./opt/rose-suite-x.conf
[env]
O=X

# ./suite.rc

# ./rose-suite.conf
[env]
O=B



# base conf installed ✅
$ cylc clean conf-test; cylc install
$ rose config --file ~/cylc-run/conf-test/runN/rose-suite.conf env O
B

# opt conf overrides base conf ✅
$ cylc clean conf-test; cylc install -O x
$ rose config --file ~/cylc-run/conf-test/runN/rose-suite.conf env O
X

# ROSE_SUITE_OPT_CONF_KEYS override base conf in the correct order ✅
$ cylc clean conf-test; ROSE_SUITE_OPT_CONF_KEYS='x y' cylc install
$ rose config --file ~/cylc-run/conf-test/runN/rose-suite.conf env O
Y

# -O applied after ROSE_SUITE_OPT_CONF_KEY ✅
$ cylc clean conf-test; ROSE_SUITE_OPT_CONF_KEYS='x y' cylc install -O x
$ rose config --file ~/cylc-run/conf-test/runN/rose-suite.conf env O
X

# -O applied in order ✅
$ cylc clean conf-test; ROSE_SUITE_OPT_CONF_KEYS='x y' cylc install -O x -O y
$ rose config --file ~/cylc-run/conf-test/runN/rose-suite.conf env O
Y

oliver-sanders · 2021-02-04T15:39:33Z

Any new “defines” should be written to a new optional configuration with a special protected name

✅ Tested -D and -S.
✅ The (cylc-install) config is added to the end of the list.

The cylc reinstall command should always move the cylc-install opt conf to the end of the list

⌛ Will test the reinstall part once we have a reinstall command.

The cylc reinstall command should have an option to delete the cylc-install opt conf

⌛ Saving this one for a future PR.

cylc install should record optional configs to the cylc-install optional config

✅ via -O CLI opt.
✅ via ROSE_SUITE_OPT_CONF_KEYS env var.
✅ cylc-install|opts are appended to rose-suite.conf|opts

oliver-sanders

All good except one last thing :(

The .rose-config_processors-file.db gets created in the source directory rather than the run directory.

This means that file installation from one installation can effect file installation in another.

It may be necessary to cd into the run dir before the file installation then back again afterwards (perhaps in a try/finally).

Otherwise all good, @datamel please review when you get time.

wxtim · 2021-02-04T16:53:53Z

The .rose-config_processors-file.db gets created in the source directory rather than the run directory.
I believe that I have now fixed this. :)

cylc/rose/entry_points.py

oliver-sanders · 2021-02-05T11:12:15Z

Good to go here!

oliver-sanders

@datamel could you review 2.

datamel · 2021-02-01T12:25:24Z

cylc/rose/entry_points.py

+            Cylc option parser object - we want to extract the following
+            values:
+            - opt_conf_keys (list or str):
+                Equivelent of ``rose suite-run --option KEY``


Typo

Suggested change

Equivelent of ``rose suite-run --option KEY``

Equivalent of ``rose suite-run --option KEY``

datamel · 2021-02-01T12:25:34Z

cylc/rose/entry_points.py

+            - defines (list of str):
+                Equivelent of ``rose suite-run --define KEY=VAL``
+            - suite_defines (list of str):
+                Equivelent of ``rose suite-run --define-suite KEY=VAL``


Suggested change

Equivelent of ``rose suite-run --define-suite KEY=VAL``

Equivalent of ``rose suite-run --define-suite KEY=VAL``

…nal-tests * master: [Test failure may be OK] Create cylc config save file (cylc#20)

…se-conf * master: [Test failure may be OK] Create cylc config save file (cylc#20)

wxtim self-assigned this Jan 5, 2021

wxtim marked this pull request as draft January 5, 2021 16:51

wxtim requested a review from oliver-sanders January 5, 2021 16:51

wxtim force-pushed the create-cylc-config-save-file branch from c8e649e to 375cb92 Compare January 5, 2021 16:56

wxtim mentioned this pull request Jan 5, 2021

Cli setting of rose opts cylc/cylc-flow#4023

Merged

3 tasks

wxtim force-pushed the create-cylc-config-save-file branch from cdc19d5 to 4636222 Compare January 6, 2021 14:03

wxtim marked this pull request as ready for review January 6, 2021 14:04

oliver-sanders reviewed Jan 7, 2021

View reviewed changes

cylc/rose/rose.py Outdated Show resolved Hide resolved

wxtim marked this pull request as draft January 8, 2021 16:13

wxtim requested review from oliver-sanders and datamel January 15, 2021 09:14

wxtim marked this pull request as ready for review January 15, 2021 09:14

wxtim commented Jan 19, 2021

View reviewed changes

tests/test_record_cylc_install_options.py Outdated Show resolved Hide resolved

wxtim force-pushed the create-cylc-config-save-file branch 2 times, most recently from 46507cf to 479c87d Compare January 20, 2021 12:03

wxtim mentioned this pull request Jan 20, 2021

actions: test code coverage #17

Merged

oliver-sanders reviewed Jan 20, 2021

View reviewed changes

cylc/rose/entry_points.py Outdated Show resolved Hide resolved

cylc/rose/entry_points.py Show resolved Hide resolved

cylc/rose/entry_points.py Outdated Show resolved Hide resolved

cylc/rose/entry_points.py Show resolved Hide resolved

wxtim force-pushed the create-cylc-config-save-file branch from eeb3429 to c23bfbf Compare January 21, 2021 10:50

oliver-sanders reviewed Jan 21, 2021

View reviewed changes

cylc/rose/entry_points.py Outdated Show resolved Hide resolved

wxtim requested a review from oliver-sanders January 21, 2021 13:39

support creation of rose-suite-cylc-install.conf

c2211c3

wxtim force-pushed the create-cylc-config-save-file branch from 615d87c to c2211c3 Compare January 27, 2021 11:59

fixed copyright message

8a69d4f

wxtim and others added 2 commits February 1, 2021 14:12

Update cylc/rose/utilities.py

8228d12

Co-authored-by: Oliver Sanders <oliver.sanders@metoffice.gov.uk>

responded to Mel's request for a functional test

ca07bc6

wxtim commented Feb 1, 2021

View reviewed changes

cylc/rose/entry_points.py Outdated Show resolved Hide resolved

Update cylc/rose/entry_points.py

6e7e017

wxtim requested a review from oliver-sanders February 1, 2021 16:52

wxtim changed the title ~~Create cylc config save file~~ [Test failure may be OK] Create cylc config save file Feb 2, 2021

oliver-sanders requested changes Feb 4, 2021

View reviewed changes

fixed location of .rose-config_processors-file.db to dest_root

33eed80

wxtim requested a review from oliver-sanders February 4, 2021 16:53

oliver-sanders reviewed Feb 4, 2021

View reviewed changes

cylc/rose/entry_points.py Outdated Show resolved Hide resolved

Update cylc/rose/entry_points.py

5439f1a

wxtim commented Feb 5, 2021

View reviewed changes

cylc/rose/entry_points.py Show resolved Hide resolved

Update cylc/rose/entry_points.py

e58c52f

wxtim requested a review from oliver-sanders February 5, 2021 12:14

oliver-sanders approved these changes Feb 5, 2021

View reviewed changes

datamel approved these changes Feb 5, 2021

View reviewed changes

datamel merged commit 8d64fd5 into cylc:master Feb 5, 2021

wxtim mentioned this pull request Feb 8, 2021

Fix the Copyright notices. #26

Closed

2 tasks

wxtim added a commit to wxtim/cylc-rose that referenced this pull request Feb 8, 2021

Merge branch 'master' into rose-fileinstall-recording.convert-functio…

0d7c88c

…nal-tests * master: [Test failure may be OK] Create cylc config save file (cylc#20)

wxtim added a commit to wxtim/cylc-rose that referenced this pull request Feb 8, 2021

Merge branch 'master' into create-cylc-config-save-file.create-log-ro…

ca4535c

…se-conf * master: [Test failure may be OK] Create cylc config save file (cylc#20)

wxtim changed the title ~~[Test failure may be OK] Create cylc config save file~~ Create cylc config save file (opt/rose-suite-cylc-install.conf) Mar 3, 2021

wxtim deleted the create-cylc-config-save-file branch March 3, 2021 08:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create cylc config save file (`opt/rose-suite-cylc-install.conf`) #20

Create cylc config save file (`opt/rose-suite-cylc-install.conf`) #20

wxtim commented Jan 5, 2021 •

edited by oliver-sanders

Loading

oliver-sanders commented Jan 15, 2021

wxtim commented Jan 18, 2021

oliver-sanders commented Jan 19, 2021

oliver-sanders commented Jan 19, 2021

wxtim commented Jan 19, 2021

oliver-sanders commented Jan 21, 2021 •

edited

Loading

oliver-sanders commented Jan 25, 2021

oliver-sanders commented Feb 2, 2021

oliver-sanders commented Feb 3, 2021

wxtim commented Feb 4, 2021

oliver-sanders commented Feb 4, 2021 •

edited

Loading

oliver-sanders commented Feb 4, 2021 •

edited

Loading

oliver-sanders commented Feb 4, 2021

oliver-sanders left a comment •

edited

Loading

wxtim commented Feb 4, 2021

oliver-sanders commented Feb 5, 2021 •

edited

Loading

oliver-sanders left a comment

datamel Feb 1, 2021

datamel Feb 1, 2021

	Equivelent of ``rose suite-run --option KEY``
	Equivalent of ``rose suite-run --option KEY``

	Equivelent of ``rose suite-run --define-suite KEY=VAL``
	Equivalent of ``rose suite-run --define-suite KEY=VAL``

Create cylc config save file (opt/rose-suite-cylc-install.conf) #20

Create cylc config save file (opt/rose-suite-cylc-install.conf) #20

Conversation

wxtim commented Jan 5, 2021 • edited by oliver-sanders Loading

Summary

Checklist

Outstanding questions

Protected filenames:

oliver-sanders commented Jan 15, 2021

wxtim commented Jan 18, 2021

oliver-sanders commented Jan 19, 2021

oliver-sanders commented Jan 19, 2021

wxtim commented Jan 19, 2021

oliver-sanders commented Jan 21, 2021 • edited Loading

oliver-sanders commented Jan 25, 2021

oliver-sanders commented Feb 2, 2021

oliver-sanders commented Feb 3, 2021

wxtim commented Feb 4, 2021

oliver-sanders commented Feb 4, 2021 • edited Loading

oliver-sanders commented Feb 4, 2021 • edited Loading

oliver-sanders commented Feb 4, 2021

oliver-sanders left a comment • edited Loading

Choose a reason for hiding this comment

wxtim commented Feb 4, 2021

oliver-sanders commented Feb 5, 2021 • edited Loading

oliver-sanders left a comment

Choose a reason for hiding this comment

datamel Feb 1, 2021

Choose a reason for hiding this comment

datamel Feb 1, 2021

Choose a reason for hiding this comment

Create cylc config save file (`opt/rose-suite-cylc-install.conf`) #20

Create cylc config save file (`opt/rose-suite-cylc-install.conf`) #20

wxtim commented Jan 5, 2021 •

edited by oliver-sanders

Loading

oliver-sanders commented Jan 21, 2021 •

edited

Loading

oliver-sanders commented Feb 4, 2021 •

edited

Loading

oliver-sanders commented Feb 4, 2021 •

edited

Loading

oliver-sanders left a comment •

edited

Loading

oliver-sanders commented Feb 5, 2021 •

edited

Loading