Cli setting of rose opts #4023

wxtim · 2021-01-05T16:59:31Z

Summary

These changes partially address #3819
These changes are downstream of changes in (cylc-rose:#20)[https://github.com/cylc/cylc-rose/pull/20]. The PR attempts to add command line options similar to the following rose suite-run options:

--opt-conf-key
--defines
--suite-defines

Checklist

Appropriate tests are included (unit and/or functional).
Appropriate change log entry included.
Documentation written as part of CLI, but not in Cylc docs. Is this OK for now?

cylc/flow/scripts/install.py

TomekTrzeciak · 2021-01-11T12:13:19Z

--opt-conf-key

--defines

--suite-defines

Sorry if I'm missing some context here, but would be the rationale of porting over --suite-defines instead of adding --set & --set-file to cylc install and making that work (#3913 made cylc and rose suite vars compatible)?

oliver-sanders · 2021-01-11T13:09:18Z

@TomekTrzeciak responded on the issue - #3819 (comment)

oliver-sanders · 2021-01-19T11:20:38Z

$ cylc install .
PluginError: Error in plugin cylc.pre_configure.rose
'NoneType' object has no attribute 'opt_conf_keys'

oliver-sanders · 2021-01-19T11:27:40Z

Ok, got it.

The pre_configure plugin hooks in cylc-flow are not passing the opts kwarg to the implementations in cylc-rose.

The plugin hook in cylc.flow.install should provide this argument.
- This is required so that opt configs can effect file installation etc.
The plugin hook in cylc.flow.parsec.fileparse shouldn't provide this argument.
- The flow config is not read in during installation (however, the global config is which is currently failing).

wxtim · 2021-01-19T17:00:44Z

@oliver-sanders I think I've excluded rose-suite-cylc-install.conf from being rsynced.

wxtim · 2021-01-20T09:51:23Z

Conflicted after merge with the cylc-install origin branch. Therefore tests aren't running.

wxtim · 2021-01-26T09:52:47Z

Due to the way that this branch is effectively built on top of @datamel 's my merges to keep it updated have resulted in a horrid looking history. Given that the final diff is likely to be fairly small it's my intention to do a git diff with master once the cylc-install branch is merged and re-create this PR branch from that diff so that the history looks less awful.

oliver-sanders · 2021-01-26T10:31:49Z

Could try squashing your commits then you can cherry-pick them/it on top.

datamel · 2021-01-26T13:00:39Z

Due to the way that this branch is effectively built on top of @datamel 's my merges to keep it updated have resulted in a horrid looking history. Given that the final diff is likely to be fairly small it's my intention to do a git diff with master once the cylc-install branch is merged and re-create this PR branch from that diff so that the history looks less awful.

Yep, sorry my cylc-install history looks so terrible! Should look a bit better now I have rebased.

wxtim · 2021-02-01T10:18:40Z

Outstanding test failure will stop failing when cylc-rose PR20 is merged to master. tests/functional/cylc-clean/00-basic.t and tests/functions/cylc-install/03...t contain a test for the presence of cylc-rose and provides extra lines for the reference test output reflecting the new capabilities of cylc-rose. The tests only have the master branch installed not cylc-rose PR20, so the tests fail. I think that this OK.

datamel

I have read through the code and had a play around with it. I am happy with this PR, although I think consider adding functional tests cylc-flow side.

tests/functional/cylc-clean/00-basic.t

cylc/flow/scripts/install.py

oliver-sanders · 2021-02-04T13:55:26Z

Tests passing with cylc-rose PR20 with latest commits.

oliver-sanders · 2021-02-04T13:58:31Z

tests/functional/cylc-clean/00-basic.t

+export ROSE_FILES=''
+python -c "import cylc.rose" > /dev/null 2>&1 &&
+    export ROSE_FILES="
+                    |-- opt
+                    |   \`-- rose-suite-cylc-install.conf
+                    |-- rose-suite.conf"


Tested with and without cylc-rose installed, all good.

oliver-sanders · 2021-02-04T13:59:44Z

tests/functional/cylc-install/03-file-transfer.t

+export ROSE_FILES=''
+if python -c "import cylc.rose" > /dev/null 2>&1; then
+    export ROSE_FILES="├── log
+│   └── install
+├── opt
+│   └── rose-suite-cylc-install.conf
+└── rose-suite.conf"
+else
+    export ROSE_FILES="└── log
+    └── install"
+fi
+
+tree -a -v -I '*.log|03-file-transfer*' --charset UTF8 --noreport "${RND_SUITE_RUNDIR}/" > 'basic-tree.out'


Tested with and without cylc-rose all good.

oliver-sanders

All good @datamel could you give this a quick once-over.

datamel

I have re-run the tests and read through the code changes since the last review. LGTM, thanks @wxtim

wxtim marked this pull request as draft January 5, 2021 16:59

wxtim added the rose-suite-run label Jan 5, 2021

wxtim force-pushed the cli-setting-of-rose-opts branch from ef70293 to d6a5461 Compare January 6, 2021 10:33

oliver-sanders reviewed Jan 7, 2021

View reviewed changes

cylc/flow/scripts/install.py Outdated Show resolved Hide resolved

oliver-sanders mentioned this pull request Jan 11, 2021

rose suite-run: plugin for rose-suite.conf functionality #3819

Closed

11 tasks

wxtim marked this pull request as ready for review January 18, 2021 10:54

wxtim marked this pull request as draft January 18, 2021 10:55

wxtim marked this pull request as ready for review January 18, 2021 16:21

wxtim requested a review from datamel January 18, 2021 16:21

wxtim self-assigned this Jan 18, 2021

wxtim requested a review from oliver-sanders January 18, 2021 16:22

wxtim marked this pull request as draft January 18, 2021 16:32

oliver-sanders mentioned this pull request Jan 19, 2021

Create cylc config save file (opt/rose-suite-cylc-install.conf) cylc/cylc-rose#20

Merged

10 tasks

wxtim marked this pull request as ready for review January 20, 2021 12:01

wxtim force-pushed the cli-setting-of-rose-opts branch 2 times, most recently from 7e2e00c to ce0baa9 Compare January 27, 2021 12:11

Enable rose-fileinstall-recording

4e9dd74

wxtim force-pushed the cli-setting-of-rose-opts branch from ce0baa9 to 4e9dd74 Compare February 1, 2021 09:31

datamel reviewed Feb 1, 2021

View reviewed changes

tests/functional/cylc-clean/00-basic.t Show resolved Hide resolved

cylc/flow/scripts/install.py Show resolved Hide resolved

run pre_configure properly if cylc install [dir] not given

627054a

wxtim requested a review from datamel February 2, 2021 08:34

wxtim changed the title ~~Cli setting of rose opts~~ [Test failure may be OK] Cli setting of rose opts Feb 2, 2021

fix test with dependency on tree

eda0b0c

wxtim force-pushed the cli-setting-of-rose-opts branch from 1a8e5e6 to eda0b0c Compare February 2, 2021 10:22

shellcheck fixed

cb21b2b

wxtim force-pushed the cli-setting-of-rose-opts branch from c95f2ab to cb21b2b Compare February 2, 2021 10:42

Merge branch 'master' into cli-setting-of-rose-opts

d7f6cb1

wxtim mentioned this pull request Feb 4, 2021

Incremental file install not working if previous source re-used metomi/rose#2438

Closed

oliver-sanders changed the title ~~[Test failure may be OK] Cli setting of rose opts~~ Cli setting of rose opts Feb 4, 2021

oliver-sanders reviewed Feb 4, 2021

View reviewed changes

oliver-sanders approved these changes Feb 4, 2021

View reviewed changes

datamel approved these changes Feb 5, 2021

View reviewed changes

datamel merged commit 9bdd5c6 into cylc:master Feb 5, 2021

wxtim deleted the cli-setting-of-rose-opts branch March 22, 2022 17:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cli setting of rose opts #4023

Cli setting of rose opts #4023

wxtim commented Jan 5, 2021 •

edited

Loading

TomekTrzeciak commented Jan 11, 2021

oliver-sanders commented Jan 11, 2021

oliver-sanders commented Jan 19, 2021

oliver-sanders commented Jan 19, 2021 •

edited

Loading

wxtim commented Jan 19, 2021

wxtim commented Jan 20, 2021

wxtim commented Jan 26, 2021

oliver-sanders commented Jan 26, 2021 •

edited

Loading

datamel commented Jan 26, 2021

wxtim commented Feb 1, 2021 •

edited

Loading

datamel left a comment

oliver-sanders commented Feb 4, 2021 •

edited

Loading

oliver-sanders Feb 4, 2021

oliver-sanders Feb 4, 2021

oliver-sanders left a comment

datamel left a comment

Cli setting of rose opts #4023

Cli setting of rose opts #4023

Conversation

wxtim commented Jan 5, 2021 • edited Loading

Summary

Checklist

TomekTrzeciak commented Jan 11, 2021

oliver-sanders commented Jan 11, 2021

oliver-sanders commented Jan 19, 2021

oliver-sanders commented Jan 19, 2021 • edited Loading

wxtim commented Jan 19, 2021

wxtim commented Jan 20, 2021

wxtim commented Jan 26, 2021

oliver-sanders commented Jan 26, 2021 • edited Loading

datamel commented Jan 26, 2021

wxtim commented Feb 1, 2021 • edited Loading

datamel left a comment

Choose a reason for hiding this comment

oliver-sanders commented Feb 4, 2021 • edited Loading

oliver-sanders Feb 4, 2021

Choose a reason for hiding this comment

oliver-sanders Feb 4, 2021

Choose a reason for hiding this comment

oliver-sanders left a comment

Choose a reason for hiding this comment

datamel left a comment

Choose a reason for hiding this comment

wxtim commented Jan 5, 2021 •

edited

Loading

oliver-sanders commented Jan 19, 2021 •

edited

Loading

oliver-sanders commented Jan 26, 2021 •

edited

Loading

wxtim commented Feb 1, 2021 •

edited

Loading

oliver-sanders commented Feb 4, 2021 •

edited

Loading