-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rose suite-run: plugin for rose-suite.conf functionality #3819
Comments
The big question: where do we put this?
I'm happy with whatever, it might be easier for the moment to develop in Cylc Flow and transfer later. The entry point should keep the codebases sufficiently separate. |
(I think the questions in the Issue description are mainly aimed at the UK Rose experts).
I'm not too bothered either, if you want to do in cylc-flow now then move it out later. |
This seems the most logical to me, at least for the time being. |
Due to the current implementation there's a significant difference between rose and cylc suite variables in that rose ones are treated as Jinja2 code, whereas cylc ones only as strings. Perhaps it would make sense to look first at implementing equivalent of #3169 to get these two on equal footing and treat them the same.
Reload is often used to update files in the suite, so I'd say yes. Unless you want to use this as an "incentive" for people to migrate off rose 😉. |
Under the hood Cylc passes these variables straight through to Jinja2, it's just the CLI that enforces string arguments so this approach should support typing from the off. |
Not enough. If you're not going to write the rose vars into the template any more, you'll need to |
Well yes, we will have to eval them so that we are passing Python objets rather than strings (we won't be able to just strip away the outer quotes and let Jinja2 implicitly eval them). This shouldn't be an issue with the internal interface. |
I had a proof-of-concept go at implementing the basic functionality of the The logic to support optional configs is present in the function Also, I haven't tried the Any thoughts. |
The At present anything you configure just gets dumped straight into the Had a quick look at the jinja2 code not expecting any surprises from that. >>> from ast import literal_eval
>>> literal_eval('42')
42
>>> literal_eval('True')
True
>>> literal_eval('"True"')
'True'
>>> literal_eval('[1, 2, 3]')
[1, 2, 3]
>>> literal_eval("{'a': 1, 'b': 2, 'c': 3}")
{'a': 1, 'b': 2, 'c': 3} |
I think that I have now got the literal eval working in that branch. |
Further details on how the
|
For reference here is some example output (suite.rc) showing the Jinja2 and environment meddling that Rose currently does and the default variables it provides:
|
I guess we can't prevent this, even though it is not advisable, except by not exporting |
It only works if we add these variables to the environment before we start to process the |
Is this still in the plan #3169 (comment):
I would really like to be able to run a generic, custom logic in a standard way at start up, not just |
Not officially tagged against Cylc8, however, pretty trivial to implement using the interface provided by this plugin. |
With |
A lot of people use this so we will need to find a way to supporting (Note it's not worth building too-advanced an interface for it now as the CLI library is likely to change) |
Interesting challenge for CLI design... |
Yeah not the nicest, perhaps (one day) we should use a load method interface for plugins, |
Assigning @wxtim as he has some some prelim investigation work with Jinja2 vars. |
Optional Conf KeysInstallationWith Rose 2019.x optional config keys can be configured in three ways:
All three should remain supported in Cylc8/Rose2.
Re-InstallationSubsequent re-installation should preserve optional configs specified by previous installations. I.E: cat > rose-suite.conf <<< opts=foo
cylc install -O bar -O baz
# optional configs foo, bar and baz applied
cylc reinstall
# optional configs foo, bar and baz still applied However, additional CLI options will be provided to permit the removal of targeted optional configs or permit rebuilding the list from scratch (as done by the original install). I.E: cylc reinstall -N bar
# optional configs foo and baz still applied, bar removed cylc reinstall --rebuild-opt-confs
# optional config foo applied (as it is specified in the rose-suite.conf)
# bar and baz removed (as they were specified by other means) CLI arguments TBC but should be something along the lines of:
|
@TomekTrzeciak - In reply to #4023 (comment) We are planning to keep Rose and Cylc templatevars separate but have unified their syntax as much as possible:
We have opted not to unify Rose and Cylc template variables because they are not directly equivalent and don't quite have the same scope: Unifying Rose and Cylc templatevars is not all that straight forward and I'm not entirely sure it makes sense:
More context:
For more information see:
|
@wxtim closed by cylc-rose PRs? |
cylc/cylc-rose#37 to replace outstanding section of this issue. |
Superseedes: #2960
Related to: #1885
Write a Cylc plugin which detects the
rose-suite.conf
file and acts accordingly.This plugin will call out to Rose2 libraries, this functionality already exists in the Rose it's just a matter of injecting it in the correct places in the Cylc logic.
The
rose-suite.conf
FileThe
rose-suite.conf
file has the following sections:opts
- Optional configuration keys.root-dir
- Affects suite installation, symlinks Cylc directories.[env]
- Environment variables.[jinja2:suiterc]
- Jinja2 variables.[empy:suiterc]
- Empy variavles.[file:*]
- File installation.opts
Rose should use this variable to determine which optional configuration files to load. This is already implemented in the existing Rose configuration logic so shouldn't require any extra work on the Cylc side.
root-dir
Obsolete, replaced by #3688, log a warning and do nothing.
Questions:
Should we make an attempt to map this onto the new CylcNo.[symlink dirs][<install target>]run
configs?[env]
Export these environment variables into the Cylc environment (i.e.os.environ[key] = var
).For now the path of least resistance is probably to just go ahead and meddle with the
flow.cylc
file, we can keep this logic in the Cylc part (rather than the plugin part) and improve it later, perhaps with a new Cylc options file or an environment variable table in the database (to match the template variables table).Questions:
How is this supposed to work, Cylc may re-execute on another host?![cylc][evironment]
flow.cylc
file (really don't want to go down that route again)?[jinja2:suiterc]
With cylc7/rose-2019 Rose prepends these variables to the top of the
suite.rc
file. It should not be necessary to edit any Cylc files to achieve this in Cylc8.[empy:suiterc]
Same as for Jinja2.
flow.cylc
file to make sure therose-suite.conf
file is configured for the correct template parser and raise an error if it is not or if no template parser is being used.[file:*]
Call out to the Rose file installation system as the current
rose suite-run
logic does.Questions:
Should we perform file installation logic on reloads?YesWhere To Inject This Logic In Cylc
At a glance I think the following commands will require loading the
rose-suite.conf
:opts
jinja2:suite.rc
empy:suite.rc
cylc run|restart
command has been re-executed on the suite server (if the suite server is not localhost)Implementation
Create two new Python entry points to keep this functionality separate from Cylc:
[env]
) and template ([*:suite.rc]
) variables.[file:*]
) for suite installation.The plugin interfaces should probably look something like this:
The plugins should output any required information using the Cylc log.
This work can be attempted in stages implementing one section at a time. The
[env]
section is probably the easiest to get started with.Checklist
Plugins:
Commands:
cylc get-suite-config
cylc run
cylc restart
cylc validate
cylc view
cylc diff
cylc view
cylc graph
- perhaps skip this one as the command only exists for test functionalitycylc reload
- perhaps leave this till last as it may be a bit more involvedThe text was updated successfully, but these errors were encountered: