-
Notifications
You must be signed in to change notification settings - Fork 93
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Automatically interpolate jinja variables from rose to cylc #2960
Comments
Hi @bschroeter - I don't think I understand your use case. Do you want all |
Hi @hjoliver, Thanks for the quick response. The use case that I am trying to solve is the ability to define configuration variables in a rose To clarify my requirement, consider the following scenario... If we had a need to set some things in the environment before executing (such as for abstraction between hosts), then we would do the following: export MY_VAR="My super secret variable"
rose suite-run -O mysite Then, in the corresponding [jinja2:suite.rc]
MY_VAR="${MY_VAR}" Lastly, we would then need to inform cylc via the [[environment]]
MY_VAR = {{MY_VAR}} These multiple handovers of variables from either the While this might not be a common use case, there is duplication of effort to ensure that jinja/rose variables are translated into cylc. What I am proposing is to define a macro that does this automatically. After I posted this issue, I had another attempt and delved into the Jinja2 source code to find a way in and managed to come up with a (rather inelegant) solution. in the .conf file [jinja2:suite.rc]
AUTOLOAD_PREFIX="AUTO_"
AUTO_MYVAR=1234 in the suite.rc (ugly but works!)
This results in a [[environment]]
# Variables automatically interpolated from Rose
MYVAR = 1234
# End automated interpolation (I'm using a removable prefix as it is simpler than checking a blacklist of keys we don't want to automatically interpolate) However, I am sure that you will agree that this is a nasty piece of work that would benefit from a nice, jinja macro or tag that would take the 6 lines of mining the template context to something like: [[environment]]
{% autoload_rose_context %} I'm happy to branch off and implement this as I am well versed in Jinja templating, however, I'd need a pointer to where in the cylc code I should inject such logic. |
Giving this a bit more thought, this is easily done with a macro... opt/rose-suite-site.conf [jinja2:suite.rc]
AUTO_MYVAR=1234
AUTO_MYOTHERVAR=True Note the prefix of 'AUTO_', which is stripped from the processed file (but tells the macro you want to auto-load the variable) macros.rc
suite.rc
Note that the resulting suite.rc.processed
Note that the 'AUTO_' prefix is stripped from the processed file This method also makes it possible to bring things all the way from the environment (as in exported prior to export MYVAR=1234
rose suite-run ... [jinja2:suite.rc]
AUTO_MYVAR=${MYVAR} Although a built-in mechanism to do this would still be nice. Cheers, B |
@bschroeter - nice work on the macro! I can see how this might be useful, I think. How about you just contribute your macro to become part of Cylc, with a little User Guide documentation on how to use it - would that be sufficient, in your view? (As an aside, note that |
I've not seen it done like that, thanks for the insight - There's definitely a number of ways to skin this cat! WRT the double handling (and someone can correct me), the reason for this separation into jinja2 destined
(or specific vars of course)
Might be slightly on a tangent, but related pull-request: |
Thanks @hjoliver, I'll have a go and send a PR. @dwsutherland, similar concept which I stumbled onto during my experimentation. One thing that I'd note is that when I did this when executing over on a PBS host it made a real mess of the environment by interpolating everything into the cylc environment. Particularly when the Cheers, B |
Yes you'd definitely need to be selective, and you could filter the environment for a variable prefix, creating your own dictionary to iterate on.. |
@bschroeter, I think what you want is a namespace object (dictionary) that contains all Below is a quick stab at how this could be implemented (untested!): diff --git a/lib/python/rose/config_processors/jinja2.py b/lib/python/rose/config_processors/jinja2.py
index 02ba10ab..0cb48085 100644
--- a/lib/python/rose/config_processors/jinja2.py
+++ b/lib/python/rose/config_processors/jinja2.py
@@ -69,6 +69,7 @@ class ConfigProcessorForJinja2(ConfigProcessorBase):
tmp_file = NamedTemporaryFile()
tmp_file.write(scheme_ln)
tmp_file.write(msg_init_ln)
+ suite_variables = ['{']
for key, node in sorted(s_node.value.items()):
if node.is_ignored():
continue
@@ -77,6 +78,10 @@ class ConfigProcessorForJinja2(ConfigProcessorBase):
except UnboundEnvironmentVariableError as exc:
raise ConfigProcessError([s_key, key], node.value, exc)
tmp_file.write(self.ASSIGN_TEMPL % (key, value))
+ suite_variables.append("'%s': %s," % (key, key))
+ suite_variables.append('}')
+ tmp_file.write(self.ASSIGN_TEMPL % ('ROSE_SUITE_VARIABLES',
+ '\n'.join(suite_variables)))
environ = kwargs.get("environ")
if environ:
tmp_file.write('[cylc]\n') All you need from there is to dump the content of [[[environment]]]
{% for key, val in ROSE_SUITE_VARIABLES.items() %}
{{key}} = {{val}}
{% endfor %} |
@bschroeter, are you still interested in this issue? If so, would you be able to put up a PR for it? I think this would be a useful feature (I might actually have a use case for it as well). |
@TomekTrzeciak, thanks for the message. Yes I'm still interested in implementing this functionality - I've been working on some non-scheduler work but coming back into it as part of another project shortly. For reference, where would be the most appropriate place to inject this feature? I'd be tempted to find a hook just prior to processing the final step of jinja interpolation but I'm not sure where in the code this happens. Once I know where, this shouldn't take too long to implement. Any advice would be greatly appreciated. |
@bschroeter I think the best place for this is currently in rose rather than cylc, where you can't tell apart a variable coming from As for the implementation, see my suggestion above that exposes |
Superseded by #3819 which will bring plugin-based support for the |
Correct me if I am wrong but at the moment when using an optional configuration file there is a bit of double handing to bring jinja variables into a suite.rc file.
For example:
opt/rose-suite.conf
suite.rc
In a quest for automation, I've played around with accessing the environment in the suite.rc to automatically interpolate any jinja2 variables straight into the environment section. However, it seems that the context is hard to access.
What would be great would be if there were the ability to "automatically import" all rose/jinja2 variables straight into the environment section with a jinja macro or similar.
So rather than the above, we could do something like this:
or even...
From a maintenance perspective, this would be a fantastic addition as we could then just edit configuration in one place, rather than having to ensure that variables are passed between syntax.
Also if there is a more straightforward way to do this that already exists, I'd be very keen to know!
Thanks,
Ben
The text was updated successfully, but these errors were encountered: