Automatically interpolate jinja variables from rose to cylc

[bschroeter]

Correct me if I am wrong but at the moment when using an optional configuration file there is a bit of double handing to bring jinja variables into a suite.rc file.

For example:

opt/rose-suite.conf

[jinja2:suite.rc]
MY_VARIABLE="Some value"

suite.rc

[environment]

MY_VARIABLE = {{MY_VARIABLE}}

In a quest for automation, I've played around with accessing the environment in the suite.rc to automatically interpolate any jinja2 variables straight into the environment section. However, it seems that the context is hard to access.

What would be great would be if there were the ability to "automatically import" all rose/jinja2 variables straight into the environment section with a jinja macro or similar.

So rather than the above, we could do something like this:

[environment]
{% for key, value in context.items() }%
{{key}} = {{value}}
{% endfor %}

or even...

[environment]
{% autoload_context %}

From a maintenance perspective, this would be a fantastic addition as we could then just edit configuration in one place, rather than having to ensure that variables are passed between syntax.

Also if there is a more straightforward way to do this that already exists, I'd be very keen to know!

Thanks,

Ben

[hjoliver]

Hi @bschroeter - I don't think I understand your use case. Do you want all rose-suite.conf suite Jinja2 variables available in [cylc][[environment]] or in task job environments? In the former case, what use would that be? (the suite server program's environment is not available to jobs); and in the latter case you would presumably be cluttering job environments with a bunch of un-needed variables as well as the few that are needed by the job. Or have I misunderstood you?!

[bschroeter]

Hi @hjoliver,

Thanks for the quick response.

The use case that I am trying to solve is the ability to define configuration variables in a rose .conf file and have them automatically interpolate into a cylc suite.rc definition. I've experimented with parsing the environment keys but you're right in that at that point the environment is different and doesn't contain the correct context. This took me down a path that mangled the execution environment so I don't recommend it!

To clarify my requirement, consider the following scenario...

If we had a need to set some things in the environment before executing (such as for abstraction between hosts), then we would do the following:

export MY_VAR="My super secret variable"
rose suite-run -O mysite

Then, in the corresponding opt/rose-suite-mysite.conf we would need to do the following to inform rose of the environment variable:

[jinja2:suite.rc]
MY_VAR="${MY_VAR}"

Lastly, we would then need to inform cylc via the suite.rc:

[[environment]]
MY_VAR = {{MY_VAR}}

These multiple handovers of variables from either the .conf or the environment works fine for one or two variables, but when there are dozens, or when the names of variables change, there are multiple places in which changes need to be made.

While this might not be a common use case, there is duplication of effort to ensure that jinja/rose variables are translated into cylc. What I am proposing is to define a macro that does this automatically.

After I posted this issue, I had another attempt and delved into the Jinja2 source code to find a way in and managed to come up with a (rather inelegant) solution.

in the .conf file

[jinja2:suite.rc]
AUTOLOAD_PREFIX="AUTO_"
AUTO_MYVAR=1234

in the suite.rc (ugly but works!)

# Variables automatically interpolated from Rose
{% set template_context = self.__dict__['_TemplateReference__context'] %}
{% for key, value in template_context.items() %}
{% if AUTOLOAD_PREFIX in key %}
{{key | replace(AUTOLOAD_PREFIX, '')}} = {{value}}
{% endif %}
{% endfor %}
# End automated interpolation

This results in a suite.rc.processed that meets my requirement!

[[environment]]
# Variables automatically interpolated from Rose
MYVAR = 1234
# End automated interpolation

(I'm using a removable prefix as it is simpler than checking a blacklist of keys we don't want to automatically interpolate)

However, I am sure that you will agree that this is a nasty piece of work that would benefit from a nice, jinja macro or tag that would take the 6 lines of mining the template context to something like:

[[environment]]
{% autoload_rose_context %}

I'm happy to branch off and implement this as I am well versed in Jinja templating, however, I'd need a pointer to where in the cylc code I should inject such logic.

[bschroeter]

Giving this a bit more thought, this is easily done with a macro...

opt/rose-suite-site.conf

[jinja2:suite.rc]
AUTO_MYVAR=1234
AUTO_MYOTHERVAR=True

Note the prefix of 'AUTO_', which is stripped from the processed file (but tells the macro you want to auto-load the variable)

macros.rc

{% macro autoload_rose_context(context) %}

# Variables automatically interpolated from Rose
{% set template_context = context.__dict__['_TemplateReference__context'] %}
{% for key, value in template_context.items() %}
{% if 'AUTO_' in key %}
{{key | replace('AUTO_', '')}} = {{value}}
{% endif %}
{% endfor %}
# End automated interpolation

{% endmacro %}

suite.rc

{% import 'macros.rc' as macros %}
...
[[environment]]
{{macros.autoload_rose_context(self)}}

Note that self must be passed to the macro to extract the context in the right scope

the resulting suite.rc.processed

[[environment]]
MYVAR=1234
MYOTHERVAR=True

Note that the 'AUTO_' prefix is stripped from the processed file

This method also makes it possible to bring things all the way from the environment (as in exported prior to rose suite-run) by using ${} syntax in the .conf file to abstract environmental configuration.

export MYVAR=1234
rose suite-run ...

[jinja2:suite.rc]
AUTO_MYVAR=${MYVAR}

Although a built-in mechanism to do this would still be nice.

Cheers, B

[hjoliver]

@bschroeter - nice work on the macro!

I can see how this might be useful, I think.

How about you just contribute your macro to become part of Cylc, with a little User Guide documentation on how to use it - would that be sufficient, in your view?

(As an aside, note that rose suite-run is being migrated from Rose to Cylc, although I don't think that's a problem for what you're suggesting ... we will still have some equivalent of rose-suite.conf for input Jinja variables).

[dwsutherland]

I've not seen it done like that, thanks for the insight - There's definitely a number of ways to skin this cat!

WRT the double handling (and someone can correct me), the reason for this separation into jinja2 destined rose-suite.conf is to enable the suite to be configurable by the rose edit interface (or other third party tools)..
You can in fact just use pure Jinja2 for most off what you want to achieve:

• You can still separate out settings by use of include files to add Jinja2 in to your suite.rc (which would avoid clutter and the additional format)

%include include/jinja2-conf.rc

• You can pull in the environment by using Jinja2 filters:
  Jinja2Filters/get_env_dic.py

#!/usr/bin/env python

# Get the current environment


def get_env_dic(ENV_DIC):
    import os
    ENV_DIC.update(os.environ)
    return ENV_DIC

(or specific vars of course)
And iterate the dictionary just as you postulated:

{% set ENV_DIC={} | get_env_dic %}
{% for ITEM, VALUE in ENV_DIC.items() %}
# {{ITEM}} = {{VALUE}}
{% endfor %}

Might be slightly on a tangent, but related pull-request:
#2963

[bschroeter]

Thanks @hjoliver, I'll have a go and send a PR.

@dwsutherland, similar concept which I stumbled onto during my experimentation. One thing that I'd note is that when I did this when executing over on a PBS host it made a real mess of the environment by interpolating everything into the cylc environment. Particularly when the $PATH and rose/cylc environment variables came in, I was left with a file that failed validation and caused a raft of other exceptions. However, this may have been a result of our execution environment so don't quote me on that.

Cheers, B

[dwsutherland]

@dwsutherland, similar concept which I stumbled onto during my experimentation. One thing that I'd note is that when I did this when executing over on a PBS host it made a real mess of the environment by interpolating everything into the cylc environment. Particularly when the $PATH and rose/cylc environment variables came in, I was left with a file that failed validation and caused a raft of other exceptions. However, this may have been a result of our execution environment so don't quote me on that.

Yes you'd definitely need to be selective, and you could filter the environment for a variable prefix, creating your own dictionary to iterate on..
(Hilary reminded me that environ is actually available directly in Jinja2:
`https://cylc.github.io/cylc/doc/built-sphinx/suite-config.html#accessing-environment-variables-with-jinja2
)

[TomekTrzeciak]

@bschroeter, I think what you want is a namespace object (dictionary) that contains all jinja2:suite.rc variables. Unfortunately, this dictionary is only available in rose suite-run and dumped from there directly into suite.rc file as separate variable definitions. So there is currently no easy way to access the entire content of jinja2:suite.rc section.

Below is a quick stab at how this could be implemented (untested!):

diff --git a/lib/python/rose/config_processors/jinja2.py b/lib/python/rose/config_processors/jinja2.py
index 02ba10ab..0cb48085 100644
--- a/lib/python/rose/config_processors/jinja2.py
+++ b/lib/python/rose/config_processors/jinja2.py
@@ -69,6 +69,7 @@ class ConfigProcessorForJinja2(ConfigProcessorBase):
             tmp_file = NamedTemporaryFile()
             tmp_file.write(scheme_ln)
             tmp_file.write(msg_init_ln)
+            suite_variables = ['{']
             for key, node in sorted(s_node.value.items()):
                 if node.is_ignored():
                     continue
@@ -77,6 +78,10 @@ class ConfigProcessorForJinja2(ConfigProcessorBase):
                 except UnboundEnvironmentVariableError as exc:
                     raise ConfigProcessError([s_key, key], node.value, exc)
                 tmp_file.write(self.ASSIGN_TEMPL % (key, value))
+                suite_variables.append("'%s': %s," % (key, key))
+            suite_variables.append('}')
+            tmp_file.write(self.ASSIGN_TEMPL % ('ROSE_SUITE_VARIABLES',
+                                                '\n'.join(suite_variables)))
             environ = kwargs.get("environ")
             if environ:
                 tmp_file.write('[cylc]\n')

All you need from there is to dump the content of ROSE_SUITE_VARIABLES dictionary containing all jinja2:suite.rc variables into cylc environment section:

            [[[environment]]]
{% for key, val in ROSE_SUITE_VARIABLES.items() %}
                {{key}} = {{val}}
{% endfor %}

[TomekTrzeciak]

@bschroeter, are you still interested in this issue? If so, would you be able to put up a PR for it? I think this would be a useful feature (I might actually have a use case for it as well).

[bschroeter]

@TomekTrzeciak, thanks for the message. Yes I'm still interested in implementing this functionality - I've been working on some non-scheduler work but coming back into it as part of another project shortly.

For reference, where would be the most appropriate place to inject this feature? I'd be tempted to find a hook just prior to processing the final step of jinja interpolation but I'm not sure where in the code this happens. Once I know where, this shouldn't take too long to implement.

Any advice would be greatly appreciated.

[TomekTrzeciak]

For reference, where would be the most appropriate place to inject this feature? I'd be tempted to find a hook just prior to processing the final step of jinja interpolation but I'm not sure where in the code this happens.

@bschroeter I think the best place for this is currently in rose rather than cylc, where you can't tell apart a variable coming from [jinja2:suite.rc] from the any other variable in the template (they are truly indistinguishable at that stage - perhaps something that could be reconsidered during planned migration of rose suite-run into cylc).

As for the implementation, see my suggestion above that exposes [jinja2:suite.rc] section as ROSE_SUITE_VARIABLES dictionary in the template.

[oliver-sanders]

Superseded by #3819 which will bring plugin-based support for the rose-suite.conf file into Cylc8.