feat(controller): Expression template tags. Resolves #4548 & #1293

[alexec]

Closes #4585
Closes #1293

[crenshaw-dev]

I feel like this would justify a new docs page or maybe an example or two. Looks like the only reference to expr so far is in the events docs.

[alexec]

@saranyaeu2987 @chermed @max-sixty @discordianfish @Ark-kun @iven @lucastheisen you've all shown strong interest in this feature - can I please ask to take a look at the syntax of expression tags and tell me what you think? Will it meet your use case? Can you see how to write your workflow using the syntax? Can you even demonstrate that?

[alexec]

Here is an example:

https://github.com/argoproj/argo-workflows/blob/363c13aeaaa13891030a21920d20e4cbcfd146ff/examples/expression-template-workflow.yaml

[max-sixty]

Having sprig in there is perfect for the use cases I had. That does everything I need. The argo syntax seems great.

One brief question, what would {{item}} return here, without the equals?

              parameters:
                - name: foo
                  value: "{{=item}}"
            # withParam must be a JSON list encoded as a string, so you use `toJson` to perform the conversion
            withParam: "{{=sprig.toJson(filter([1, 3], {# > 1}))}}"

[alexec]

One brief question, what would {{item}} return here, without the equals?

That would be {{item}} which is the item as usual. In this case withParam: "{{=sprig.toJson(filter([1, 3], {# > 1}))}}" would evaluate to withParam: "[3]" which mean one item that is 3 (int).

[max-sixty]

One brief question, what would {{item}} return here, without the equals?

That would be {{item}} which is the item as usual. In this case withParam: "{{=sprig.toJson(filter([1, 3], {# > 1}))}}" would evaluate to withParam: "[3]" which mean one item that is 3 (int).

Right — if we had withParam: "[3]", then {{item}} would be OK in the value field IIUC?

Is the difference here that if you use an expression in the withParam field, then the value field needs to be {{=item}}?

[alexec]

One brief question, what would {{item}} return here, without the equals?

That would be {{item}} which is the item as usual. In this case withParam: "{{=sprig.toJson(filter([1, 3], {# > 1}))}}" would evaluate to withParam: "[3]" which mean one item that is 3 (int).

Right — if we had withParam: "[3]", then {{item}} would be OK in the value field IIUC?

Is the difference here that if you use an expression in the withParam field, then the value field needs to be {{=item}}?

yes, that works too

[simster7]

Really good use case for this:

    podSpecPatch: '{"containers":[{"name":"main", "resources":{"requests":{"cpu":
      {{=retries * 100}} }}}]}'

[alexec]

100% coverage:

[sarabala1979]

@alexec Thanks. I really like this custom wrapper on the template and expression language. It gives us the control to extend the expression function and includes a new template language in the future.
After reviewing this PR. I kind of thinking to refactor conditional artifact for use this template instead of direct expr. It will be the same template language across spec.

[alexec]

@alexec Thanks. I really like this custom wrapper on the template and expression language. It gives us the control to extend the expression function and includes a new template language in the future.
After reviewing this PR. I kind of thinking to refactor conditional artifact for use this template instead of direct expr. It will be the same template language across spec.

I'm not a fan of {{ and }}, due to the Helm conflict. I still think it would be nice to have fromExpression, I don't think you need to refactor. Unless, could fromExpression be redundant? It's hard for me to make a call on that. We still want number and string env funcs.

[lucastheisen]

@alexec , my primary wish list item was the addition of control structures (or something else capable of dynamic argument list based on workflow parameter). This is not intended to enable that, correct? Its possible i missed some way of leveraging sprig to generate an argument list with one of the items being optional, but i am not seeing it.

Otherwise, the addition of sprig expressions is definitely a benefit (we do date calculations frequently).

[alexec]

@alexec , my primary wish list item was the addition of control structures

No. The example you have is not valid YAML and cannot be implemented except by a pre-processor like Helm.

I think you could achieve the same goal like this:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: argo-test-echo-
spec:
  entrypoint: echo
  arguments:
    parameters:
      - name: dry-run
        value: "false"
  templates:
    - name: echo
      container:
        command:
          - sh
          - -c
          - |
            echo {{= workflow.parameters['dry-run'] ? ' "DRY RUN: "' : ''}} hello world
        image: argoproj/argosay:v1

[lucastheisen]

@alexec , templating templated templates is never easy (we deploy our WorkflowTemplate's using ansible so thats another layer on top of this), but your suggestion:

{{= workflow.parameters['"dry-run"] ? '
      - "DRY RUN: "
'}}

seems to indicate possible. Right now, that's probably enough.

[sarabala1979]

I'm not a fan of {{ and }}, due to the Helm conflict. I still think it would be nice to have fromExpression, I don't think you need to refactor. Unless, could fromExpression be redundant? It's hard for me to make a call on that. We still want number and string env funcs.

I kind of thinking on the user side to handle the two different formats. yes I am going to add number, string and jsonpath env function on env. This template is using expr underneath so, we can include those function into env.

current. expression: number(input.parameters.num) ==1? true: false

if we use template here the format
{{=number(input.parameters.num) ==1? true: false}}

[crenshaw-dev]

@lucastheisen everything inside of source is just a string, as far as Kubernetes' YAML parser is concerned. So the possibilities within that string are endless.

But argument lists aren't strings - they're lists. Lists of Arguments (represented as lists of objects in YAML and then deserialized to Go data structures in Argo code).

Argo can't provide a templating tool to create dynamic argument lists, because the argument lists have to be valid YAML before they get to Argo in the first place.

[sarabala1979]

Initial Review. I am cloned the PR and trying in my local

[sarabala1979]

are you missing any TODO here?

[alexec]

this should use ResolveVar, yes

[alexec]

actually, not really feasible, code here is very bespoke to metrics

[sarabala1979]

I think the key should be `inputs.artifacts '+a.Name

[alexec]

correct. i misunderstnand the old value

[alexec]

you are correct

[sarabala1979]

s.scope is already map[string]interface{}. can we just add new keys to it?

[alexec]

would modify it, want to copy

[sarabala1979]

LGMT

[therc]

Thanks for adding this feature. I assume that it will be in 3.0, but not cherry-picked in 2.12 or any other 2.x releases, correct? I can't wait to try it.