Support utility / string modification template functions

[discordianfish]

FEATURE REQUEST:
Argo should provide a (limited) set of functions available in templates to support basic tasks like string modification.

Rational:
People can use a step/task to parse parameters and return them with modifications.
Yet this seems overkill for simple string manipulation.

Specifically I'm using argo in CI/CD. I have the revision and and the current ref as input parameter but I want to tag my images with a shorted revision and and the branch or tag as parsed from the ref.

Implementation:
Argo currently uses fasttemplate which unfortunately doesn't have good support for function (pipelines). I'd suggest to use the stdlib's text/template, given that I assume that templating performance isn't really relevant.

As a first take, I'd suggest adding the most common functions from the strings package like Join, Split*, Trim* as well as a SubString function.

[Ark-kun]

I think we'll quickly see that the text templating path leads nowhere.
Even with the current very limited placeholder support, it's impossible to call some programs that use go format templates (e.g. docker) since their templates clash with Argo syntax.

I have experience with many custom text-based templating languages and all of them were painful or impossible to use for some cases.

The solution is to use structural templates. There are many advantages of structural templates over text-based templates:

• Templating system is very easy to extend.
• No need to reinvent the wheel and write a complicated fragile parser that's still going to break.
• Templates do not conflict with any verbatim texts.

With structural templates we can even support multiple templating engines.

Structural templates utilize the language that Argo workflows already use - YAML.

Here is a pretty complex example that uses concat and substring to construct the full image name out of the base image name and the shortened revision:

command: [
  docker,
  tag,
  {inputValue: source-image},
  {concat: [{inputValue: image-name}, ':', {substring: {str: {inputValue: revision}, count: 6}]}
]

P.S. {inputValue: source-image} is the structural equivalent of "{{inputs.parameters.source-image}}".

[iven]

@Ark-kun It seems that structural templates can do well with string manipulations, which may be called "filters" in template languages like Jinja.
But Jinja also support control structures like for-loops, conditionals, how can we implement it use structural templates? For example, I want to generate k8s resource manifests based on input parameters, I can write like this if argo supports control structures:

    resource:
      action: apply
      manifest: |
        apiVersion: networking.istio.io/v1alpha3
        kind: VirtualService
        metadata:
          name: {{ inputs.parameters.app_name }}
        spec:
          hosts:
            - {{ inputs.parameters.domain_name }}
          {% if inputs.parameters.protocol == 'http' %}
          gateways:
            - {{ inputs.parameters.app_name }}
            - mesh
          {% endif %}
          http:
            {% for deployment in inputs.parameters.deployments %}
            - match:
                - headers:
                    traffic-source:
                      exact: xxx
              route:
                - destination:
                    host: {{ deployment.name }}
                    subset: {{ deployment.stage }}
            {% endfor %}

[discordianfish]

I wouldn't want control structures, just string manipulation filters. It's easy to generate the templates from e.g jsonnet, but that doesn't help if you need to access the output of a step and want to do simple string manipulation on it.

For example, you have a step that gets a git revision and you want to take the first 8 chars of it. You now either need to use a custom image for that step or run an additional step just to do run a bash script getting you the first 8 chars.

This is no black and white issue, I think some string manipulation functions make sense. Otherwise people need to wrap the argo templates for even simple use cases.

[lucastheisen]

I think control structures have value as well. The use case that comes to mind is supplemental args. For example:

argo submit --from workflowtemplate/foo --parameter "dry-run=true"

with

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: argo-test-echo
spec:
  entrypoint: echo
  arguments:
    parameters:
    - name: dry-run
      value: false
  serviceAccountName: argo
  templates:
  - name: echo
    container:
      command:
      - echo
      {% if workflow.parameters.dry-run %}
      - "DRY RUN: "
      {% endif %}
      - hello
      - world
      image: centos:7

The only way i can think of to do this now is to have different templates for each combination of supported arguments and use when clauses.

[discordianfish]

@lucastheisen You can do that in bash and set a parameter via the templating. Control structures in the templating layer makes the template unparsable without having to evaluate the template first. That's make everything less predictable and less readable.
IMO templating should only exist within a field of the workflow manifest

[lucastheisen]

@discordianfish , fair enough. I mostly didn't want to have to drop into a shell. The containers I am running have an ENTRYPOINT (that is effectively unknown at call time). So for me to do it in bash means I have to have knowledge of that ENTRYPOINT so I can call it after doing the dynamic processing of the parameters, something like:

    container:
      command:
      - bash
      - -c
      args:
      - |
        # assumes image ENTRYPOINT is /docker-entrypoint.sh
        command=(
          /docker-entrypoint.sh
          "{{workflow.parameters.start}}"
          "{{workflow.parameters.end}}"
        )
        if [[ -n "{{workflow.parameters.dry-run}}" ]]; then
          command+=(--dry-run)
        fi
        exec "${command[@]}"

There are lots of ways around this, and I just don't really like any of them.

[max-sixty]

Here's a question I asked on SO whose answer linked here, and might be a useful case — simple date strings. These currently require an additional step, or fairly verbose use of variables.

https://stackoverflow.com/questions/64268801/formatting-dates-in-argo/64319266#64319266

And thanks to @mac9416 for the thoughtful answer!

[alexec]

@max-sixty would you like to investigate how this might be done using fasttemplate? That would be the first step to a solution.

[max-sixty]

Yes I will take a look at that, I know my contributions has been overdue!

[max-sixty]

@max-sixty would you like to investigate how this might be done using fasttemplate? That would be the first step to a solution.

Fasttemplate

IIUC, fasttemplate isn't useful at all for these use cases, because it only does string substitution, no processing — and we already have that in argo. To quote from the readme:

Fasttemplate performs only a single task - it substitutes template placeholders with user-defined values. At high speed :)

In order to give anything like conditionals, or datetime processing, etc, we'd need a more fully featured library. I don't know the go space that well, but have used gomplate successfully, and it has a responsive maintainer.

Any other suggestions on these?

Valid YAML?

One dichotomy that's worth considering is "are these still valid YAML?"

• The datetime string examples I gave would still be valid YAML, since the template is a string in an existing field (e.g. value: {{workflow.creationTimestamp | time.Unix}}). Values could be passed into a templating tool as part of running the workflow.
• Some of the conditionals above would not be valid YAML — the workflow would be passed into a templating tool when parsing the workflow e.g.:

      command:
      - echo
      {% if workflow.parameters.dry-run %}
      - "DRY RUN: "
      {% endif %}
  

[alexec]

Example of the proposed solution - PLEASE COMMENT!

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: expression-
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: task-0
            template: pod-0
            arguments:
              parameters:
                - name: foo
                  value: "{{=int(item) + 1}}"
            # stringify: withParam must be a JSON list encoded as a string
            withParam: "{{=toJson(filter([1, 3, 7], {# > 1}))}}"
    - name: pod-0
      inputs:
        parameters:
          - name: foo
      container:
        image: argoproj/argosay:v2
        args: [ echo, "hello {{=int(inputs.parameters.foo) * 10}}" ]

[discordianfish]

@alexec Nice! The only thing is that I'd consider other libs as well, given how expensive it will be to change this later. That's why my suggestion was to use the string package as oppose to a more complex lib. But happy to have this functionality one way or the other!

[yoshiya8]

I totally agree with @lucastheisen regarding control structures. I cannot tell you how many of my workflow templates have the same thing over and over with various combinations of container parameters. Control structures would be a great support to the DRY principle.