feat(controller): Expression template tags. Resolves #4548 & #1293 (#…

…5115) Signed-off-by: Alex Collins <alex_collins@intuit.com>
argoproj · Feb 27, 2021 · e6fa41a · e6fa41a
1 parent 54b04c6
commit e6fa41a
Show file tree

Hide file tree

Showing 30 changed files with 737 additions and 265 deletions.
diff --git a/docs/environment-variables.md b/docs/environment-variables.md
@@ -13,6 +13,7 @@ Note that these environment variables may be removed at any time.
 | `ARCHIVED_WORKFLOW_GC_PERIOD` | `time.Duration` | The periodicity for GC of archived workflows. |
 | `ARGO_TRACE` | `bool` | Whether to enable tracing statements in Argo components. |
 | `DEFAULT_REQUEUE_TIME` | `time.Duration` | The requeue time for the rate limiter of the workflow queue. |
+| `EXPRESSION_TEMPLATES` | `bool` | Escape hatch to disable expression templates. Default `true`. |
 | `GZIP_IMPLEMENTATION` | `string` | The implementation of compression/decompression. Currently only "PGZip" and "GZip" are supported. Defaults to "PGZip". |
 | `LEADER_ELECTION_IDENTITY` | `string` | The ID used for workflow controllers to elect a leader. |
 | `LEADER_ELECTION_LEASE_DURATION` | `time.Duration` | The duration that non-leader candidates will wait to force acquire leadership. |

diff --git a/docs/fields.md b/docs/fields.md
@@ -94,6 +94,8 @@ Workflow is the definition of a workflow resource
 
 - [`exit-handlers.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/exit-handlers.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`forever.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/forever.yaml)
 
 - [`fun-with-gifs.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/fun-with-gifs.yaml)
@@ -445,6 +447,8 @@ WorkflowSpec is the specification of a Workflow.
 
 - [`exit-handlers.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/exit-handlers.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`forever.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/forever.yaml)
 
 - [`fun-with-gifs.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/fun-with-gifs.yaml)
@@ -814,6 +818,8 @@ CronWorkflowSpec is the specification of a CronWorkflow
 
 - [`exit-handlers.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/exit-handlers.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`forever.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/forever.yaml)
 
 - [`fun-with-gifs.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/fun-with-gifs.yaml)
@@ -1141,6 +1147,8 @@ WorkflowTemplateSpec is a spec of WorkflowTemplate.
 
 - [`exit-handlers.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/exit-handlers.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`forever.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/forever.yaml)
 
 - [`fun-with-gifs.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/fun-with-gifs.yaml)
@@ -1452,6 +1460,8 @@ Arguments to a template
 
 - [`exit-handler-step-level.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/exit-handler-step-level.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`global-outputs.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/global-outputs.yaml)
 
 - [`global-parameters.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/global-parameters.yaml)
@@ -1798,6 +1808,8 @@ Template is a reusable and composable unit of execution in a workflow
 
 - [`exit-handlers.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/exit-handlers.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`forever.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/forever.yaml)
 
 - [`fun-with-gifs.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/fun-with-gifs.yaml)
@@ -2176,6 +2188,8 @@ Outputs hold parameters, artifacts, and results from a step
 
 - [`data-transformations.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/data-transformations.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`fun-with-gifs.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/fun-with-gifs.yaml)
 
 - [`global-outputs.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/global-outputs.yaml)
@@ -2393,6 +2407,8 @@ Parameter indicate a passed string parameter to a service template with an optio
 
 - [`exit-handler-step-level.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/exit-handler-step-level.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`global-outputs.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/global-outputs.yaml)
 
 - [`global-parameters.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/global-parameters.yaml)
@@ -2653,6 +2669,8 @@ DAGTemplate is a template subtype for directed acyclic graph templates
 
 - [`exit-handler-dag-level.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/exit-handler-dag-level.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`key-only-artifact.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/key-only-artifact.yaml)
 
 - [`loops-dag.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/loops-dag.yaml)
@@ -2810,6 +2828,8 @@ Inputs are the mechanism for passing parameters, artifacts, volumes from one tem
 
 - [`exit-handler-step-level.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/exit-handler-step-level.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`global-outputs.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/global-outputs.yaml)
 
 - [`handle-large-output-results.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/handle-large-output-results.yaml)
@@ -3544,6 +3564,8 @@ ValueFrom describes a location in which to obtain the value to a parameter
 
 - [`data-transformations.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/data-transformations.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`global-outputs.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/global-outputs.yaml)
 
 - [`handle-large-output-results.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/handle-large-output-results.yaml)
@@ -3648,6 +3670,8 @@ MetricLabel is a single label for a prometheus metric
 
 - [`daemoned-stateful-set-with-service.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/daemoned-stateful-set-with-service.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`hello-world.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/hello-world.yaml)
 
 - [`k8s-owner-reference.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/k8s-owner-reference.yaml)
@@ -3715,6 +3739,8 @@ DAGTask represents a node in the graph during DAG execution
 
 - [`exit-handler-dag-level.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/exit-handler-dag-level.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`key-only-artifact.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/key-only-artifact.yaml)
 
 - [`loops-dag.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/loops-dag.yaml)
@@ -4151,6 +4177,8 @@ ObjectMeta is metadata that all persisted resources must have, which includes al
 
 - [`exit-handlers.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/exit-handlers.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`forever.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/forever.yaml)
 
 - [`fun-with-gifs.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/fun-with-gifs.yaml)
@@ -4721,6 +4749,8 @@ A single application container that you want to run within a pod.
 
 - [`exit-handlers.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/exit-handlers.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`forever.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/forever.yaml)
 
 - [`fun-with-gifs.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/fun-with-gifs.yaml)
@@ -5352,6 +5382,8 @@ PersistentVolumeClaimSpec describes the common attributes of storage devices and
 
 - [`exit-handlers.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/exit-handlers.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`forever.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/forever.yaml)
 
 - [`fun-with-gifs.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/fun-with-gifs.yaml)
@@ -5979,6 +6011,8 @@ EnvVarSource represents a source for the value of an EnvVar.
 
 - [`data-transformations.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/data-transformations.yaml)
 
+- [`expression-tag-template-workflow.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/expression-tag-template-workflow.yaml)
+
 - [`global-outputs.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/global-outputs.yaml)
 
 - [`handle-large-output-results.yaml`](https://github.com/argoproj/argo-workflows/blob/master/examples/handle-large-output-results.yaml)

diff --git a/docs/variables.md b/docs/variables.md
@@ -2,42 +2,128 @@
 
 Some fields in a workflow specification allow for variable references which are automatically substituted by Argo.
 
-??? note "How to use variables"
-    Variables are enclosed in curly braces and **may** include whitespace between the brackets and variable.
-
-    ``` yaml
-    apiVersion: argoproj.io/v1alpha1
-    kind: Workflow
-    metadata:
-      generateName: hello-world-parameters-
-    spec:
-      entrypoint: whalesay
-      arguments:
+## How to use variables
+
+Variables are enclosed in curly braces:
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Workflow
+metadata:
+  generateName: hello-world-parameters-
+spec:
+  entrypoint: whalesay
+  arguments:
+    parameters:
+      - name: message
+        value: hello world
+  templates:
+    - name: whalesay
+      inputs:
         parameters:
-        - name: message
-          value: hello world
-      templates:
-      - name: whalesay
-        inputs:
-          parameters:
           - name: message
-        container:
-          image: docker/whalesay
-          command: [cowsay]
-        # args: ["{{ inputs.parameters.message }}"]       <- good
-          args: ["{{inputs.parameters.message}}"]         #  good
-    ```
+      container:
+        image: docker/whalesay
+        command: [ cowsay ]
+        args: [ "{{inputs.parameters.message}}" ] 
+```
 
 The following variables are made available to reference various metadata of a workflow:
 
-## All Templates
+## Template Tag Kinds
+
+There are two kinds of template tag:
+
+* **simple** The default, e.g. `{{workflow.name}}`
+* **expression** Where`{{` is immediately followed by `=`, e.g. `{{=workflow.name}}`.
+
+### Simple
+
+The tag is substituted with the variable that has a name the same as the tag.
+
+Simple tags **may** have whitespace between the brackets and variable.
+
+```yaml
+args: [ "{{ inputs.parameters.message }}" ]  
+```
+
+### Expression
+
+> Since v3.1
+
+The tag is substituted with the result of evaluating the tag as an expression.
+
+[Learn about the expression syntax](https://github.com/antonmedv/expr/blob/master/docs/Language-Definition.md).
+
+#### Examples
+
+Plain list:
+
+```
+[1, 2]
+```
+
+Filter a list:
+
+```
+filter([1, 2], { # > 1})
+```
+
+Map a list:
+
+```
+map([1, 2], { # * 2 })
+```
+
+We provide some core functions:
+
+Cast to int:
+
+```
+asInt(inputs.parameters["my-int-param"])
+```
+
+Cast to flloat:
+
+```
+asFloat(inputs.parameters["my-float-param"])
+```
+
+Cast to string:
+
+```
+string(1)
+```
+
+Convert to a JSON string (needed for `withParam`):
+
+```
+toJson([1, 2])
+```
+
+You can also use [Sprig functions](http://masterminds.github.io/sprig/):
+
+Trim a string:
+
+```
+sprig.trim(inputs.parameters["my-string-param"])
+```
+
+!!! Warning In Sprig functions, errors are not often not raised. E.g. if `int` is used on an invalid value, it
+returns `0`. Please review the Sprig documentation to understand which functions do and which do not.
+
+## Reference
+
+### All Templates
+
 | Variable | Description|
 |----------|------------|
 | `inputs.parameters.<NAME>`| Input parameter to a template |
 | `inputs.parameters`| All input parameters to a template as a JSON string |
 | `inputs.artifacts.<NAME>` | Input artifact to a template |
 
-## Steps Templates
+### Steps Templates
+
 | Variable | Description|
 |----------|------------|
 | `steps.<STEPNAME>.id` | unique id of container step |
@@ -51,7 +137,8 @@ The following variables are made available to reference various metadata of a wo
 | `steps.<STEPNAME>.outputs.parameters.<NAME>` | Output parameter of any previous step. When the previous step uses 'withItems' or 'withParams', this contains a JSON array of the output parameter values of each invocation |
 | `steps.<STEPNAME>.outputs.artifacts.<NAME>` | Output artifact of any previous step |
 
-## DAG Templates
+### DAG Templates
+
 | Variable | Description|
 |----------|------------|
 | `tasks.<TASKNAME>.id` | unique id of container task |
@@ -65,7 +152,8 @@ The following variables are made available to reference various metadata of a wo
 | `tasks.<TASKNAME>.outputs.parameters.<NAME>` | Output parameter of any previous task. When the previous task uses 'withItems' or 'withParams', this contains a JSON array of the output parameter values of each invocation |
 | `tasks.<TASKNAME>.outputs.artifacts.<NAME>` | Output artifact of any previous task |
 
-## Container/Script Templates
+### Container/Script Templates
+
 | Variable | Description|
 |----------|------------|
 | `pod.name` | Pod name of the container/script |
@@ -74,13 +162,15 @@ The following variables are made available to reference various metadata of a wo
 | `outputs.artifacts.<NAME>.path` | Local path of the output artifact |
 | `outputs.parameters.<NAME>.path` | Local path of the output parameter |
 
-## Loops (withItems / withParam)
+### Loops (withItems / withParam)
+
 | Variable | Description|
 |----------|------------|
 | `item` | Value of the item in a list |
 | `item.<FIELDNAME>` | Field value of the item in a list of maps |
 
-## Metrics
+### Metrics
+
 When emitting custom metrics in a `template`, special variables are available that allow self-reference to the current
 step.
 
@@ -92,7 +182,8 @@ step.
 | `inputs.parameters.<NAME>` | Input parameter of the metric-emitting template |
 | `outputs.parameters.<NAME>` | Output parameter of the metric-emitting template |
 | `outputs.result` | Output result of the metric-emitting template |
-| `resourcesDuration.{cpu,memory}` | Resources duration **in seconds**. Must be one of `resourcesDuration.cpu` or `resourcesDuration.memory`, if available. For more info, see the [Resource Duration](resource-duration.md) doc.|
+| `resourcesDuration.{cpu,memory}` | Resources duration **in
+seconds**. Must be one of `resourcesDuration.cpu` or `resourcesDuration.memory`, if available. For more info, see the [Resource Duration](resource-duration.md) doc.|
 
 ### Realtime Metrics
 
@@ -101,12 +192,15 @@ real time, set `realtime: true` under `gauge` (note: only Gauge metrics allow fo
 currently available for real time emission:
 
 For `Workflow`-level metrics:
+
 * `workflow.duration`
 
 For `Template`-level metrics:
+
 * `duration`
 
-## Global
+### Global
+
 | Variable | Description|
 |----------|------------|
 | `workflow.name` | Workflow name |
@@ -124,7 +218,8 @@ For `Template`-level metrics:
 | `workflow.priority` | Workflow priority |
 | `workflow.duration` | Workflow duration estimate, may differ from actual duration by a couple of seconds |
 
-## Exit Handler
+### Exit Handler
+
 | Variable | Description|
 |----------|------------|
 | `workflow.status` | Workflow status. One of: `Succeeded`, `Failed`, `Error` |