docs: Adding documentation for tags & resources + updating JSON-schem…

…a for templates Signed-off-by: Romain Beuque <romain.beuque@corp.ovh.com>
ovh · Apr 10, 2020 · be39395 · be39395
1 parent 117f981
commit be39395
Show file tree

Hide file tree

Showing 4 changed files with 184 additions and 28 deletions.
diff --git a/README.md b/README.md
@@ -201,6 +201,7 @@ The following templating functions are available:
 - `blocked`: boolean (default: false): no tasks can be created from this template
 - `hidden`: boolean (default: false): the template is not listed on the API, it is concealed to regular users
 - `retry_max`: int (default: 100): maximum amount of consecutive executions of a task based on this template, before being blocked for manual review
+- `tags`: templateable map, used to filter tasks (see [tags](#tags))
 
 ### Inputs
 
@@ -229,6 +230,25 @@ See the example template above to see variables in action. The expression in a v
 
 The JavaScript evaluation is done using [otto](https://github.com/robertkrimen/otto).
 
+### Tags <a name="tags"></a>
+
+Tags are a map of strings property of a task. They will be used in the task listing to search for some tasks using filters. With tags, uTask can be used as a task backend by others APIs.
+
+Tags values are expected to be a `string`: it support all uTask templating on values. To remove a tag from a task, use the empty value `""`.
+
+```yaml
+  tags:
+      customer: "{{.input.customer_id}}"
+      type: "billing"
+```
+
+In this example, tag `customer` will be templated from the task inputs, and allow others APIs to search all the tasks for a given customer.
+
+Tags can be added to a task:
+- from the template definition of the task
+- while creating a task, requester can input custom tags
+- during the execution, using the [`tag` builtin plugin](./pkg/plugins/builtin/tag/README.md)
+
 ### Steps
 
 A step is the smallest unit of work that can be performed within a task. At is's heart, a step defines an **action**: several types of actions are available, and each type requires a different configuration, provided as part of the step definition. The state of a step will change during a task's resolution process, and determine which steps become elligible for execution. Custom states can be defined for a step, to fine-tune execution flow (see below).
@@ -297,14 +317,20 @@ steps:
 
 - `name`: a unique identifier
 - `description`: a human readable sentence to convey the step's intent
+- `action`: see [Action](#step-action)
+- `foreach`: see [Loops](#step-foreach)
 - `dependencies`: a list of step names on which this step waits before running
+- `idempotent`: a boolean indicating if this step is safe to be replayed in case of uTask instance crash
+- `json_schema`: a JSON-Schema object to validate the step output
+- `resources`: a list of resources that will be used by this step to apply some rate-limiting (see [resources](#resources))
+- `custom_states`: a list of user-defined states for this step, to be used inside conditions
 - `retry_pattern`: (`seconds`, `minutes`, `hours`) define on what temporal order of magnitude the re-runs of this step should be spread (default = `seconds`)
 
 <p align="center">
 <img src="./assets/img/utask_backoff.png" width="70%">
 </p>
 
-#### Action
+#### Action <a name="step-action"></a>
 
 The `action` field of a step defines the actual workload to be performed. It consists of at least a `type` chosen among the registered action plugins, and a `configuration` fitting that plugin. See below for a detailed description of builtin plugins. For information on how to develop your own action plugins, refer to [this section](#plugins).
 
@@ -408,8 +434,9 @@ Browse [builtin actions](./pkg/plugins/builtin)
 |**`email`**   | Send an email | [Access plugin doc](./pkg/plugins/builtin/email/README.md)
 |**`ping`**    | Send a ping to an hostname *Warn: This plugin will keep running until the count is done* | [Access plugin doc](./pkg/plugins/builtin/ping/README.md)
 |**`script`**    | Execute a script under `scripts` folder | [Access plugin doc](./pkg/plugins/builtin/script/README.md)
+|**`tag`**    | Add tags to the current running task | [Access plugin doc](./pkg/plugins/builtin/tag/README.md)
 
-#### Loops
+#### Loops  <a name="step-foreach"></a>
 
 A step can be configured to take a json-formatted collection as input, in its `foreach` property. It will be executed once for each element in the collection, and its result will be a collection of each iteration. This scheme makes it possible to chain several steps with the `foreach` property.
 
@@ -442,6 +469,43 @@ This output can be then passed to another step in json format:
 foreach: '{{.step.prefixStrings.children | toJson}}'
 ```
 
+### Resources <a name="resources"></a>
+
+Resources can be declared to throttle the number of parallel access to a 'resource'. Each resources are labels that can correspond to a physical or logical device/object that will be used inside an action, on which you want to limit the number of parallel accesses. Those labels can be very specific (example: an IP/port combination `172.17.0.1:5432`) or not (example: `all-databases`).
+
+Resources are configured in the `utask-cfg` configuration, indicating a name and a number of maximum parallel accesses.
+
+```json
+{
+    "resource_limits": {
+        "redis-foobar": 2,
+        "internet-gateway": 1000,
+        "database": 14
+    }
+}
+```
+
+Note: resources maximum parallel accesses are defined **per instances**. In this example, if you have 3 instances, then, `redis-foobar` can have up to 6 parallel accesses.
+
+Resources are available to be used inside steps, ensuring that the declared resources won't be accessed more than expected.
+
+```yaml
+steps:
+  getUser:
+    description: Get user
+    resources: ["redis-foobar", "internal-gateway"]
+    action:
+      type: http
+      configuration:
+        url: http://example.org/addToCache
+        method: POST
+        body: '{"cache_method":"redis", "data":"hello"}'
+```
+
+If a resource is already accessed at maximum capacity in others tasks/steps, the step execution will wait until a slot is available.
+
+If a resource declared in a step doesn't exist in the configuration of current uTask instance, then no restriction is applied and the resource can be accessed freely, without limitation.
+
 ### Task templates validation
 
 A JSON-schema file is available to validate the syntax of task templates, it's available in `hack/template-schema.json`.

diff --git a/examples/templates/hello-world-now.yaml b/examples/templates/hello-world-now.yaml
@@ -1,44 +1,59 @@
-name:             hello-world-now
-description:      Say hello to the world, now!
+name: hello-world-now
+description: Say hello to the world, now!
 long_description: This task prints out a greeting to the entire world, after retrieving the current UTC time from an external API
-doc_link:         https://en.wikipedia.org/wiki/%22Hello,_World!%22_program
+doc_link: https://en.wikipedia.org/wiki/%22Hello,_World!%22_program
 
-title_format:     Say hello in {{.input.language}}
+title_format: Say hello in {{.input.language}}
 result_format:
-  echo_message: '{{.step.sayHello.output.message}}'
-  echo_when:    '{{.step.sayHello.output.when}}'
+  echo_message: "{{.step.sayHello.output.message}}"
+  echo_when: "{{.step.sayHello.output.when}}"
 
-allowed_resolver_usernames:   []
+allowed_resolver_usernames: []
 allow_all_resolver_usernames: true
 auto_runnable: true
-blocked:       false
-hidden:        false
+blocked: false
+hidden: false
 
 variables:
-- name: english-message
-  value: Hello World!
-- name: spanish-message
-  expression: |-
-    // a short javascript snippet
-    var h = 'Hola';
-    var m = 'mundo';
-    h + ' ' + m + '!';
+  - name: english-message
+    value: Hello World!
+  - name: spanish-message
+    expression: |-
+      // a short javascript snippet
+      var h = 'Hola';
+      var m = 'mundo';
+      h + ' ' + m + '!';
 
 inputs:
-- name: language
-  description: The language in which you wish to greet the world
-  legal_values: [english, spanish]
-  optional: true
-  default: english
+  - name: language
+    description: The language in which you wish to greet the world
+    legal_values: [english, spanish]
+    optional: true
+    default: english
 
 steps:
   getTime:
     description: Get UTC time
+    idempotent: true
+    resources: ["worldclockapi"]
+    retry_pattern: minutes
     action:
       type: http
       configuration:
         url: http://worldclockapi.com/api/json/utc/now
         method: GET
+    json_schema:
+      "$schema": "http://json-schema.org/draft-07/schema#"
+      type: object
+      required: [currentDateTime, isDayLightSavingsTime, dayOfTheWeek]
+      properties:
+        currentDateTime:
+          type: string
+          pattern: '^\d+-\d+-\d+T\d+:\d+Z$'
+        isDayLightSavingsTime:
+          type: boolean
+        dayOfTheWeek:
+          type: string
   sayHello:
     description: Echo a greeting in your language of choice
     dependencies: [getTime]
@@ -49,5 +64,4 @@ steps:
           message: >-
             {{if (eq .input.language "english")}}{{eval "english-message"}}
             {{else if (eq .input.language "spanish")}}{{eval "spanish-message"}}{{end}}
-          when: '{{.step.getTime.output.currentDateTime}}'
-
+          when: "{{.step.getTime.output.currentDateTime}}"
diff --git a/hack/template-schema.json b/hack/template-schema.json
@@ -27,6 +27,15 @@
                 "action"
             ],
             "properties": {
+                "name": {
+                    "type": "string",
+                    "title": "Step unique name",
+                    "description": "Will identify the step uniquely",
+                    "default": "",
+                    "examples": [
+                        "get_UTC_time"
+                    ]
+                },
                 "description": {
                     "type": "string",
                     "title": "Step description",
@@ -70,13 +79,24 @@
                         "type": "string"
                     }
                 },
+                "resources": {
+                    "type": "array",
+                    "description": "Declares resources that will be used during this step",
+                    "items": {
+                        "type": "string"
+                    }
+                },
                 "foreach": {
                     "type": "string",
                     "description": "Elements on which the step will loop"
                 },
                 "json_schema": {
                     "type": "object",
                     "description": "Elements on which the step will loop"
+                },
+                "idempotent": {
+                    "type": "boolean",
+                    "description": "Indicates if this step is safe to be retried when uTask crashes"
                 }
             }
         },
@@ -109,6 +129,9 @@
                 {
                     "$ref": "#/definitions/ActionAPIOVH"
                 },
+                {
+                    "$ref": "#/definitions/ActionTag"
+                },
                 {
                     "type": "object",
                     "title": "Generic action",
@@ -433,6 +456,20 @@
                         },
                         "resolver_usernames": {
                             "type": "string"
+                        },
+                        "watcher_usernames": {
+                            "type": "string"
+                        },
+                        "delay": {
+                            "type": "string"
+                        },
+                        "tags": {
+                            "type": "object",
+                            "patternProperties": {
+                                ".*": {
+                                    "type": "string"
+                                }
+                            }
                         }
                     }
                 },
@@ -447,6 +484,40 @@
             "additionalProperties": false,
             "description": "Subtask action will spawn a subtask of the specified template, and wait for resolution"
         },
+        "ActionTag": {
+            "type": "object",
+            "properties": {
+                "type": {
+                    "const": "tag"
+                },
+                "configuration": {
+                    "type": "object",
+                    "additionalProperties": false,
+                    "required": [
+                        "tags"
+                    ],
+                    "properties": {
+                        "tags": {
+                            "type": "object",
+                            "patternProperties": {
+                                ".*": {
+                                    "type": "string"
+                                }
+                            }
+                        }
+                    }
+                },
+                "base_configuration": {
+                    "type": "string"
+                },
+                "base_output": {
+                    "type": "object"
+                }
+            },
+            "title": "Tag Action",
+            "additionalProperties": false,
+            "description": "Tag action will add some tags to the task"
+        },
         "ActionPing": {
             "type": "object",
             "properties": {
@@ -947,6 +1018,15 @@
                 }
             }
         },
+        "tags": {
+            "type": "object",
+            "description": "Defines this template tags",
+            "patternProperties": {
+                ".*": {
+                    "type": "string"
+                }
+            }
+        },
         "base_configurations": {
             "type": "object",
             "description": "Global configurations that will be used as base for steps inside this template"

diff --git a/pkg/plugins/builtin/tag/README.md b/pkg/plugins/builtin/tag/README.md
@@ -5,7 +5,6 @@ This plugin updates the tags of the current task. Existing tags are overwritten
 ## Configuration
 
 |Fields|Description
-| ---    | ---             |
 | ------ | --------------- |
 | `tags` | key/values tags |
 
@@ -20,5 +19,4 @@ action:
     tags:
       foo: bar
       bar: # deleted
-
 ```