Skip to content

Commit

Permalink
Add decorator syntax guide
Browse files Browse the repository at this point in the history
Signed-off-by: Elliot Gunton <elliotgunton@gmail.com>
  • Loading branch information
elliotgunton committed Jun 13, 2024
1 parent 73861b6 commit 267eaf9
Show file tree
Hide file tree
Showing 3 changed files with 178 additions and 0 deletions.
166 changes: 166 additions & 0 deletions docs/user-guides/decorators.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
# Decorators

Proposed in [HEP0001](https://github.com/argoproj-labs/hera/blob/73861b/proposals/heps/0001-decorators.md), decorators
for the other main template types were added in Hera v5.16. They can be used to write Workflows in a more familiar
Pythonic fashion, taking inspiration from projects such as FastAPI. This feature is intended to replace the context
manager syntax (but the context manager syntax will continue to be supported).

The decorators introduced in v5.16 are members of the `Workflow` class, meaning they can manipulate values within that
`Workflow`. The new decorators introduced are:

* `dag`
* `steps`
* `container`
* `script`
* `set_entrypoint`

If you want to declare inputs and outputs in your templates, you must use the special `Input` and `Output` Pydantic
classes from `hera.workflows` so that Hera can deduce the inputs and outputs of your templates from the fields declared
in the classes.

## `dag` and `steps`

The `dag` and `steps` decorator bring brand-new functionality to Hera, as they allow you to run and test your dag and
steps functions entirely locally. Note that as you are running pure Python code, features of Argo Workflows are not
supported when running locally, such as expressions. We plan on adding support for expressions so that conditional tasks
can be correctly skipped.

The `dag` decorator is also special in that it is able to automatically deduce the dependency relationships between
tasks. It does this by tracking the parameters and artifacts passed between tasks because the code within the function
must be written in a natural running order (which is also why it can be run locally). Note that only a basic "depends"
relationship is deduced, more complex dependencies are not yet supported.

The `steps` decorator allows you to use the `parallel` function from `hera.workflows`, which is used to open a context under which all the steps will run in parallel.

The choice between `dag` and `steps` to arrange your templates mostly comes down to personal preference. If you want to
run as many _dependent_ templates in parallel as possible then use a DAG, which will figure out which tasks can run
first. If you want to run templates sequentially, and have more control over the running order and when to parallelise,
use `steps`.

## `container`

The `container` decorator is a convenient way to declare your container templates in Python, though the feature set is
limited. You can specify the `command` and `args` in the decorator arguments, and the inputs and outputs in the function
signature. You can leave the function as a stub or add some code to be able to run the function locally - consider it as
a "mockable" function.

## `script`

The new `script` decorator works the same as the old decorator in terms of declaring `script` attributes in the
decorator function. For the inputs and outputs of the function however, you now must use a subclass of the `Input` and
`Output`. This is the same behaviour as the experimental [Script Runner IO](./script-runner-io.md) for the old script
decorator.

Using this decorator will enforce usage of the `RunnerScriptConstructor`, so you must ensure you use an image built from
your code.

## `set_entrypoint`

The `set_entrypoint` decorator is a simple decorator with no arguments, used to set the `entrypoint` of the `Workflow` that you're declaring.

## Using decorators

Let's take a look at an example using scripts and steps.

```py
from hera.shared import global_config
from hera.workflows import Input, Output, WorkflowTemplate

global_config.experimental_features["decorator_syntax"] = True

w = WorkflowTemplate(name="my-template")

class MyInput(Input):
user: str

class MyOutput(Output):
my_str: str

@w.script()
def hello_world(my_input: MyInput) -> MyOutput:
output = MyOutput()
output.my_str = f"Hello Hera User: {my_input.user}!"
return output

# You can pass script kwargs (including an alternative public template name) in the decorator
@w.script(name="goodbye-world", labels={"my-label": "my-value"})
def goodbye(my_input: MyInput) -> Output:
output = Output()
output.result = f"Goodbye Hera User: {my_input.user}!"
return output


@w.set_entrypoint
@w.steps()
def my_steps() -> None:
hello_world(my_input=MyInput(user="elliot"))
goodbye(my_input=MyInput(user="elliot"))
```

For the line-by-line explanation, let's start with

```py
global_config.experimental_features["decorator_syntax"] = True
```

The new decorators are an experimental feature, so must be turned on via the `global_config`.

Then, in the world of decorators in Hera, we declare a `WorkflowTemplate` upfront, instead of using the `with WorkflowTemplate` syntax:

```py
w = WorkflowTemplate(name="my-template")
```

We need to use the decorators which are members of `Workflow`, as they link the given function as a template to the `Workflow`.

Then, we declare some Input/Output classes, inheriting from the Pydantic classes:

```py
class MyInput(Input):
user: str

class MyOutput(Output):
my_str: str
```

Here, we're setting up the `MyInput` class which will be converted to a set of template inputs; here, that only consists
of the `user` parameter, which is of type `str` - Hera will do the type checking for you when running with the
[Hera Runner](./script-basics.md#runnerscriptconstructor)!. We also have the template outputs contained in `MyOutput`. The `Output` class also contains the
`exit_code` and `result` fields, which are used by the Hera Runner to exit a script with the given exit code, and to
print a value to stdout to act as a step or task's
[special "result" output parameter](https://argo-workflows.readthedocs.io/en/stable/walk-through/output-parameters/#result-output-parameter).

Next, we declare two `script` templates:

```py
@w.script()
def hello_world(my_input: MyInput) -> MyOutput:
output = MyOutput()
output.my_str = f"Hello Hera User: {my_input.user}!"
return output

# You can pass script kwargs (including an alternative public template name) in the decorator
@w.script(name="goodbye-world", labels={"my-label": "my-value"})
def goodbye(my_input: MyInput) -> Output:
output = Output()
output.result = f"Goodbye Hera User: {my_input.user}!"
return output
```

Note that in `hello_world` we are using a `MyOutput` class, which means we can set the `my_str` output parameter, while
in `goodbye`, we are using the basic `Output` class, so we set the special `result` output parameter.

Then, we set up a `steps` template, setting it as the entrypoint, as follows:

```py
@w.set_entrypoint
@w.steps()
def my_steps() -> None:
hello_world(my_input=MyInput(user="elliot"))
goodbye(my_input=MyInput(user="elliot"))
```

We can simply call the script templates, passing the input objects in.

For more complex examples, including use of a dag, see
[the "experimental" examples](../examples/workflows/experimental/new_dag_decorator_params.md).
11 changes: 11 additions & 0 deletions docs/walk-through/advanced-hera-features.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,17 @@ global_config.experimental_features["script_pydantic_io"] = True

Read the full guide on script pydantic IO in [the script user guide](../user-guides/script-runner-io.md).

### Decorators for main template types

Decorators for dags, steps and containers are provided alongside a new script decorator, letting your declare Workflows via Python functions alone.

To enable the decorators, you must set the `experimental_feature` flag `decorator_syntax`

```py
global_config.experimental_features["decorator_syntax"] = True
```

Read the full guide on decorators in [the decorator user guide](../user-guides/decorators.md).

## Graduated features

Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ nav:
- Script Basics: user-guides/script-basics.md
- Script Annotations: user-guides/script-annotations.md
- Script Runner IO: user-guides/script-runner-io.md
- Decorators: user-guides/decorators.md
- Expr Transpiler: user-guides/expr.md
- Examples:
- About: examples/workflows-examples.md
Expand Down

0 comments on commit 267eaf9

Please sign in to comment.