Skip to content

Commit

Permalink
Add Experimental Features as a user guide
Browse files Browse the repository at this point in the history
* Also some small formatting fixes

Signed-off-by: Elliot Gunton <elliotgunton@gmail.com>
  • Loading branch information
elliotgunton committed Dec 10, 2024
1 parent aaabe25 commit 655911f
Show file tree
Hide file tree
Showing 6 changed files with 80 additions and 64 deletions.
76 changes: 76 additions & 0 deletions docs/user-guides/experimental-features.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Experimental Features

From time to time, Hera will release a new feature under the "experimental feature" flag while we develop the feature
and ensure stability. Once the feature is stable and we have decided to support it long-term, it will "graduate" into
a fully-supported feature.

To enable experimental features you must set the feature by name to `True` in the `global_config.experimental_features`
dictionary before using the feature:

```py
global_config.experimental_features["NAME_OF_FEATURE"] = True
```

Note that experimental features are subject to breaking changes in future releases of the same major version. We will
usually announce changes in [the Hera slack channel](https://cloud-native.slack.com/archives/C03NRMD9KPY).

## Currently supported experimental features:

### Script IO Models

Hera provides Pydantic models for you to create subclasses from, which allow you to more easily declare script template
inputs. Any fields that you declare in your subclass of `Input` will become input parameters or artifacts, while
`Output` fields will become output parameters artifacts. The fields that you declare can be `Annotated` as a `Parameter`
or `Artifact`, as any fields with a basic type will become `Parameters`. Turning on the `script_pydantic_io` flag will
automatically enable the `script_annotations` experimental feature.

To enable Hera input/output models, you must set the `experimental_feature` flag `script_pydantic_io`

```py
global_config.experimental_features["script_pydantic_io"] = True
```

Read the full guide on script pydantic IO in [the script user guide](../user-guides/script-runner-io.md).

### Decorators for main template types

Decorators for dags, steps and containers are provided alongside a new script decorator, letting your declare Workflows via Python functions alone.

To enable the decorators, you must install the optional extras under `experimental`

```bash
pip install hera[experimental]
```

You can then set the `experimental_feature` flag `decorator_syntax` in your code

```py
global_config.experimental_features["decorator_syntax"] = True
```

Read the full guide on decorators in [the decorator user guide](../user-guides/decorators.md).

## Graduated features

Once an experimental feature is robust and reliable, we "graduate" them to allow their use without setting the
`experimental_features` flag of the `global_config`. This comes with better support and guarantees for their feature
set. We list graduated features here so you can keep up to date.

### `RunnerScriptConstructor` (since 5.10)

The `RunnerScriptConstructor` found in `hera.workflows.script` and seen in the
[callable script example](../examples/workflows/scripts/callable_script.md) is a robust way to run Python functions on
Argo. The image used by the script should be built from the source code package itself and its dependencies, so that the
source code's functions, dependencies, and Hera itself are available to run. The `RunnerScriptConstructor` is also
compatible with Pydantic so supports deserializing inputs to Python objects and serializing outputs to json strings.

Read [the Script Guide](../user-guides/script-basics.md#runnerscriptconstructor) to learn more!

### Script Annotations (since 5.19)

Annotation syntax using `typing.Annotated` is supported for `Parameter`s and `Artifact`s as inputs and outputs for
functions decorated as `scripts`. They use `Annotated` as the type in the function parameters and allow us to simplify
writing scripts with parameters and artifacts that require additional fields such as a `description` or alternative
`name`.

Read the full guide on script annotations in [the script user guide](../user-guides/script-annotations.md).
2 changes: 0 additions & 2 deletions docs/user-guides/script-runner-io.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,6 @@ def my_function(my_input: MyInput) -> MyOutput:
another_param_int_output=-1,
a_str_param_output="Hello, world!",
)

```

Using the IO classes requires use of the Hera Runner and the `"script_pydantic_io"` experimental feature flag to be
Expand Down Expand Up @@ -149,7 +148,6 @@ class MyOutput(Output):
@script(constructor="runner")
def pydantic_io() -> MyOutput:
return MyOutput(exit_code=1, result="Test!", param_int=42, artifact_int=my_input.param_int)
```

See the full Pydantic IO example [here](../examples/workflows/experimental/script_runner_io.md)!
61 changes: 1 addition & 60 deletions docs/walk-through/experimental-hera-features.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,63 +15,4 @@ global_config.experimental_features["NAME_OF_FEATURE"] = True
Note that experimental features are subject to breaking changes in future releases of the same major version. We will
usually announce changes in [the Hera slack channel](https://cloud-native.slack.com/archives/C03NRMD9KPY).

## Currently supported experimental features:

### Script IO Models

Hera provides Pydantic models for you to create subclasses from, which allow you to more easily declare script template
inputs. Any fields that you declare in your subclass of `Input` will become input parameters or artifacts, while
`Output` fields will become output parameters artifacts. The fields that you declare can be `Annotated` as a `Parameter`
or `Artifact`, as any fields with a basic type will become `Parameters`. Turning on the `script_pydantic_io` flag will
automatically enable the `script_annotations` experimental feature.

To enable Hera input/output models, you must set the `experimental_feature` flag `script_pydantic_io`

```py
global_config.experimental_features["script_pydantic_io"] = True
```

Read the full guide on script pydantic IO in [the script user guide](../user-guides/script-runner-io.md).

### Decorators for main template types

Decorators for dags, steps and containers are provided alongside a new script decorator, letting your declare Workflows via Python functions alone.

To enable the decorators, you must install the optional extras under `experimental`

```bash
pip install hera[experimental]
```

You can then set the `experimental_feature` flag `decorator_syntax` in your code

```py
global_config.experimental_features["decorator_syntax"] = True
```

Read the full guide on decorators in [the decorator user guide](../user-guides/decorators.md).

## Graduated features

Once an experimental feature is robust and reliable, we "graduate" them to allow their use without setting the
`experimental_features` flag of the `global_config`. This comes with better support and guarantees for their feature
set. We list graduated features here so you can keep up to date.

### `RunnerScriptConstructor` (since 5.10)

The `RunnerScriptConstructor` found in `hera.workflows.script` and seen in the
[callable script example](../examples/workflows/scripts/callable_script.md) is a robust way to run Python functions on
Argo. The image used by the script should be built from the source code package itself and its dependencies, so that the
source code's functions, dependencies, and Hera itself are available to run. The `RunnerScriptConstructor` is also
compatible with Pydantic so supports deserializing inputs to Python objects and serializing outputs to json strings.

Read [the Script Guide](../user-guides/script-basics.md#runnerscriptconstructor) to learn more!

### Script Annotations (since 5.19)

Annotation syntax using `typing.Annotated` is supported for `Parameter`s and `Artifact`s as inputs and outputs for
functions decorated as `scripts`. They use `Annotated` as the type in the function parameters and allow us to simplify
writing scripts with parameters and artifacts that require additional fields such as a `description` or alternative
`name`.

Read the full guide on script annotations in [the script user guide](../user-guides/script-annotations.md).
Read about current experimental features in [the user guide](../user-guides/experimental-features.md).
2 changes: 1 addition & 1 deletion docs/walk-through/template-features.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Template Features

This section signposts some `template` features found in Argo and their equivalent in Hera, but are beyond the scope of the Walk Through.
This section signposts some `template` features found in Argo and their equivalent in Hera, which are beyond the scope of the Walk Through.

## Template-Level Lifecycle Hooks

Expand Down
2 changes: 1 addition & 1 deletion docs/walk-through/workflow-features.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Workflow Features

This section signposts some more Workflow features found in Argo, but are beyond the scope of the Walk Through.
This section signposts some more Workflow features found in Argo, which are beyond the scope of the Walk Through.

## Workflow-Level Lifecycle Hooks

Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ nav:
- Script Runner IO: user-guides/script-runner-io.md
- Decorators: user-guides/decorators.md
- Expr Transpiler: user-guides/expr.md
- Experimental Features: user-guides/experimental-features.md
- Examples:
- About: examples/workflows-examples.md
- Idiomatic Hera Examples:
Expand Down

0 comments on commit 655911f

Please sign in to comment.