From 964f4008a9fceccdb5f066a395b4cd68dbfafc2a Mon Sep 17 00:00:00 2001 From: Elliot Gunton Date: Wed, 11 Dec 2024 10:24:54 +0000 Subject: [PATCH] Update walk through and user guides (#1288) **Pull Request Checklist** - [x] Fixes #1273 - [ ] ~Tests added~ Docs only - [x] Documentation/examples added - [x] [Good commit messages](https://cbea.ms/git-commit/) and/or PR title **Description of PR** Main changes: * Surface the "hera developer features" walk through section in the walk through before the "authentication" section so users start seeing Hera's unique features earlier * Added the "Set class defaults" section here * Separate out the experimental features into its own section as it's big enough * Rename the "advanced features" section to "further reading" to be less scary * Remove the under-utilised Pydantic integration section, move relevant sections of text to the user guides - new "Integrated Pydantic Support" section in script-basics.md for the RunnerScriptConstructor * Add more motivation to the Runner IO to make it more obvious why it's needed --------- Signed-off-by: Elliot Gunton --- Makefile | 8 +- .../experimental-features.md} | 90 +------------ docs/user-guides/script-annotations.md | 2 + docs/user-guides/script-basics.md | 28 +++++ docs/user-guides/script-runner-io.md | 58 ++++++++- docs/walk-through/authentication.md | 2 +- .../experimental-hera-features.md | 18 +++ docs/walk-through/hera-developer-features.md | 118 ++++++++++++++++++ docs/walk-through/pydantic-support.md | 20 --- ...plate-features.md => template-features.md} | 4 +- ...kflow-features.md => workflow-features.md} | 6 +- mkdocs.yml | 13 +- 12 files changed, 241 insertions(+), 126 deletions(-) rename docs/{walk-through/advanced-hera-features.md => user-guides/experimental-features.md} (50%) create mode 100644 docs/walk-through/experimental-hera-features.md create mode 100644 docs/walk-through/hera-developer-features.md delete mode 100644 docs/walk-through/pydantic-support.md rename docs/walk-through/{advanced-template-features.md => template-features.md} (94%) rename docs/walk-through/{advanced-workflow-features.md => workflow-features.md} (90%) diff --git a/Makefile b/Makefile index 71543de47..c0b9d70a3 100644 --- a/Makefile +++ b/Makefile @@ -125,10 +125,16 @@ examples: ## Generate documentation files for examples @(cd docs && poetry run python generate.py) .PHONY: build-docs -build-docs: ## Generate (and host) documentation locally +build-docs: ## Generate documentation locally @python -m pip install --exists-action=w --no-cache-dir -r docs/requirements.txt @python -m mkdocs build --clean --site-dir build/docs/html --config-file mkdocs.yml +# If you run this target mkdocs will watch the `docs` folder, so any changes +# will be reflected in your browser momentarily (without refreshing!) +.PHONY: host-docs +host-docs: ## Host and open the documentation locally (and rebuild automatically) + @python -m mkdocs serve --open --clean --config-file mkdocs.yml + .PHONY: regenerate-example regenerate-example: ## Regenerates the yaml for a single example, using EXAMPLE_FILENAME envvar regenerate-example: install diff --git a/docs/walk-through/advanced-hera-features.md b/docs/user-guides/experimental-features.md similarity index 50% rename from docs/walk-through/advanced-hera-features.md rename to docs/user-guides/experimental-features.md index 820446f48..d757cb70a 100644 --- a/docs/walk-through/advanced-hera-features.md +++ b/docs/user-guides/experimental-features.md @@ -1,92 +1,4 @@ -# Advanced Hera Features - -This section is used to publicize Hera's features beyond the essentials covered in the walk through. Note that these -features do not exist in Argo as they are specific to Hera. - -## Pre-Build Hooks - -Hera offers a pre-build hook feature through `hera.shared.register_pre_build_hook` with huge flexibility to do pre-build -processing on any type of `template` or `Workflow`. For example, it can be used to conditionally set the `image` of a -`Script`, or set which cluster to submit a `Workflow` to. - -To use this feature, you can write a function that takes an object of type `template` or `Workflow`, does some -processing on the object, then returns it. - -For a simple example, we'll write a function that adds an annotation with a key of "hera", and value of "This workflow -was written in Hera!" - -```py -from hera.shared import register_pre_build_hook -from hera.workflows import Workflow - -@register_pre_build_hook -def set_workflow_default_labels(workflow: Workflow) -> Workflow: - if workflow.annotations is None: - workflow.annotations = {} - - workflow.annotations["hera-annotation"] = "This workflow was written in Hera!" - return workflow - -``` - -Now, any time `build` is called on the Workflow (e.g. to submit it or dump it to yaml), it will add in the annotation! - -## Load YAML from File - -Hera's `Workflow` classes offer a collection of `to` and `from` functions for `dict`, `yaml` and `file`. This -means you can load YAML files and manipulate them as Hera objects! - -```py - with Workflow.from_file("./workflow.yaml") as w: - w.entrypoint = "my-new-dag-entrypoint" - - with DAG(name="my-new-dag-entrypoint"): - ... # Add some tasks! - - w.create() # And submit to Argo directly from Hera! -``` - -The following are all valid assertions: - -```py -with Workflow(name="w") as w: - pass - -assert w == Workflow.from_dict(w.to_dict()) -assert w == Workflow.from_yaml(w.to_yaml()) -assert w == Workflow.from_file(w.to_file()) -``` - -## Submit WorkflowTemplates and ClusterWorkflowTemplates as Workflows - -This feature is available for `WorkflowTemplates` and `ClusterWorkflowTemplates`, and helps you, as a dev, iterate on -your `WorkflowTemplate` until it's ready to be deployed. Calling `create_as_workflow` on a `WorkflowTemplate` will -create a `Workflow` on the fly which is submitted to the Argo cluster directly and given a generated name, meaning you -don't need to first submit the `WorkflowTemplate` itself! What this means is you don't need to keep deleting your -`WorkflowTemplate` and submitting it again, to then run `argo submit --from WorkflowTemplate/my-wt` while iterating -on your `WorkflowTemplate`. - -```py -with WorkflowTemplate( - name="my-wt", - namespace="my-namespace", - workflows_service=ws, -) as wt: - cowsay = Container(name="cowsay", image="docker/whalesay", command=["cowsay", "foo"]) - with Steps(name="steps"): - cowsay() - -wt.create_as_workflow(generate_name="my-wt-test-1-") # submitted and given a generated name by Argo like "my-wt-test-1-abcde" -wt.create_as_workflow() # submitted and given a generated name by Argo like "my-wtabcde" -wt.create_as_workflow() # submitted and given a generated name by Argo like "my-wtvwxyz" -``` - -`generate_name` is an optional parameter in case you want to control the exact value of the generated name, similarly to -the regular `Workflow`, otherwise the name of the `WorkflowTemplate` will be used verbatim for `generate_name`. The -Workflow submitted will always use `generate_name` so that you can call it multiple times in a row without naming -conflicts. - -## Experimental Features +# Experimental Features From time to time, Hera will release a new feature under the "experimental feature" flag while we develop the feature and ensure stability. Once the feature is stable and we have decided to support it long-term, it will "graduate" into diff --git a/docs/user-guides/script-annotations.md b/docs/user-guides/script-annotations.md index 1a9138353..c28a9d11d 100644 --- a/docs/user-guides/script-annotations.md +++ b/docs/user-guides/script-annotations.md @@ -121,6 +121,8 @@ def read_dict_artifact(dict_artifact: Annotated[dict, Artifact(loader=ArtifactLo return dict_artifact["my-key"] ``` +#### Pydantic Integration + A dictionary artifact would have no validation on its contents, so having safe code relies on you knowing or manually validating the keys that exist in it. Instead, by specifying a Pydantic type, the dictionary can be automatically validated and parsed to that type: diff --git a/docs/user-guides/script-basics.md b/docs/user-guides/script-basics.md index 9e904fde0..5b2083f6d 100644 --- a/docs/user-guides/script-basics.md +++ b/docs/user-guides/script-basics.md @@ -193,3 +193,31 @@ give a real `image` here, but we assume it exists in this example. Finally, the passed to `source` is dumped to a file, and then the filename is passed as the final `arg` to the `command`. Therefore, the `source` will actually contain a list of parameters as dictionaries, which are dumped to a file which is passed to `hera.workflows.runner`. Of course, this is all handled for you! + +#### Integrated Pydantic Support + +As Argo deals with a limited set of YAML objects (YAML is generally a superset of JSON), Pydantic support is practically +built-in to Hera through Pydantic's serialization to, and from, JSON. Using Pydantic objects (instead of dictionaries) +in Runner Script templates makes them less error-prone, and easier to write! Using Pydantic classes in function inputs +is as simple as inheriting from Pydantic's `BaseModel`. +[Read more about Pydantic models here](https://docs.pydantic.dev/latest/usage/models/). + +```py +from pydantic import BaseModel +from hera.workflows import script + +class MyModel(BaseModel): + my_int: int + my_string: str + +@script(constructor="runner") +def my_pydantic_function(my_pydantic_input: MyModel): + print(my_pydantic_input.my_string, my_pydantic_input.my_int) +``` + +Your functions can also return objects that are serialized, passed to another `Step` as a string argument, and then +de-serialized in another function. This flow can be seen in +[the callable scripts example](../examples/workflows/scripts/callable_script.md). + +Read on to [Script Annotations](script-annotations.md) to learn how to write Script template functions even more +effectively! diff --git a/docs/user-guides/script-runner-io.md b/docs/user-guides/script-runner-io.md index a928e307d..09d4573e4 100644 --- a/docs/user-guides/script-runner-io.md +++ b/docs/user-guides/script-runner-io.md @@ -1,8 +1,59 @@ # Script Runner IO -Hera provides the `Input` and `Output` Pydantic classes which can be used to more succinctly write your -script function inputs and outputs, and requires use of the Hera Runner. Use of these classes also requires the -`"script_pydantic_io"` experimental feature flag to be enabled: +Hera provides the `Input` and `Output` Pydantic classes to help combat sprawling function declarations of inputs and +outputs when using script annotations. It has the added bonus of letting you return values by referencing their name, +instead of setting outputs by position in a `Tuple`. + +It lets you go from a function declaration and `return` like this: + +```py +@script(constructor="runner") +def my_function( + param_int: Annotated[int, Parameter(name="param-input")] = 42, + an_object: Annotated[MyObject, Parameter(name="obj-input")] = MyObject( + a_dict={"my-key": "a-value"}, a_str="hello world!" + ), + artifact_int: Annotated[int, Artifact(name="artifact-input", loader=ArtifactLoader.json)], +) -> Tuple[ + Annotated[int, Parameter(name="param-int-output")], + Annotated[int, Parameter(name="another-param-int-output")], + Annotated[str, Parameter(name="a-str-param-output")], +]: + print(param_int) + ... + return 42, -1, "Hello, world!" # Hope I didn't mix these up! +``` + +to a function that uses the `Input` and `Output` classes: + +```py +from hera.workflows.io import Input, Output + +class MyInput(Input): + param_int: Annotated[int, Parameter(name="param-input")] = 42 + an_object: Annotated[MyObject, Parameter(name="obj-input")] = MyObject( + a_dict={"my-key": "a-value"}, a_str="hello world!" + ) + artifact_int: Annotated[int, Artifact(name="artifact-input", loader=ArtifactLoader.json)] + +class MyOutput(Output): + param_int_output: Annotated[int, Parameter(name="param-int-output")], + another_param_int_output: Annotated[int, Parameter(name="another-param-int-output")], + a_str_param_output: Annotated[str, Parameter(name="a-str-param-output")], + +@script(constructor="runner") +def my_function(my_input: MyInput) -> MyOutput: + print(my_input.param_int) + ... + return MyOutput( + param_int_output=42, + another_param_int_output=-1, + a_str_param_output="Hello, world!", + ) +``` + +Using the IO classes requires use of the Hera Runner and the `"script_pydantic_io"` experimental feature flag to be +enabled: ```py global_config.experimental_features["script_pydantic_io"] = True @@ -99,7 +150,6 @@ class MyOutput(Output): @script(constructor="runner") def pydantic_io() -> MyOutput: return MyOutput(exit_code=1, result="Test!", param_int=42, artifact_int=my_input.param_int) - ``` See the full Pydantic IO example [here](../examples/workflows/experimental/script_runner_io.md)! diff --git a/docs/walk-through/authentication.md b/docs/walk-through/authentication.md index 397f991da..4a7d0ef70 100644 --- a/docs/walk-through/authentication.md +++ b/docs/walk-through/authentication.md @@ -1,4 +1,4 @@ -# Authentication +# Authenticating in Hera The way you authenticate generally depends on your unique organization setup. You could either directly authenticate against the Argo server, or you handle authentication directly through the reverse proxy, or even both. diff --git a/docs/walk-through/experimental-hera-features.md b/docs/walk-through/experimental-hera-features.md new file mode 100644 index 000000000..7984130df --- /dev/null +++ b/docs/walk-through/experimental-hera-features.md @@ -0,0 +1,18 @@ + +# Experimental Hera Features + +From time to time, Hera will release a new feature under the "experimental feature" flag while we develop the feature +and ensure stability. Once the feature is stable and we have decided to support it long-term, it will "graduate" into +a fully-supported feature. + +To enable experimental features you must set the feature by name to `True` in the `global_config.experimental_features` +dictionary before using the feature: + +```py +global_config.experimental_features["NAME_OF_FEATURE"] = True +``` + +Note that experimental features are subject to breaking changes in future releases of the same major version. We will +usually announce changes in [the Hera slack channel](https://cloud-native.slack.com/archives/C03NRMD9KPY). + +Read about current experimental features in [the user guide](../user-guides/experimental-features.md). diff --git a/docs/walk-through/hera-developer-features.md b/docs/walk-through/hera-developer-features.md new file mode 100644 index 000000000..84ffcbba3 --- /dev/null +++ b/docs/walk-through/hera-developer-features.md @@ -0,0 +1,118 @@ +# Hera Developer Features + +Here we showcase more features that help developers write Workflows in Hera. Note that these features do not exist in +Argo as they are specific to Hera, being a Python library. + +## Set Class Defaults + +You are able to set basic default values for any attributes in Hera's custom classes, such as `Script`, `Container` and +`Workflow`, by using `hera.shared.global_config.set_class_defaults`. You pass the class you want to set defaults on, and +then kwargs for all the attributes you want to set (with their respective values). For example, if you wanted to set some +default values for any `Container` objects you create, you would do: + +```py +from hera.shared import global_config +from hera.workflows import Container + +global_config.set_class_defaults( + Container, + image="my-image:latest", + image_pull_policy="Always", + command=["cowsay"], +) +``` + +And then use the `Container` class in your Workflows as normal, but now the `Container` will have pre-populated default +attributes when created: + +```py +with Workflow(name="w") as w: + do_a_cowsay = Container(name="cowsay-container", args=["Hello, world!"]) + with Steps(name="steps"): + do_a_cowsay() +``` + +Notice how we do not need to set `image` or `command`! + +## Pre-Build Hooks + +If [class defaults](#set-class-defaults) don't meet your needs, Hera also offers a pre-build hook feature through +`hera.shared.register_pre_build_hook` with huge flexibility to do pre-build processing on any type of `template` or +`Workflow`. For example, it can be used to conditionally set the `image` of a `Script`, or set which cluster to submit a +`Workflow` to. + +To use this feature, you can write a function that takes an object of type `template` or `Workflow`, does some +processing on the object, then returns it. + +For a simple example, we'll write a function that adds an annotation with a key of "hera-annotation", and value of "This +workflow was written in Hera!" + +```py +from hera.shared import register_pre_build_hook +from hera.workflows import Workflow + +@register_pre_build_hook +def set_workflow_default_labels(workflow: Workflow) -> Workflow: + if workflow.annotations is None: + workflow.annotations = {} + + workflow.annotations["hera-annotation"] = "This workflow was written in Hera!" + return workflow +``` + +Now, any time `build` is called on the Workflow (e.g. to submit it or dump it to yaml), it will add in the annotation! + +## Load YAML from File + +Hera's `Workflow` classes offer a collection of `to` and `from` functions for `dict`, `yaml` and `file`. This +means you can load YAML files and manipulate them as Hera objects! + +```py + with Workflow.from_file("./workflow.yaml") as w: + w.entrypoint = "my-new-dag-entrypoint" + + with DAG(name="my-new-dag-entrypoint"): + ... # Add some tasks! + + w.create() # And submit to Argo directly from Hera! +``` + +The following are all valid assertions: + +```py +with Workflow(name="w") as w: + pass + +assert w == Workflow.from_dict(w.to_dict()) +assert w == Workflow.from_yaml(w.to_yaml()) +assert w == Workflow.from_file(w.to_file()) +``` + +## Submit WorkflowTemplates and ClusterWorkflowTemplates as Workflows + +This feature is available for `WorkflowTemplates` and `ClusterWorkflowTemplates`, and helps you, as a dev, iterate on +your `WorkflowTemplate` until it's ready to be deployed. Calling `create_as_workflow` on a `WorkflowTemplate` will +create a `Workflow` on the fly which is submitted to the Argo cluster directly and given a generated name, meaning you +don't need to first submit the `WorkflowTemplate` itself! What this means is you don't need to keep deleting your +`WorkflowTemplate` and submitting it again, to then run `argo submit --from WorkflowTemplate/my-wt` while iterating +on your `WorkflowTemplate`. + +```py +with WorkflowTemplate( + name="my-wt", + namespace="my-namespace", + workflows_service=ws, +) as wt: + cowsay = Container(name="cowsay", image="docker/whalesay", command=["cowsay", "foo"]) + with Steps(name="steps"): + cowsay() + +wt.create_as_workflow(generate_name="my-wt-test-1-") # submitted and given a generated name by Argo like "my-wt-test-1-abcde" +wt.create_as_workflow() # submitted and given a generated name by Argo like "my-wtabcde" +wt.create_as_workflow() # submitted and given a generated name by Argo like "my-wtvwxyz" +``` + +`generate_name` is an optional parameter in case you want to control the exact value of the generated name, similarly to +the regular `Workflow`, otherwise the name of the `WorkflowTemplate` will be used verbatim for `generate_name`. The +Workflow submitted will always use `generate_name` so that you can call it multiple times in a row without naming +conflicts. diff --git a/docs/walk-through/pydantic-support.md b/docs/walk-through/pydantic-support.md deleted file mode 100644 index 825e6a9ac..000000000 --- a/docs/walk-through/pydantic-support.md +++ /dev/null @@ -1,20 +0,0 @@ -# Integrated Pydantic Support - -## The why and what - -As Argo deals with YAML objects, which are actually a subset of JSON, Pydantic support is practically built-in to Hera -through Pydantic's serialization (to/from JSON) features. Using Pydantic objects (instead of dictionaries) in Script -templates makes them less error-prone, and easier to write! Using Pydantic classes yourself is as simple as inheriting -from Pydantic's `BaseModel`. [Read more about Pydantic models here](https://docs.pydantic.dev/latest/usage/models/). - -## In Hera - -Hera offers some features that explicitly rely on Pydantic. This includes the Hera Runner, which uses Pydantic to -validate the function call. Using Pydantic classes in your function parameters unlocks the powerful serializing and -de-serializing features of Pydantic when running on Argo. Your functions can return objects that are serialized, passed -to another `Step` as a string argument, and then de-serialized in another function. This flow can be seen in -[the callable scripts example](../examples/workflows/scripts/callable_script.md). - -The Script Runner IO experimental feature provides a way to specify template inputs and outputs using the class fields -of the special `Input` and `Output` classes in Hera. Read more in the -[Script Runner IO guide](../user-guides/script-runner-io.md). diff --git a/docs/walk-through/advanced-template-features.md b/docs/walk-through/template-features.md similarity index 94% rename from docs/walk-through/advanced-template-features.md rename to docs/walk-through/template-features.md index ad66b4fa4..87510c030 100644 --- a/docs/walk-through/advanced-template-features.md +++ b/docs/walk-through/template-features.md @@ -1,6 +1,6 @@ -# Advanced Template Features +# Template Features -This section exemplifies `template` features found in Argo, but are beyond the scope of the Walk Through. +This section signposts some `template` features found in Argo and their equivalent in Hera, which are beyond the scope of the Walk Through. ## Template-Level Lifecycle Hooks diff --git a/docs/walk-through/advanced-workflow-features.md b/docs/walk-through/workflow-features.md similarity index 90% rename from docs/walk-through/advanced-workflow-features.md rename to docs/walk-through/workflow-features.md index 09f89557e..70d9ec9e0 100644 --- a/docs/walk-through/advanced-workflow-features.md +++ b/docs/walk-through/workflow-features.md @@ -1,6 +1,6 @@ -# Advanced Workflow Features +# Workflow Features -This section exemplifies Workflow features found in Argo, but are beyond the scope of the Walk Through. +This section signposts some more Workflow features found in Argo, which are beyond the scope of the Walk Through. ## Workflow-Level Lifecycle Hooks @@ -13,7 +13,7 @@ or when the workflow has a `Running` status. ## Exit Handlers (deprecated) Exit handlers are templates that always execute (regardless of success or failure) at the end of a workflow. They were -replaced by workflow-level ([above](#workflow-level-lifecycle-hooks)) and [template-level hooks](./advanced-template-features.md#template-level-lifecycle-hooks). +replaced by workflow-level ([above](#workflow-level-lifecycle-hooks)) and [template-level hooks](./template-features.md#template-level-lifecycle-hooks). Some use cases listed from [Argo Workflows](https://argoproj.github.io/argo-workflows/walk-through/exit-handlers/) include: diff --git a/mkdocs.yml b/mkdocs.yml index 7556b8830..b8a7d3b1a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -18,12 +18,12 @@ nav: - walk-through/loops.md - walk-through/conditionals.md - walk-through/suspending.md - - Advanced Features: - - walk-through/authentication.md - - walk-through/advanced-workflow-features.md - - walk-through/advanced-template-features.md - - walk-through/advanced-hera-features.md - - walk-through/pydantic-support.md + - walk-through/hera-developer-features.md + - walk-through/authentication.md + - walk-through/experimental-hera-features.md + - Further Reading: + - walk-through/workflow-features.md + - walk-through/template-features.md - User Guides: - Core Concepts: user-guides/core-concepts.md - Scripts: @@ -32,6 +32,7 @@ nav: - Script Runner IO: user-guides/script-runner-io.md - Decorators: user-guides/decorators.md - Expr Transpiler: user-guides/expr.md + - Experimental Features: user-guides/experimental-features.md - Examples: - About: examples/workflows-examples.md - Idiomatic Hera Examples: