Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix output param getter #565

Merged
merged 4 commits into from
Apr 8, 2023
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 98 additions & 0 deletions docs/examples/workflows/dag-with-script-output-param-passing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Dag-With-Script-Output-Param-Passing






=== "Hera"

```python linenums="1"
from hera.workflows import (
DAG,
Parameter,
Task,
Workflow,
models as m,
script,
)


@script(outputs=[Parameter(name="a", value_from=m.ValueFrom(path="/test"))])
def out():
with open("/test", "w") as f_out:
f_out.write("test")


@script()
def in_(a):
print(a)


with Workflow(generate_name="script-output-param-passing-", entrypoint="d") as w:
with DAG(name="d"):
t1: Task = out()
t2 = in_(arguments=t1.get_parameter("a"))
t1 >> t2
```

=== "YAML"

```yaml linenums="1"
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: script-output-param-passing-
spec:
entrypoint: d
templates:
- dag:
tasks:
- name: out
template: out
- arguments:
parameters:
- name: a
value: '{{tasks.out.outputs.parameters.a}}'
depends: out
name: in-
template: in-
name: d
- name: out
outputs:
parameters:
- name: a
valueFrom:
path: /test
script:
command:
- python
image: python:3.8
source: "import os\nimport sys\nsys.path.append(os.getcwd())\nwith open(\"/test\"\
, \"w\") as f_out:\n f_out.write(\"test\")\n"
- inputs:
parameters:
- name: a
name: in-
script:
command:
- python
image: python:3.8
source: 'import os

import sys

sys.path.append(os.getcwd())

import json

try: a = json.loads(r''''''{{inputs.parameters.a}}'''''')

except: a = r''''''{{inputs.parameters.a}}''''''


print(a)

'
```

55 changes: 55 additions & 0 deletions docs/examples/workflows/upstream/retry_script.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Retry Script

> Note: This example is a replication of an Argo Workflow example in Hera. The upstream example can be [found here](https://github.com/argoproj/argo-workflows/blob/master/examples/retry-script.yaml).




=== "Hera"

```python linenums="1"
from hera.workflows import RetryStrategy, Workflow, script


@script(image="python:alpine3.6", retry_strategy=RetryStrategy(limit=10), add_cwd_to_sys_path=False)
def retry_script():
import random
import sys

exit_code = random.choice([0, 1, 1])
sys.exit(exit_code)


with Workflow(generate_name="retry-script-", entrypoint="retry-script") as w:
retry_script()
```

=== "YAML"

```yaml linenums="1"
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: retry-script-
spec:
entrypoint: retry-script
templates:
- name: retry-script
retryStrategy:
limit: '10'
script:
command:
- python
image: python:alpine3.6
source: 'import random

import sys


exit_code = random.choice([0, 1, 1])

sys.exit(exit_code)

'
```

26 changes: 26 additions & 0 deletions examples/workflows/dag-with-script-output-param-passing.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
from hera.workflows import (
DAG,
Parameter,
Task,
Workflow,
models as m,
script,
)


@script(outputs=[Parameter(name="a", value_from=m.ValueFrom(path="/test"))])
def out():
with open("/test", "w") as f_out:
f_out.write("test")


@script()
def in_(a):
print(a)


with Workflow(generate_name="script-output-param-passing-", entrypoint="d") as w:
with DAG(name="d"):
t1: Task = out()
t2 = in_(arguments=t1.get_parameter("a"))
t1 >> t2
55 changes: 55 additions & 0 deletions examples/workflows/dag-with-script-output-param-passing.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: script-output-param-passing-
spec:
entrypoint: d
templates:
- dag:
tasks:
- name: out
template: out
- arguments:
parameters:
- name: a
value: '{{tasks.out.outputs.parameters.a}}'
depends: out
name: in-
template: in-
name: d
- name: out
outputs:
parameters:
- name: a
valueFrom:
path: /test
script:
command:
- python
image: python:3.8
source: "import os\nimport sys\nsys.path.append(os.getcwd())\nwith open(\"/test\"\
, \"w\") as f_out:\n f_out.write(\"test\")\n"
- inputs:
parameters:
- name: a
name: in-
script:
command:
- python
image: python:3.8
source: 'import os

import sys

sys.path.append(os.getcwd())

import json

try: a = json.loads(r''''''{{inputs.parameters.a}}'''''')

except: a = r''''''{{inputs.parameters.a}}''''''


print(a)

'
20 changes: 20 additions & 0 deletions examples/workflows/upstream/retry-script.upstream.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# This example demonstrates the use of retries for a single script.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: retry-script-
spec:
entrypoint: retry-script
templates:
- name: retry-script
retryStrategy:
limit: "10"
script:
image: python:alpine3.6
command: ["python"]
# fail with a 66% probability
source: |
import random;
import sys;
exit_code = random.choice([0, 1, 1]);
sys.exit(exit_code)
24 changes: 24 additions & 0 deletions examples/workflows/upstream/retry-script.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: retry-script-
spec:
entrypoint: retry-script
templates:
- name: retry-script
retryStrategy:
limit: '10'
script:
command:
- python
image: python:alpine3.6
source: 'import random

import sys


exit_code = random.choice([0, 1, 1])

sys.exit(exit_code)

'
14 changes: 14 additions & 0 deletions examples/workflows/upstream/retry_script.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
from hera.workflows import RetryStrategy, Workflow, script


@script(image="python:alpine3.6", retry_strategy=RetryStrategy(limit=10), add_cwd_to_sys_path=False)
def retry_script():
import random
import sys

exit_code = random.choice([0, 1, 1])
sys.exit(exit_code)


with Workflow(generate_name="retry-script-", entrypoint="retry-script") as w:
retry_script()
4 changes: 2 additions & 2 deletions src/hera/workflows/retry_strategy.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ class RetryStrategy(_BaseModel):
affinity: Optional[RetryAffinity] = None
backoff: Optional[Backoff] = None
expression: Optional[str] = None
limit: Optional[Union[int, str]] = None
limit: Optional[Union[str, int, IntOrString]] = None
retry_policy: Optional[Union[str, RetryPolicy]] = None

@validator("retry_policy", pre=True)
Expand All @@ -52,7 +52,7 @@ def _convert_limit(cls, v):
if v is None or isinstance(v, IntOrString):
return v

return IntOrString(__root__=str(v)) # int or str
return str(v) # int or str

def build(self) -> _ModelRetryStrategy:
return _ModelRetryStrategy(
Expand Down
6 changes: 2 additions & 4 deletions src/hera/workflows/task.py
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ def finished_at(self) -> str:
def result(self) -> str:
return f"{{{{tasks.{self.name}.outputs.result}}}}"

def get_parameters_as(self, name):
def get_parameters_as(self, name: str) -> Parameter:
"""Gets all the output parameters from this task"""
return Parameter(name=name, value=f"{{{{tasks.{self.name}.outputs.parameters}}}}")

Expand Down Expand Up @@ -181,11 +181,9 @@ def get_parameter(self, name: str) -> Parameter:

obj = next((output for output in parameters if output.name == name), None)
if obj is not None:
obj.value = f"{{{{tasks.{self.name}.outputs.parameters.{name}}}}}"
return Parameter(
name=obj.name,
value=obj.value,
value_from=obj.value_from,
value=f"{{{{tasks.{self.name}.outputs.parameters.{name}}}}}",
global_name=obj.global_name,
Copy link
Collaborator

@sambhav sambhav Apr 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need these other things like global name and description for arguments? I am not sure if they would be valid as argument params.

description=obj.description,
)
Expand Down