Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for recursive list, dict, and tuple ref parsing for pipeline controller.add step() parameter override #1099

Merged
merged 3 commits into from
Sep 9, 2023
Merged

Add support for recursive list, dict, and tuple ref parsing for pipeline controller.add step() parameter override #1099

merged 3 commits into from
Sep 9, 2023

Conversation

natephysics
Copy link
Contributor

Related Issue \ discussion

#1089

Patch Description

Adds support when using PipelineController.add_step(). Currently, it's impossible to parse anything besides strings with _parse_step_ref() in parameter_override. This patch allows list, dics, and tuples to be recursively parsed for step reference substitution. Since recursive features can add computational complexity the new feature is hidden behind a flag that can be passed when adding a step, which makes this a non-breaking change.

Testing Instructions

If recursively_parse_parameters=True in PipelineController.add_step(), recursively parse parameters from parameter_override in lists, dicts, or tuples.
Examples:
- parameter_override={'Args/input_file': ['${<step_name>.artifacts.<artifact_name>.url}', 'file2.txt']} will be correctly parsed.
- parameter_override={'Args/input_file': ('${<step_name_1>.parameters.Args/input_file}', '${<step_name_2>.parameters.Args/input_file}')} will be correctly parsed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants