Add --only-output-json-directory Argo Workflow option
#1947
Conversation
trhodeos
changed the title
--only-yaml argo-workflows cli option.--only-json-directory argo-workflows cli option.
trhodeos
changed the title
--only-json-directory argo-workflows cli option.--only-cron-workflow-template-only, --only-event-sources-only, and --only-workflow-template-only cli options
trhodeos
changed the title
--only-cron-workflow-template-only, --only-event-sources-only, and --only-workflow-template-only cli options--only-cron-workflow-only, --only-event-sources-only, and --only-workflow-template-only cli options
trhodeos
changed the title
--only-cron-workflow-only, --only-event-sources-only, and --only-workflow-template-only cli options--only-cron-workflow-json, --only-event-sources-json, and --only-workflow-template-json cli options
trhodeos
changed the title
--only-cron-workflow-json, --only-event-sources-json, and --only-workflow-template-json cli options--only-cron-workflow-json, --only-event-sources-json, and --only-workflow-template-json Argo Workflow options
|
@savingoyal any chance you can take another look? Thanks in advance! |
|
@savingoyal friendly ping.. Would I be able to get another review? |
| @@ -3754,8 +3806,7 @@ class ArgoWorkflowTrigger(object): | |||
| # https://github.com/argoproj/argo-events/blob/master/api/sensor.md#argoproj.io/v1alpha1.ArgoWorkflowTrigger | |||
|
|
|||
| def __init__(self): | |||
| tree = lambda: defaultdict(tree) | |||
| self.payload = tree() | |||
| self.payload = _tree() | |||
There was a problem hiding this comment.
this change seems unrelated to the PR. can you consider pulling this out as a different PR or skipping it?
There was a problem hiding this comment.
Yeah, let me revert
| @@ -2731,7 +2745,7 @@ def _compile_sensor(self): | |||
| ) | |||
|
|
|||
| # Useful to paint the UI | |||
| trigger_annotations = { | |||
| { | |||
There was a problem hiding this comment.
@trhodeos we have impending plans to move away from one sensor -> one workflow template mapping (to save resource costs) to one sensor -> multiple workflow templates which will break the expectations in this PR for --only-event-sensor-json. can you help us with the timelines you are shooting for this PR? happy to chat over zoom if that's helpful.
There was a problem hiding this comment.
depending on your use case, an immediate workaround could also be to override argo_client.py through the extensions such that rather than submitting the CRDs to kubernetes, it just prints them out to stderr.
There was a problem hiding this comment.
but regardless this change is in the right direction. can you help us with details on how you have tested this change? we should be able to test this PR ourselves some point next week.
There was a problem hiding this comment.
One potential solution is to only add a CLI arg --output-json-directory-only and just push all json blobs to that directory. This would essentially allow us to not be tied to the number of resources we generate (as in, adding more sensors is not a problem).
I like the argo_client.py idea, though we'd still like to push directly to argo while we migrate off of using metaflow directly. Still possible, but may require a bit more thinking.
For context: we are trying to use metaflow to generate the resources, spit them out to a gitops repo which has ArgoCD hooked up to it, for better staging of deploys.
As for testing, I'll test locally tomorrow!
This supports outputting _all_ k8s templates into stdout (rather than just the workflowtemplate one via --only-json). Fixes 1940, 1730
trhodeos
changed the title
--only-cron-workflow-json, --only-event-sources-json, and --only-workflow-template-json Argo Workflow options--only-output-json-directory Argo Workflow option
This supports outputting all k8s templates into stdout (rather than just the workflowtemplate one via --only-json).
Fixes #1940, #1730