Skip to content
This repository has been archived by the owner on Oct 9, 2023. It is now read-only.

Fixing max parallelism #611

Merged
merged 2 commits into from
Aug 30, 2023
Merged

Fixing max parallelism #611

merged 2 commits into from
Aug 30, 2023

Conversation

pingsutw
Copy link
Member

TL;DR

I run two tasks in a workflow, but max Parallelism is being set to 4.

{"json":{"exec_id":"ahcxqvqbwjs5zczg8d9b","node":"n0","ns":"flytesnacks-development","res_ver":"180043","routine":"worker-2","src":"handler.go:582","tasktype":"python-task","wf":"flytesnacks:development:getting_start.wf"},"level":"info","msg":"Parallelism now set to [1].","ts":"2023-08-29T14:37:15-07:00"}
{"json":{"exec_id":"ahcxqvqbwjs5zczg8d9b","node":"n0","ns":"flytesnacks-development","res_ver":"180043","routine":"worker-2","src":"handler.go:540","tasktype":"python-task","wf":"flytesnacks:development:getting_start.wf"},"level":"info","msg":"Parallelism now set to [2].","ts":"2023-08-29T14:37:15-07:00"}
{"json":{"exec_id":"ahcxqvqbwjs5zczg8d9b","node":"n0","ns":"flytesnacks-development","res_ver":"180043","routine":"worker-2","src":"handler.go:91","wf":"flytesnacks:development:getting_start.wf"},"level":"info","msg":"regular node detected, (no future file found)","ts":"2023-08-29T14:37:15-07:00"}
{"json":{"exec_id":"ahcxqvqbwjs5zczg8d9b","node":"n1","ns":"flytesnacks-development","res_ver":"180043","routine":"worker-2","src":"handler.go:183","wf":"flytesnacks:development:getting_start.wf"},"level":"info","msg":"Dynamic handler.Handle's called with phase 0.","ts":"2023-08-29T14:37:15-07:00"}
{"json":{"exec_id":"ahcxqvqbwjs5zczg8d9b","node":"n1","ns":"flytesnacks-development","res_ver":"180043","routine":"worker-2","src":"handler.go:582","tasktype":"python-task","wf":"flytesnacks:development:getting_start.wf"},"level":"info","msg":"Parallelism now set to [3].","ts":"2023-08-29T14:37:15-07:00"}
{"json":{"exec_id":"ahcxqvqbwjs5zczg8d9b","node":"n1","ns":"flytesnacks-development","res_ver":"180043","routine":"worker-2","src":"handler.go:540","tasktype":"python-task","wf":"flytesnacks:development:getting_start.wf"},"level":"info","msg":"Parallelism now set to [4].","ts":"2023-08-29T14:37:15-07:00"}

We use defer func to update the max parallelism, so we don't need to update the parallelism in other places in the handle()

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

^^^

Tracking Issue

NA

Follow-up issue

NA

Signed-off-by: Kevin Su <pingsutw@apache.org>
@codecov
Copy link

codecov bot commented Aug 29, 2023

Codecov Report

Merging #611 (5198686) into master (ebc0ae7) will increase coverage by 0.42%.
The diff coverage is n/a.

❗ Current head 5198686 differs from pull request most recent head 3448fcd. Consider uploading reports for the commit 3448fcd to get more accurate results

Additional details and impacted files

@hamersaw hamersaw merged commit 476d416 into master Aug 30, 2023
14 checks passed
@hamersaw hamersaw deleted the fixing-parallelism branch August 30, 2023 01:50
eapolinario pushed a commit that referenced this pull request Sep 6, 2023
Signed-off-by: Kevin Su <pingsutw@apache.org>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants