Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add job group execution limit option #4457

Merged
merged 7 commits into from
Aug 8, 2023
Merged

Conversation

mdegat01
Copy link
Contributor

Proposed change

Replace process_lock decorator job decorator by adding a new JOB_GROUP execution limitation option. This option allows a thread to grab the lock for the group and then freely make calls to methods which share this common lock while still locking other callers out. Instead of having public methods using a lock and private methods that route around the lock to allow one method to call another after obtaining the singular lock on the class.

Lock ownership is tracked via a contextvar which is cloned per asyncio task. This way we can easily determine what task has grabbed hold of a job group lock and which haven't.

In terms of one job calling another, each job defines what it has access to irrespective of its parent. So if a job in a job group calls another job in that job group, that works and the chain can continue. If a job calls another job in a different job group or not in a job group at all then the inner job cannot make more calls in the original job group until control returns to the parent.

This is the start of #1946 . It implements job groups and leverages them. Progress/stage options exist but are not currently used and there is no way to list all running jobs with the API. Those things can come in a subsequent PR.

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New feature (which adds functionality to the supervisor)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to documentation pull request:
  • Link to cli pull request:

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • The code has been formatted using Black (black --fast supervisor tests)
  • Tests have been added to verify that the new code works.

If API endpoints of add-on configuration are added/changed:

@mdegat01 mdegat01 added the new-feature A new feature label Jul 27, 2023
@mdegat01 mdegat01 requested review from agners and pvizeli July 27, 2023 20:52
@mdegat01 mdegat01 force-pushed the job-groups-lock-management branch from 1b7015a to 2ee6377 Compare July 28, 2023 03:17
Copy link
Member

@agners agners left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than that, LGTM.

supervisor/jobs/__init__.py Outdated Show resolved Hide resolved
supervisor/exceptions.py Outdated Show resolved Hide resolved
supervisor/jobs/__init__.py Show resolved Hide resolved
@mdegat01 mdegat01 force-pushed the job-groups-lock-management branch from b4b55ea to c4d7849 Compare August 8, 2023 20:03
@mdegat01 mdegat01 merged commit 1611bec into main Aug 8, 2023
@mdegat01 mdegat01 deleted the job-groups-lock-management branch August 8, 2023 20:49
@github-actions github-actions bot locked and limited conversation to collaborators Aug 10, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants