Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix shutdown on Ctrl+C for Python source stages #1839

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented Aug 9, 2024

Description

  • Ensure that Python sources which run indefinitely actually shutdown when requested by the pipline (ex: when the user hits Ctrl+C).
  • The KafkaSourceStage already handled this properly, move the stop method from this class into the SingleOutputSource.
  • Add request_stop and is_stop_requested methods to SingleOutputSource.
  • Update all existing source stages which run indefinitely.
  • Add new should_stop_fn constructor argument to DirectoryWatcher, sources which use the watcher (AutoencoderSourceStage & AppShieldSourceStage) pass in their is_stop_requested method allowing the watcher to know when to shut down.
  • Move the RSS source generator method from the RSS source module into the RSSController, and update the RSSSourceStage to use the RSSController directly rather than the RSS module. This avoids the problem where modules require all config values to be JSON serializable, preventing a callback function from being passed to a module.

Closes #1837

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

@dagardner-nv dagardner-nv added bug Something isn't working non-breaking Non-breaking change labels Aug 9, 2024
@dagardner-nv dagardner-nv self-assigned this Aug 9, 2024
@dagardner-nv dagardner-nv requested a review from a team as a code owner August 9, 2024 22:20
@dagardner-nv dagardner-nv marked this pull request as draft August 9, 2024 22:20
@dagardner-nv dagardner-nv changed the title Fix shutdown logic for Python source stages Fix shutdown on Ctrl+C for Python source stages Aug 9, 2024
@dagardner-nv dagardner-nv marked this pull request as ready for review August 9, 2024 22:35
@dagardner-nv dagardner-nv requested a review from a team as a code owner August 13, 2024 21:07
@dagardner-nv
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 5fdbcb9 into nv-morpheus:branch-24.10 Aug 14, 2024
11 checks passed
@dagardner-nv dagardner-nv deleted the david-polling-source-stage-ctrl-c-18337 branch August 14, 2024 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working non-breaking Non-breaking change
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

[BUG]: Many Python source stages do not stop if the user hits Ctrl+C
2 participants