Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deflake fork tests #36109

Closed
wants to merge 2 commits into from
Closed

Conversation

pcmoritz
Copy link
Contributor

@pcmoritz pcmoritz commented Jun 6, 2023

Why are these changes needed?

This fixes #36001

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@pcmoritz
Copy link
Contributor Author

pcmoritz commented Jun 6, 2023

Actually after looking at this more, I realize we actually don't need the subscribers in the worker processes. So a better course of action that keeps fork working in macOS would be:

  1. Remove the import thread
  2. Instantiate the following variables only on the driver: worker.gcs_error_subscriber, worker.gcs_log_subscriber, and get rid of worker.gcs_function_key_subscriber which is not needed any more after we get rid of the import thread

@pcmoritz pcmoritz closed this Jun 6, 2023
@pcmoritz
Copy link
Contributor Author

pcmoritz commented Jun 6, 2023

(^ I made sure the above deflakes the test on a local branch)

pcmoritz added a commit that referenced this pull request Jun 14, 2023
After #36293 is merged we can now fix this.

This fixes #36001 -- I can repro both the hanging and the fact that this PR fixes it on my mac laptop. The rationale for the fix is described in #36109 (comment)
arvind-chandra pushed a commit to lmco/ray that referenced this pull request Aug 31, 2023
After ray-project#36293 is merged we can now fix this.

This fixes ray-project#36001 -- I can repro both the hanging and the fact that this PR fixes it on my mac laptop. The rationale for the fix is described in ray-project#36109 (comment)

Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CI] osx://python/ray/tests:test_basic_4 is failing/flaky on master.
2 participants