Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RLlib] Various updates to the Release CI RLlib #36883

Conversation

avnishn
Copy link
Member

@avnishn avnishn commented Jun 27, 2023

  • change most of the release test instances to use the faster g5 instances. In theory this should allow us to reduce the timeout window for these tests, since they run faster
  • disable rl_module/learner_api for ppo with attention or LSTM

Signed-off-by: Avnish avnishnarayan@gmail.com

Why are these changes needed?

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

avnishn added 2 commits June 27, 2023 14:54
Signed-off-by: Avnish <avnishnarayan@gmail.com>
Signed-off-by: Avnish <avnishnarayan@gmail.com>
@avnishn
Copy link
Member Author

avnishn commented Jun 27, 2023

@can-anyscale
Copy link
Collaborator

CI doesn't build ray wheel if you change only files under release. Maybe change something in rllib and undo it later to trigger the ray wheel.

Signed-off-by: Avnish <avnishnarayan@gmail.com>
@avnishn
Copy link
Member Author

avnishn commented Jun 28, 2023

hopefully that fixes the wheel problem?
https://buildkite.com/ray-project/release-tests-pr/builds/43874

avnishn added 4 commits June 28, 2023 17:09
…gle node

Signed-off-by: Avnish <avnishnarayan@gmail.com>
Signed-off-by: Avnish <avnishnarayan@gmail.com>
Signed-off-by: Avnish <avnishnarayan@gmail.com>
Signed-off-by: Avnish <avnishnarayan@gmail.com>
@avnishn avnishn requested a review from a team as a code owner June 29, 2023 17:24
@@ -73,7 +73,7 @@ def set_cluster_env(self, cluster_env: Dict[str, Any]):
self.cluster_env_name = (
f"{self.project_name}_{self.project_id[4:8]}"
f"__env__{self.test.get_name().replace('.', '_')}__"
f"{dict_hash(self.cluster_env)}"
f"{dict_hash(self.cluster_env)}__2164728348"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this magic number? a timestamp?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@can-anyscale suggested I do this for getting cluster envs to rebuild properly.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel that you did not answer my question :)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the number in our bank account in 2 years, let's move fast!

@can-anyscale
Copy link
Collaborator

@aslonnie just told me that we have stopped building anyscale/ray-ml:nightly-py37 since 15 days ago so it's critical that all ml release tests need to migrate to py38 at this point, cc: @krfricke

Since the branch cut is tomorrow, let's finish this today or tomorrow so we have proper release test signals during branch cut

Signed-off-by: Avnish <avnishnarayan@gmail.com>
arvind-chandra pushed a commit to lmco/ray that referenced this pull request Aug 31, 2023
Signed-off-by: Avnish <avnishnarayan@gmail.com>
Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants