Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] ray list tasks filter state and name on gcs side #46270

Merged
merged 8 commits into from
Jun 26, 2024

Conversation

jjyao
Copy link
Collaborator

@jjyao jjyao commented Jun 26, 2024

Why are these changes needed?

Push down the task state and name filtering to gcs

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com>
Comment on lines +310 to +313
elif key == "name":
req_filters.name = value
elif key == "state":
req_filters.state = value
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main change 1

@@ -423,6 +423,27 @@ void GcsTaskManager::HandleGetTaskEvents(rpc::GetTaskEventsRequest request,
return false;
}

if (filters.has_state()) {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

main change 2

@@ -244,6 +232,9 @@ message TaskStateUpdate {
optional int32 worker_pid = 12;
// Is task paused by debugger.
optional bool is_debugger_paused = 13;
// Key is the integer value of TaskStatus enum (protobuf doesn't support Enum as key).
// Value is the timestamp when status changes to the target status indicated by the key.
map<int32, int64> state_ts = 14;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

main change 3

Copy link
Contributor

@rickyyx rickyyx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty clean to me!

src/ray/gcs/pb_util.h Show resolved Hide resolved
src/ray/protobuf/gcs_service.proto Show resolved Hide resolved
Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com>
@jjyao jjyao added the go add ONLY when ready to merge, run all tests label Jun 26, 2024

// Figure out the latest state of a task.
ray::rpc::TaskStatus state = ray::rpc::TaskStatus::NIL;
if (task_event.has_state_updates()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we are also doing something similar at the client when aggregating later, i wonder if this would go out of sync with that routine.

I wonder if it makes sense to add a current_state to the task event when returned (even when there's no state filter), so we don't need to do that at the client.

This probably could go in another PR.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, we could do that. But based on the current logic, it shouldn't go out of sync since we just iterate through TaskStatus and find the last state that has timestamp.

Having a current_state or define the common function in cython is a good follow-up to do.

Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com>
Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com>
Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com>
Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com>
Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com>
Signed-off-by: Jiajun Yao <jeromeyjj@gmail.com>
@jjyao jjyao marked this pull request as ready for review June 26, 2024 14:35
@jjyao jjyao requested review from a team, ericl, pcmoritz and raulchen as code owners June 26, 2024 14:35
@jjyao jjyao merged commit 29fa5cd into ray-project:master Jun 26, 2024
6 checks passed
@jjyao jjyao deleted the jjyao/filterr branch June 26, 2024 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
go add ONLY when ready to merge, run all tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants