-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Data] Allow specifing both num_cpus
and num_gpus
for map APIs
#47995
Conversation
Signed-off-by: Scott Lee <sjl@anyscale.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OOC -- do you know why we can't set both?
@bveeramani according to the docstring:
|
actually, I'm not sure if this issue still exists with a lot of scheduler changes. |
@raulchen should we just go ahead and remove the restriction then? any other testing that i should do / concerns to be addressed? |
@scottjlee maybe let's update the error to a warning first |
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
num_cpus
and num_gpus
cannot be both set for map operationsnum_cpus
and num_gpus
for map APIs
Signed-off-by: Scott Lee <sjl@anyscale.com>
Why are these changes needed?
Currently, we enforce that
num_cpus
andnum_gpus
cannot be both set for map operations. This PR enables the user to specify both of these parameters, as we believe that with the recent improvements to the scheduler, Ray Data should be able to smoothly support this scenario. However, we will warn users that this is still an experimental feature.Updated doc example (updated for all map APIs):
Related issue number
Checks
git commit -s
) in this PR.scripts/format.sh
to lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/
under thecorresponding
.rst
file.