[Core] GPU placement group doesn't honor the bundle index

[jjyao]

What happened + What you expected to happen

For the reproduction script below, I would expect that each bundle is tied to a GPU so the output should be "0, 1, 2, 3" but now it's "0, 0, 1, 1"

Versions / Dependencies

master

Reproduction script

import ray
from ray.util.placement_group import placement_group
from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy

ray.init(num_gpus=4)

@ray.remote(num_gpus=0.5, num_cpus=0)
def gpu_task():
    import os
    print(f"task is running on GPU {os.environ['CUDA_VISIBLE_DEVICES']}")

pg = placement_group([{'GPU': 1}] * 4, strategy="SPREAD")
ray.get(pg.ready())
futures = [gpu_task.options(
    scheduling_strategy=PlacementGroupSchedulingStrategy(
        placement_group=pg, placement_group_bundle_index=i)).remote() for i in range(4)]
ray.get(futures)

Issue Severity

No response

[jjyao]

cc @rkooo567

[cadedaniel]

Isn't this expected unless using STRICT_SPREAD? i.e. no strong guarantees on spread, just an atttempt?

edit: I see the slack discussion now, will read there.

[wuisawesome]

Btw how are we supposed to interpret the @ray.remote(num_gpus=0.5, num_cpus=0)? Is that supposed to be overridden by the placement group? I actually would've thought that this should hang forever because there aren't enough resources (We ask for 0.5 gpu's, in addition to the placement group but all the gpu's are reserved)

[rkooo567]

Right now, this means the task needs 0.5 GPU from a given placement group bundle (if specified). If not, take 0.5 GPU from any random bundles

[stale]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[Basasuya]

@rkooo567 any progression for this problem? we have the same problem
But we have no idea about why this would happen

[MengjinYan]

Update: I'm actively working on the issue right now. We have a path forward to fix the bug and I'm currently finishing up on the code changes. I'll have a PR soon.