[Tune] The usage tune.with_parameters leads to unwanted data sharing and possible corrupted results

[jbedorf]

What happened + What you expected to happen

Using the tune.with_parameters functionality results in that Trainables/runs have the risk of receiving the wrong data during launch and/or during restart when using a single Ray cluster. This leads to unexpected and possibly completely wrong results.

Imagine the scenario:

• Launch run X, but the cluster is currently full and as such becomes pending.
• Launch run Y on the same cluster.

Now what happens is that the parameters of run Y are stored under the same key as those of X, and as such overwriting the original values of run X. Now when run X would launch it would read the settings of run Y. This leads to unexpected behaviour as you never know for certain if you are using the right data. Something similar happens if a run is restarted due to a failure as there is a chance the original configuration has been overwritten by that of run launched at a later time.

This is caused by this construct where the prefix will always be the same as it is based on the name of the trainable and does not contain a unique element. However, there must be another cause, as just adding a unique value to the prefix is not enough to solve the problem. I suspect that the _Inner class is not unique either, but I didn't investigate it further as I used another work around.

I solved the issue by just not using the with_parameters construct and instead manually add/remove objects to the object store.

Versions / Dependencies

Master. Python 3.8. Linux

Reproduction script

Usage of the script below (Note make resources_per_trial larger than 50% of your CPUs):

• In terminal 1, launch Ray (ray start --head)
• In terminal 2, run: python tune_test.py 1
• In terminal 3 (withint 30 seconds of first launch),run: python tune_test.py 2

What happens:

• The first run gets launched, and will run for 30 seconds
• The second run gets launched and will be pending
• First run crashes after 30 seconds and gets automatically restarted.
• First run will report that the wrong configuration is used (ValueError: Config: 1 does not match data: 2)

import sys
import time

import ray
from ray import tune

ray.init(address="auto")


class MyTrainable(tune.Trainable):
    def setup(self, config, data=None):
        self.count = 0
        if config["name"] != data["name"]:
            raise ValueError(f"Config: {config['name']} does not match data: {data['name']}")

    def step(self):
        time.sleep(1)
        self.count += 1
        if self.count > 30:
            raise ValueError("Crashing the worker")
        return {"loss": self.count, "done": self.count > 100}


data = {"name": sys.argv[1]}
config = {"name": sys.argv[1]}

tuner = tune.run(
    tune.with_parameters(MyTrainable, data=data),
    resources_per_trial={"cpu": 5},
    config=config,
    max_failures=2,
)

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[justinvyu]

Hi @jbedorf,

Thanks for the detailed description and reproduction script. The issue here seems to be how Tune is registering the MyTrainable. At the start of the first experiment, the Trainable gets registered in the global key value store here, under its name "MyTrainable". Then, the second experiment will actually register again under the same name, this time with new data attached, and this will be used when recovering the trial from the first experiment.

cc: @krfricke This seems related to adding better support for running multiple Tune experiments in parallel.

[xwjiang2010]

Could you elaborate why you want to launch multiple tune runs concurrently in one ray cluster?

[jbedorf]

Due to infrastructure reasons we have a single large Ray cluster used for multiple different tasks and by different users. This works mostly fine, except for the above described situation.

[xwjiang2010]

Got it, I am not sure that Tune is designed to support running multiple jobs in the same ray cluster.
Usually people start multiple clusters with each cluster doing one tuning job.

[hora-anyscale]

Per Triage Sync: Need to confirm in docs this is not supported

[jbedorf]

Got it, I am not sure that Tune is designed to support running multiple jobs in the same ray cluster. Usually people start multiple clusters with each cluster doing one tuning job.

That is interesting. My understanding was that the it is more efficient to have a few large clusters (e.g. less overhead of having multiple head nodes). Especially in multi-user situations, with restricted access permissions, to for example Kubernetes clusters you would expect multiple users sharing a single Ray cluster. Similarly, why would you need a HA Ray cluster if you don't plan on having it shared/robust as users should just spin up small and temporary clusters.

From the general documentation the advised method to run jobs on said cluster is via Ray AIR. Given that the Tuner.fit() is a wrapper around Ray Tune, all those would face similar issues. Similar with features like Ray Jobs, you would expect long running clusters with the ability to do different work in parallel.

What would you suggest to use in situations where the cluster is shared by multiple teams?

[xwjiang2010]

This is the limitation of ray tune. I am not knowledgable enough to speak about the ideal setup of multi-tenancy in Ray. @richardliaw could you loop in the right people here? Thanks!

[walid0925]

fwiw, I've recently run into the same issue even without using tune.with_parameters and just using partial

I launched multiple processes using the same trainable, aka an objective function - but using different datasets for each. one was a regression task and one was a classification task, and the results in the Tune CLI showed that these were intermingling

[krfricke]

The main problem here again is that Ray Tune uses a global key value store to register both the trainable and its parameters. The workaround here is to rename the trainable (or override the __name__ attribute) though multi tenancy is generally not supported at the moment - i.e. we never test it and we don't guarantee that it works. We hope to refactor some of this logic in the future, but it's lower priority.

[jbedorf]

Thanks for the insight @krfricke . Is the multi-tenancy specific to Tune (and thus Ray Air Tuner) or is this Ray wide. Given the addition of Ray Jobs, and services like KubeRay with shareable endpoints I would assume it is set up for sharing. Otherwise there is not much need for having something like Ray Jobs, which makes me wonder what is the supported way to share and use a cluster among multiple users.

[krfricke]

We're working on resolving this issue and hope to include a fix for Ray 2.4.

In the meantime, for reference, this is how you can workaround the issue in practice.

For generic Ray Tune trainables:

    import uuid

    MyTrainable.__name__ = "MyTrainable" + uuid.uuid4().hex[:8]

    tuner = Tuner(
        MyTrainable,
        # ...
    )
    tuner.fit()

For AIR trainers:

    import uuid
    from ray.train.torch import TorchTrainer

    TorchTrainer.__name__ = "TorchTrainer_" + uuid.uuid4().hex[:8]

    trainer = TorchTrainer(
        # ...
    )
    trainer.fit()

[justinvyu]

This has been fixed in 2.4+: #33095