[tune] Better error message for all pending trials

[richardliaw]

What is the problem?

I've seen numerous times that trials can be printed as all "pending".

We should detect if we're on autoscaling cluster and print a relevant status message. cc @krfricke

[richardliaw]

One example here is: https://discuss.ray.io/t/all-trials-pending-never-running/2758/3

[krfricke]

Yes, this will be very helpful and presumably help uncover other problems as well (e.g. with placement group scheduling)

[richardliaw]

@xwjiang2010 FYI

[xwjiang2010]

Just want to add what I observed in my repro set up.

It seems before migrating to PG, we have this. After migrating to PG, we return here and Executor.has_resources_for_trial() is returning True, although it's clearly otherwise.

@krfricke

[krfricke]

Yes, this was a request by the autoscaler team to ensure that we request unavailable resources (by scheduling placement groups) so that autoscaling is triggered. We might add a check there to only return True if is_ray_cluster() is True