[Core] Fix jobs longer than 12 days

[Michaelvll]

ray.wait will timeout after 12 days, which is not enough for long-running services, and we have to wait for the job to finish.

Reference: https://github.com/ray-project/ray/blob/ray-2.9.3/python/ray/_private/worker.py#L2845-L2846

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  • sky launch -c test-4 "echo hi; sleep 100; echo bye" --cloud gcp --cpus 2 --num-nodes 4
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[concretevitamin]

Great catch @Michaelvll! +1 on checking with user whether SkyServe's replica recovery kicked in. (LMSys definitely ran longer than 12 days and sounds like this error was transparently recovered?)

[Michaelvll]

Great catch @Michaelvll! +1 on checking with user whether SkyServe's replica recovery kicked in. (LMSys definitely ran longer than 12 days and sounds like this error was transparently recovered?)

Yes, supposedly the issue should be caught by the controller and recovered. The issue was introduced by #3081, where we changed ray.get to ray.wait while the former one does not have the issue for the timeout: https://github.com/ray-project/ray/blob/62655e11ed76509b78654b60be67bc59f8f3460a/python/ray/_private/worker.py#L818

Not sure if LMSYS was using the older version.

[concretevitamin]

Yes, supposedly the issue should be caught by the controller and recovered.

Maybe we can test for this by setting timeout to, say, 10 mins, and observe after 10mins of serving what happens?

[Michaelvll]

Maybe we can test for this by setting timeout to, say, 10 mins, and observe after 10mins of serving what happens?

Ahh, actually, after thinking more, the problem was probably more severe, as the controller job will fail due to the issue, so the recovery will fail...