[serve] Non-atomic shutdown logic may lead to unexpected behavior

[edoakes]

The shutdown logic currently happens in three steps from the client:

1. we call shutdown.remote() on the controller, which signals that the proxies and deployment should be shut down
2. we wait for all deployments to be shut down
3. we ray.kill the controller

However, because these three operations all run from the client, the user could interrupt them and cause the controller to be shut down but not killed. This can lead to an unexpected state, such as the controller silently ignoring updates or no longer health checking HTTP proxies.

We should make the shutdown logic atomic from the perspective of the controller: this should set a shutting_down flag that causes the normal update loop to shut down the components and then have the controller exit itself using exit_actor. We should also reject any state updates once the controller is shutting down.

[edoakes]

Next step: @GeneDer to reproduce the "phantom HTTP proxy" issue we saw by manually triggering this issue.

[edoakes]

Let's also please improve the logging around shutdown when we fix this.

[GeneDer]

This is confirmed with the following step to reproduce:

1. add a long sleep of 10s https://github.com/ray-project/ray/blob/ray-2.5.0/python/ray/serve/_private/client.py#L110
2. start ray locally with ray start --head
3. run ray with config locally serve run config.yaml
4. checked both http://localhost:8000/ and http://localhost:8265/#/serve looks good at this point
5. control + c twice to force kill
6. http://localhost:8000/ is not reachable at this point
7. http://localhost:8265/#/serve continue to show proxy healthy and the deployment is running
8. http://localhost:8265/#/actors is showing HTTPProxyActor has a state of DEAD

[edoakes]

This has been happening frequently on anyscale workspaces (reported by customers and internal folks), so bumping it to a P0 release blocker.

[zhe-thoughts]

Small point: let's close release-blocking issues only after the cherry-pick PR is merged into the release branch

[GeneDer]

Cherry-pick PR is merged #37211