[serve] Faster detection of dead replicas

[zcin]

Why are these changes needed?

Detect replica death earlier on handles/routers. Currently routers will process replica death if the actor death error is thrown during active probing or system message.

1. cover one more case: process replica death if error is thrown while request was being processed on the replica.
2. improved handling: if error is detected on the system message, meaning router found out replica is dead after assigning a request to that replica, retry the request.

Performance evaluation

(master results pulled from https://buildkite.com/ray-project/release/builds/21404#01917375-2b1e-4cba-9380-24e557a42a42)

Latency:

metric	master	this PR	% change
http_p50_latency	3.9672044999932154	3.9794859999986443	0.31
http_1mb_p50_latency	4.283115999996312	4.1375990000034335	-3.4
http_10mb_p50_latency	8.212248500001351	8.056774499998198	-1.89
grpc_p50_latency	2.889802499964844	2.845889500008525	-1.52
grpc_1mb_p50_latency	6.320479999999407	9.85005449996379	55.84
grpc_10mb_p50_latency	92.12763850001693	106.14903449999247	15.22
handle_p50_latency	1.7775379999420693	1.6373455000575632	-7.89
handle_1mb_p50_latency	2.797253500034458	2.7225929999303844	-2.67
handle_10mb_p50_latency	11.619127000017215	11.39100950001648	-1.96

Throughput:

metric	master	this PR	% change
http_avg_rps	359.14	357.81	-0.37
http_100_max_ongoing_requests_avg_rps	507.21	515.71	1.68
grpc_avg_rps	506.16	485.92	-4.0
grpc_100_max_ongoing_requests_avg_rps	506.13	486.47	-3.88
handle_avg_rps	604.52	641.66	6.14
handle_100_max_ongoing_requests_avg_rps	1003.45	1039.15	3.56

Results: everything except for grpc results are within noise. As for grpc results, they have always been relatively noisy (see below), so the results are actually also within the noise that we've been seeing. There is also no reason why latency for a request would only increase for grpc and not http or handle for the changes in this PR, so IMO this is safe.

Related issue number

closes #47219

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[edoakes]

Looks good, only nits. Thanks for the detailed benchmarking, it's fantastic! 💪

[edoakes]

hm probably this message should just be logging inside the callback so it's consistent across callsites

[zcin]

do you mean to unify the logging when an error is received on system message (line 559) vs during actual request (line 505)?

[edoakes]

yes and also during active probing

[zcin]

@edoakes Hmm, took a look at code and might be hard to unify. For active probing, tasks are launched + processed in the scheduler, and it seems more straightforward to directly deal with exceptions from probing tasks in the scheduler instead of using a ray object ref callback.