Revert https://github.com/ray-project/ray/pull/46393

[GeneDer]

Why are these changes needed?

Did some bisecting and found out this commit was causing Serve's performance test latency to spike. Reverting 05067f4 to go back to previous state.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

Tested with this build and seeing the following performance metrics:

{
"perf_metric_name": "handle_1mb_p50_latency",
"perf_metric_type": "LATENCY",
"perf_metric_value": 4.620143499948881
},
{
"perf_metric_name": "handle_1mb_p90_latency",
"perf_metric_type": "LATENCY",
"perf_metric_value": 4.826495099950989
},
{
"perf_metric_name": "handle_1mb_p95_latency",
"perf_metric_type": "LATENCY",
"perf_metric_value": 4.9588876999621325
},
{
"perf_metric_name": "handle_1mb_p99_latency",
"perf_metric_type": "LATENCY",
"perf_metric_value": 5.219728340008487
},
{
"perf_metric_name": "handle_10mb_p50_latency",
"perf_metric_type": "LATENCY",
"perf_metric_value": 43.33374250001043
},
{
"perf_metric_name": "handle_10mb_p90_latency",
"perf_metric_type": "LATENCY",
"perf_metric_value": 44.26841059998878
},
{
"perf_metric_name": "handle_10mb_p95_latency",
"perf_metric_type": "LATENCY",
"perf_metric_value": 44.49023079997119
},
{
"perf_metric_name": "handle_10mb_p99_latency",
"perf_metric_type": "LATENCY",
"perf_metric_value": 45.28908214994885
},

[jjyao]

cc @hongchaodeng

[aslonnie]

again, this really should be implemented as a feature flag and being rolled out with a more organized manner.

release tests need to test the same way as production.