[core][dashboard][agent] add configurable timeouts for rt env agent a…

…nd job_supervisor (#47481)

GcsClient has a configurable timeout `nums_py_gcs_reconnect_retry`.
However in GcsAioClient it's default to 5 and there's no way to control
it. This PR changes the caller to use the flag
`gcs_rpc_server_reconnect_timeout_s` to make it configurable. It's
already used in agent.py but not in rt env agent and job_supervisor.
This PR fixes all GcsAioClient caller in non-test python codebase.

Note that head.py has retry=0 which should mean infinite retry but it
did not work. Fixes by checking 0-ness.

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

python/ray/_private/gcs_aio_client.py

@@ -8,6 +8,7 @@
 )
 import ray._private.utils
 from ray._private.ray_constants import env_integer
+import ray
 
 # Number of executor threads. No more than this number of concurrent GcsAioClient calls
 # can happen. Extra requests will need to wait for the existing requests to finish.
@@ -52,8 +53,8 @@ def __init__(
         executor=None,
         nums_reconnect_retry: int = 5,
     ):
-        # See https://github.com/ray-project/ray/blob/d0b46eff9ddcf9ec7256dd3a6dda33e7fb7ced95/python/ray/_raylet.pyx#L2693 # noqa: E501
-        timeout_ms = 1000 * (nums_reconnect_retry + 1)
+        # This must be consistent with GcsClient.__cinit__ in _raylet.pyx
+        timeout_ms = ray._config.py_gcs_connect_timeout_s() * 1000
         self.inner = NewGcsClient.standalone(
             str(address), cluster_id=None, timeout_ms=timeout_ms
         )