Skip to content

Commit

Permalink
[GCS FT] Update Redis connection configs (#40860)
Browse files Browse the repository at this point in the history
## Why are these changes needed?

```
RAY_CONFIG(int64_t, redis_db_connect_retries, 120)
RAY_CONFIG(int64_t, redis_db_connect_wait_milliseconds, 500)
```

`redis_db_connect_retries * redis_db_connect_wait_milliseconds` is not the upper bound but the lower bound for the timeout of reconnection. Check the function [ConnectWithRetries](https://sourcegraph.com/github.com/ray-project/ray@ray-2.7.1/-/blob/src/ray/gcs/redis_context.cc?L360) for more details.

In my experiments, the combination of `redis_db_connect_retries = 600` and `redis_db_connect_wait_milliseconds = 100` takes 13 mins to fail. See [this gist](https://gist.github.com/kevin85421/271841ddc5816340765a44520de834c1) for more details about the experiment. The combination of `redis_db_connect_retries = 120` and `redis_db_connect_wait_milliseconds = 500` takes 3m10s to fail. See [this gist](https://gist.github.com/kevin85421/63fc2936430472213396c59e1a8a7783) for more details.
  • Loading branch information
kevin85421 authored Nov 18, 2023
1 parent 709dc1b commit 25bee34
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions src/ray/common/ray_config_def.h
Original file line number Diff line number Diff line change
Expand Up @@ -321,9 +321,9 @@ RAY_CONFIG(int, worker_oom_score_adjustment, 1000)
/// NOTE: Linux, Unix and MacOS only.
RAY_CONFIG(int, worker_niceness, 15)

/// Allow up to 60 seconds for connecting to Redis.
RAY_CONFIG(int64_t, redis_db_connect_retries, 600)
RAY_CONFIG(int64_t, redis_db_connect_wait_milliseconds, 100)
/// Allow at least 60 seconds for connecting to Redis.
RAY_CONFIG(int64_t, redis_db_connect_retries, 120)
RAY_CONFIG(int64_t, redis_db_connect_wait_milliseconds, 500)

/// Number of retries for a redis request failure.
RAY_CONFIG(size_t, num_redis_request_retries, 5)
Expand Down

0 comments on commit 25bee34

Please sign in to comment.