[Core] GCS FT with redis sentinel #47335

kanwang · 2024-08-26T13:27:08Z

Why are these changes needed?

We want to use redis sentinel to support Ray GCS FT. I opened a ticket here: #46983. Redis sentinel should provide high availability without worrying too much about redis cluster operations.

Related issue number

Closes #46983

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

kanwang · 2024-08-26T13:35:58Z

@jjyao @rkooo567 would you be able to help review this?

Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

kanwang · 2024-09-08T20:34:38Z

hey @rkooo567 gentle ping on this PR.

kanwang · 2024-09-24T16:41:01Z

Hey,
Gentle ping on this. Could I get some review please? thank you!

anyscalesam · 2024-10-09T19:39:17Z

@kanwang per Ray Slack; aiming to get to this early next week

kanwang · 2024-10-17T13:28:59Z

hey @anyscalesam
Any movement here? sorry for keep bugging you guys!

jjyao · 2024-10-17T15:12:59Z

Hi @kanwang sorry, the team is a bit busy right now. We will review it ASAP.

kanwang · 2024-10-25T13:55:31Z

Hey team, any chance that we can get this reviewed this week or early next week? Thank you! cc @jjyao

jjyao

Very sorry for the late review. I'll actively review this PR from now on.

doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.md

jjyao · 2024-10-29T06:21:14Z

src/ray/gcs/redis_context.cc

@@ -506,6 +564,14 @@ Status RedisContext::Connect(const std::string &address,
  // Ray has some restrictions for RedisDB. Validate it here.
  ValidateRedisDB(*this);


If it's Redis Sentinel, what will INFO CLUSTER returns?

It will return empty - basically that section doesn't exist. Here's a full result of INFO on redis sentinel:

# Server redis_version:7.0.7 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:5dad631ce1b0fc10 redis_mode:sentinel os:Linux 6.8.0-1017-aws x86_64 arch_bits:64 monotonic_clock:POSIX clock_gettime multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:9.4.0 process_id:7 process_supervised:no run_id:de8c925350decfbb0abd5940bf718855eb191f8d tcp_port:26379 server_time_usec:1730346466295003 uptime_in_seconds:728236 uptime_in_days:8 hz:14 configured_hz:10 lru_clock:2293218 executable:/redis-sentinel config_file:/etc/redis/sentinel.conf io_threads_active:0 # Clients connected_clients:3 cluster_connections:0 maxclients:10000 client_recent_max_input_buffer:20480 client_recent_max_output_buffer:0 blocked_clients:0 tracking_clients:0 clients_in_timeout_table:0 # Stats total_connections_received:121374 total_commands_processed:2334908 instantaneous_ops_per_sec:3 total_net_input_bytes:135052866 total_net_output_bytes:163753996 total_net_repl_input_bytes:0 total_net_repl_output_bytes:0 instantaneous_input_kbps:0.17 instantaneous_output_kbps:0.02 instantaneous_input_repl_kbps:0.00 instantaneous_output_repl_kbps:0.00 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:12828 evicted_keys:0 evicted_clients:0 total_eviction_exceeded_time:0 current_eviction_exceeded_time:0 keyspace_hits:0 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 pubsubshard_channels:0 latest_fork_usec:0 total_forks:0 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 total_active_defrag_time:0 current_active_defrag_time:0 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_error_replies:97084 dump_payload_sanitizations:0 total_reads_processed:2502068 total_writes_processed:2380696 io_threaded_reads_processed:0 io_threaded_writes_processed:0 reply_buffer_shrinks:29 reply_buffer_expands:0 # CPU used_cpu_sys:1282.061897 used_cpu_user:931.763629 used_cpu_sys_children:0.032320 used_cpu_user_children:0.026019 used_cpu_sys_main_thread:1282.061999 used_cpu_user_main_thread:931.762826 # Sentinel sentinel_masters:1 sentinel_tilt:0 sentinel_tilt_since_seconds:-1 sentinel_running_scripts:0 sentinel_scripts_queue_length:0 sentinel_simulate_failure_flags:0 master0:name=redis-ha,status=ok,address=10.112.187.90:6379,slaves=2,sentinels=3

jjyao · 2024-10-29T06:23:56Z

src/ray/gcs/redis_context.cc

+    // directly. continue otherwise.
+    return sentinel_status;
+  }
+
  // Find the true leader


This is needed only for Redis Cluster.

I feel our high level code for RedisContext::Connect should be

if (redis_cluster) { ValidateRedisCluster() // make sure it only has 1 shard FindMasterUsingMOVED() } else { // redis sentinel FindMasterUsingSentinel() }

thanks for the suggestion! refactored a little bit. so now the logic in RedisContext::Connect should be mostly untouched: added one validation of redis_sentienel and an else branch to handle sentinel.

this should be ready for re-review.

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: kanwang <kerrywang881122@gmail.com>

Signed-off-by: Kan Wang <kan.wang@datadoghq.com> Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

kanwang · 2024-11-18T22:57:52Z

Thanks! Addressed the feedbacks.

Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

jjyao

LG. I'll review the tests later.

Have you tested with real Redis Sentinel?

jjyao · 2024-11-19T01:37:15Z

src/ray/gcs/redis_context.cc

+bool isRedisSentinel(RedisContext &context) {
+  auto reply = context.RunArgvSync(std::vector<std::string>{"INFO", "SENTINEL"});
+  if (reply->IsNil() || reply->IsError() || reply->ReadAsString().length() == 0) {
+    RAY_LOG(INFO) << "failed to get redis sentinel info, continue as a regular redis.";


I'd remove this log and add logs inside ConnectRedisCluster and ConnectRedisSentinel like

RAY_LOG(INFO) << "Connect to Redis cluster/sentinel";

src/ray/gcs/redis_context.cc

jjyao · 2024-11-19T01:39:43Z

src/ray/gcs/redis_context.cc

+      ::redisCommandArgv(context.sync_context(), cmds.size(), argv.data(), argc.data()));
+
+  RAY_CHECK(redis_reply) << "failed to get redis sentinel masters info";
+  RAY_CHECK(redis_reply->type == REDIS_REPLY_ARRAY)


RAY_CHECK_EQ

src/ray/gcs/redis_context.cc

jjyao · 2024-11-19T01:42:28Z

src/ray/gcs/redis_context.cc

+    RAY_LOG(INFO) << "connecting to the redis primary node behind sentinel: " << actual_ip
+                  << ":" << actual_port;
+    context.Disconnect();
+    return context.Connect(actual_ip, std::stoi(actual_port), password, enable_ssl);


When we connect to the primary, IsRedisSentinel will return false now?

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: kanwang <kerrywang881122@gmail.com>

Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

kanwang · 2024-11-19T04:07:44Z

Have you tested with real Redis Sentinel?

Yes, I did test it on redis sentinel locally (with a remote redis sentinel instance).

When we connect to the primary, IsRedisSentinel will return false now?

That is correct. Sentinel is basically a special redis instance that runs in front of redis and its replicas. It knows the topology/failover of the redis instance and its replica. Once you queried the primary address, you basically connect to redis instance directly.

One thing that is still missing is reconnecting. when redis failover, the primary replica information is updated in sentinel, but Ray won't re-connect until head restart. It's acceptable for us for the moment. I've opened an issue before #46982 and I saw there was a more recent issue being worked on #47419. They are not exactly same issue (v.s. sentinel reconnect), but in general if head tries to re-connect to redis instead of fail, I think all cases will be solved. I will see if there's more updates there. or I can try to contribute that too.

src/ray/gcs/redis_context.cc

jjyao

Looks great!!

jjyao · 2024-11-20T00:48:55Z

python/ray/tests/conftest.py

+    ]
+    # setup replicas of the master
+
+    time.sleep(1)


Can we do better than a sleep: i.e. more deterministic

Good point! I updated to a wait function to check all redis instances are up.

jjyao · 2024-11-20T00:49:36Z

python/ray/tests/conftest.py

+    for port in redis_ports[2:]:
+        redis_cli = get_redis_cli(port, False)
+        redis_cli.replicaof("127.0.0.1", master_port)
+    sentinel_process = start_redis_sentinel_instance(


indentation?

jjyao · 2024-11-20T00:50:33Z

python/ray/_private/test_utils.py

@@ -98,6 +98,18 @@ def redis_replicas():
    return int(os.environ.get("TEST_EXTERNAL_REDIS_REPLICAS", "1"))


+def enable_redis_sentinel():


this is not used?

python/ray/_private/test_utils.py

python/ray/tests/test_gcs_fault_tolerance.py

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: kanwang <kerrywang881122@gmail.com>

jjyao · 2024-11-20T06:48:39Z

Let me know when it's ready for another review.

Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

kanwang · 2024-11-20T18:22:23Z

Thank you for approving! Looks like all tests passed. Can you help with merging?

rueian · 2024-11-30T01:34:35Z

Hi @kanwang, does this change already fit your needs? I am a little concerned about the implementation because it apparently does not fully follow the protocol of Redis Sentinel and Redis Cluster.

kanwang · 2024-12-02T04:25:46Z

Hey @rueian It does solve our problem at the moment. Based on my understanding this is how a client should be connecting to Redis Sentinel. It is not handling reconnection, which probably requires a bigger change in Ray. What particular part of the protocol is it not following for Redis Sentinel? Let me know - if it's critical/breaking I can send a fix too.

rueian · 2024-12-02T05:42:05Z

Hi @kanwang, good to know your problem has been solved.

However, the reconnection issue is exactly what I am concerned about. Both Redis Sentinel and Redis Cluster require clients to monitor their topology proactively and connect to the right nodes accordingly, not just at the beginning but throughout the application's lifetime. I know this kind of active monitoring is extremely difficult to implement. But if we don't do that we probably can not say GCS FT supports Sentinel and Cluster publicly because it still fails on Redis topology changes. Another easier way is querying the master before sending a command every time. Also, the way of querying master should be SENTINEL get-master-addr-by-name master-name not SENTINEL MASTERS because there could be multiple master set in a sentinel cluster.

In terms of the Redis Cluster, the DEL DUMMY approach is brilliant, but there are few issues:

The dummy command may need to pass authentication first.
The master of DUMMY is likely not the master of our real command.
Actually, instead of pre-flighting a dummy command, we should just handle the MOVE error every time since the topology may be changed at any time.

Implementing full support to Redis Sentinel and Redis Cluster is extremely hard. Not to say even with the above aggressive reconnection approaches, we still need to keep discovering the latest topology of Sentinel and Cluster, otherwise, GCS FT will still fail if all preconfigured Redis addresses are down.

Signed-off-by: Kan Wang <kan.wang@datadoghq.com> Signed-off-by: Connor Sanders <connor@elastiflow.com>

Signed-off-by: Kan Wang <kan.wang@datadoghq.com> Signed-off-by: hjiang <dentinyhao@gmail.com>

jjyao · 2024-12-11T19:52:31Z

@rueian

Our current solution is that when topology change, GCS will exit and it will be restarted and during restart, it will find the new master. It's not ideal and we want GCS to automatically find the new master without exiting and this is what @MortalHappiness is working on.

kanwang added 2 commits August 23, 2024 09:05

[Core] Ray GCS FT support redis sentinel

c6758bb

Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

update docs and comments

053383f

Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

kanwang changed the title ~~Kan.wang/gcs with sentinel~~ [Core] GCS FT with redis sentinel Aug 26, 2024

kanwang and others added 2 commits August 26, 2024 10:39

lint

3451698

Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

Merge branch 'master' into kan.wang/gcs-with-sentinel

7c22b35

kanwang marked this pull request as ready for review August 26, 2024 15:05

kanwang requested review from architkulkarni, maxpumperla, pcmoritz, kevin85421 and a team as code owners August 26, 2024 15:05

anyscalesam added triage Needs triage (eg: priority, bug/not-bug, and owning component) core Issues that should be addressed in Ray Core labels Aug 26, 2024

anyscalesam assigned rkooo567 Aug 26, 2024

anyscalesam added P1 Issue that should be fixed within a few weeks and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Aug 26, 2024

jjyao added the go add ONLY when ready to merge, run all tests label Oct 29, 2024

jjyao reviewed Oct 29, 2024

View reviewed changes

jjyao assigned jjyao and unassigned rkooo567 Oct 29, 2024

kanwang and others added 2 commits October 30, 2024 23:45

Update doc/source/cluster/kubernetes/user-guides/kuberay-gcs-ft.md

5d5bf40

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: kanwang <kerrywang881122@gmail.com>

refactor based on feedback

b21dbf1

Signed-off-by: Kan Wang <kan.wang@datadoghq.com> Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

update

68c4a2b

Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

fix build

7743033

Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

jjyao reviewed Nov 19, 2024

View reviewed changes

kanwang and others added 3 commits November 18, 2024 22:20

Update src/ray/gcs/redis_context.cc

8db8208

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: kanwang <kerrywang881122@gmail.com>

Update src/ray/gcs/redis_context.cc

a404a05

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: kanwang <kerrywang881122@gmail.com>

update

c2b4fb7

Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

jjyao reviewed Nov 20, 2024

View reviewed changes

src/ray/gcs/redis_context.cc Outdated Show resolved Hide resolved

src/ray/gcs/redis_context.cc Outdated Show resolved Hide resolved

src/ray/gcs/redis_context.cc Outdated Show resolved Hide resolved

src/ray/gcs/redis_context.cc Outdated Show resolved Hide resolved

jjyao reviewed Nov 20, 2024

View reviewed changes

kanwang and others added 7 commits November 19, 2024 23:44

Update src/ray/gcs/redis_context.cc

28a7a98

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: kanwang <kerrywang881122@gmail.com>

Update src/ray/gcs/redis_context.cc

5b17761

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: kanwang <kerrywang881122@gmail.com>

Update src/ray/gcs/redis_context.cc

b7487da

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: kanwang <kerrywang881122@gmail.com>

Update src/ray/gcs/redis_context.cc

d4502ed

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: kanwang <kerrywang881122@gmail.com>

Update python/ray/_private/test_utils.py

dec3e70

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: kanwang <kerrywang881122@gmail.com>

Update python/ray/tests/test_gcs_fault_tolerance.py

dfeb6d5

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: kanwang <kerrywang881122@gmail.com>

Merge branch 'master' into kan.wang/gcs-with-sentinel

3d22195

kanwang added 2 commits November 20, 2024 11:15

udpate tests

837b208

Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

update

0e648e9

Signed-off-by: Kan Wang <kan.wang@datadoghq.com>

jjyao approved these changes Nov 20, 2024

View reviewed changes

jjyao merged commit 52b0768 into ray-project:master Nov 20, 2024
5 checks passed

jecsand838 pushed a commit to jecsand838/ray that referenced this pull request Dec 4, 2024

[Core] GCS FT with redis sentinel (ray-project#47335)

3558655

Signed-off-by: Kan Wang <kan.wang@datadoghq.com> Signed-off-by: Connor Sanders <connor@elastiflow.com>

dentiny pushed a commit to dentiny/ray that referenced this pull request Dec 7, 2024

[Core] GCS FT with redis sentinel (ray-project#47335)

a557c31

Signed-off-by: Kan Wang <kan.wang@datadoghq.com> Signed-off-by: hjiang <dentinyhao@gmail.com>

rueian mentioned this pull request Dec 11, 2024

[Core] Support multi-shard redis cluster #42919

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Core] GCS FT with redis sentinel #47335

[Core] GCS FT with redis sentinel #47335

kanwang commented Aug 26, 2024 •

edited

Loading

kanwang commented Aug 26, 2024

kanwang commented Sep 8, 2024

kanwang commented Sep 24, 2024

anyscalesam commented Oct 9, 2024

kanwang commented Oct 17, 2024

jjyao commented Oct 17, 2024

kanwang commented Oct 25, 2024

jjyao left a comment

jjyao Oct 29, 2024

kanwang Oct 31, 2024

jjyao Oct 29, 2024

kanwang Nov 1, 2024

kanwang Nov 1, 2024

kanwang commented Nov 18, 2024

jjyao left a comment

jjyao Nov 19, 2024

jjyao Nov 19, 2024

jjyao Nov 19, 2024

kanwang commented Nov 19, 2024 •

edited

Loading

jjyao left a comment

jjyao Nov 20, 2024

kanwang Nov 20, 2024

jjyao Nov 20, 2024

jjyao Nov 20, 2024

jjyao commented Nov 20, 2024

kanwang commented Nov 20, 2024

rueian commented Nov 30, 2024

kanwang commented Dec 2, 2024 •

edited

Loading

rueian commented Dec 2, 2024 •

edited

Loading

jjyao commented Dec 11, 2024

		@@ -506,6 +564,14 @@ Status RedisContext::Connect(const std::string &address,
		// Ray has some restrictions for RedisDB. Validate it here.
		ValidateRedisDB(*this);

		@@ -98,6 +98,18 @@ def redis_replicas():
		return int(os.environ.get("TEST_EXTERNAL_REDIS_REPLICAS", "1"))


		def enable_redis_sentinel():

[Core] GCS FT with redis sentinel #47335

[Core] GCS FT with redis sentinel #47335

Conversation

kanwang commented Aug 26, 2024 • edited Loading

Why are these changes needed?

Related issue number

Checks

kanwang commented Aug 26, 2024

kanwang commented Sep 8, 2024

kanwang commented Sep 24, 2024

anyscalesam commented Oct 9, 2024

kanwang commented Oct 17, 2024

jjyao commented Oct 17, 2024

kanwang commented Oct 25, 2024

jjyao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kanwang commented Nov 18, 2024

jjyao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kanwang commented Nov 19, 2024 • edited Loading

jjyao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jjyao commented Nov 20, 2024

kanwang commented Nov 20, 2024

rueian commented Nov 30, 2024

kanwang commented Dec 2, 2024 • edited Loading

rueian commented Dec 2, 2024 • edited Loading

jjyao commented Dec 11, 2024

kanwang commented Aug 26, 2024 •

edited

Loading

kanwang commented Nov 19, 2024 •

edited

Loading

kanwang commented Dec 2, 2024 •

edited

Loading

rueian commented Dec 2, 2024 •

edited

Loading