[QUESTION] RedisCluster Class surports multiple ip address At initialization？

[HHcamel]

HI,
In cluster mode, if the Node with the configured IP address is down, other nodes are normal. In this case, RedisCluster is initialized with the configured IP address. Can RedisCluster be used normally?
Example as follows:

// redis cluster node's ips, 172.16.3.2/172.16.3.2/172.16.3.4, 
// the node with 172.16.3.2 is bad, but redis cluster can use.
ConnectionOptions connection_options;
connection_options.host = "172.16.3.2";  
connection_options.port = 6379; 
connection_options.password = "auth"; // Optional. No password by default.

RedisCluster cluster1(connection_options);

Thanks for your answer and help.

[wingunder]

Hi @HHcamel,

You could (and probably should) use Redis Sentinel for this. Sentinel works with Redis in cluster mode as well. Please have a look at the redis-plus-plus Sentinel documentation for implementation examples. You'll however need to first setup Sentinel servers, for monitoring your Redis cluster. There are many examples on the www on how to do this.

There are however other ways to handle masters that are down, but your design/architecture must be done in such a way that you can use this. For instance, if your workers would use Redis Streams, and they ACK every handled message, it really doesn't matter if a worker has no access to a healthy master. The other workers would just inherit the load, that the worker without a functional master would have had.

Hope this helped.
Regards

[HHcamel]

Hi, @wingunder ,
Actually, I don't really understand.
In the first paragraph, can Redis Sentinel and Redis Cluster be deployed together? However, I am actually connecting to an already deployed Redis Cluster, which does not have Redis Sentinel deployed.
As for the second paragraph, I don't quite understand it either.
My problem is that the node with the configured IP is abnormal. After the initialization of RedisCluster, it should never get the information of Redis Cluster, although the Redis cluster can be used normally.

Thank you for your help!

[wingunder]

Hi @HHcamel,

In the first paragraph, can Redis Sentinel and Redis Cluster be deployed together? However, I am actually connecting to an already deployed Redis Cluster, which does not have Redis Sentinel deployed.

Redis Sentinel is just a collection of (Redis) servers that monitors either:

• a Redis instance an its replicated Redis nodes

or

• a Redis Cluster that also has replicated Redis nodes

Redis Sentinel can either be deployed in combination (together) with Redis non-clustered instances or Redis Cluster.

I cannot explain how Redis Sentinel works in this PR. Please refer to at least this: https://redis.io/topics/sentinel

My problem is that the node with the configured IP is abnormal. After the initialization of RedisCluster, it should never get the information of Redis Cluster, although the Redis cluster can be used normally.

Your second sentence, above, is hard to understand, but I'll try to explain.
Let's say master A dies, and your RedisCluster class can only connect to A.
If your cluster is set up right, the cluster will promote a slave, say X, to a master, replacing A.
Your cluster should still be a 100% functional, however your app only knows A, so your app will fail to work correctly.

You now have one of two choices:

1. Accept the fact that your app can't function, and rely on other apps, connected to healthy nodes, to do your failing app's work, during the time that it is not functional.
2. Use Sentinel to find another working Redis Cluster or Redis node to connect to.

Regards

[sewenew]

@HHcamel You only need to ensure that the configured node is alive when RedisCluster is initialized. Once RedisCluster connects the node, and gets the slot-node mapping, it does not matter whether the configured node is down or not. The configured node is a seed, used to get all nodes in the cluster. Redis Cluster will do the failover, if one master is down. And RedisCluster can get the new master address from other nodes.

So, normally there's no need to set multiple addresses at initialization.

Regards

[sewenew]

@wingunder In fact, we don't need to use Redis Sentinel to do failover for nodes in Redis Cluster, since Redis Cluster will do the failover by the nodes in the cluster. Redis Sentinel is used for failover of standalone deployment of Redis.

Regards

[wingunder]

Hi @sewenew,

@wingunder In fact, we don't need to use Redis Sentinel to do failover for nodes in Redis Cluster, since Redis Cluster will do the failover by the nodes in the cluster. Redis Sentinel is used for failover of standalone deployment of Redis.

Yes, you're right.
If it fails graciously (like with a fail-over), the RedisCluster connection will be redirected. No problem.
However, the way I understand it, if someone pulls the power plug on the server where the RedisCluster connection is currently made to, the connection will be gone, right? If I understood this wrong, then all my comments about Sentinel in this context are redundant.

Sentinel can be used for finding healthy Redis nodes (regardless if it is a clustered or non-clustered set of Redis servers), and this is what I was trying to explain. If the app is connected to a healthy master and it goes down so catastrophically, that you lose your connection, you can't just re-connect, as that host:port is down. In this case, you could use Sentinel to connect to a healthy master or slave.

[sewenew]

@wingunder If RedisCluster already connects to the cluster, it works normally even if the node is gone. Because it will get the new master address from other nodes in the cluster, and connects to this new master. As you mentioned in the comments, it does not reconnect to the old master.

When working with standalone deployment of Redis, Redis Sentinel works exactly as you mentioned.

However, I don’t think Redis sentinel can be used to monitor nodes in Redis Cluster. Since both sentinel and the cluster itself will do failover, and that might cause conflict, e.g. they might elect different new masters. And this is what I wanted to explain :)

Regards

[HHcamel]

HI, @sewenew ,
I understand what you're saying, but the problem is that I can't guarantee that the configured nodes are active when RedisCluster is initialized.
This may also exist in practice, for example, if the configured node goes down and the app restarts unexpectedly, then the app can no longer connect to the RedisCluster.
It's just my doubts.
Thank you for your help!

[wingunder]

Hi @sewenew

However, I don’t think Redis sentinel can be used to monitor nodes in Redis Cluster. Since both sentinel and the cluster itself will do failover, and that might cause conflict, e.g. they might elect different new masters. And this is what I wanted to explain :)

Interestingly enough, I tried this out last month and I could successfully run Redis Sentinel that monitors a Redis Cluster. I have not had problems with Sentinel trying to failover the cluster, as I set the timeouts for sentinel to be long enough, so that the cluster can do the failover first. Once the failover is done, by the cluster, Sentinel is happy again and does nothing.

The nice thing is however, that one can get info on the masters in the cluster, via Sentinel. For example:

redis-cli -h hostname -p 7100 sentinel masters | grep -A1 -e name -e flags -we port

I'm aware that one can also call cluster nodes on one of the cluster nodes, but this only works if one knows the host:port of at least one of the healthy nodes in the cluster:

redis-cli -h hostname -p 7000 -c cluster nodes

The Sentinel solution for monitoring a Redis Cluster could be seen as an overhead, but this is an independent and easy way to find out the host:port and status of all masters of a Redis Cluster. One can also log (or monitor) what happens to nodes under certain conditions, by subscribing to the Sentinel events. This, I found very handy and much easier than trying to monitor what happens in a Redis Cluster. As far as I know, with Redis Cluster, one cannot subscribe to cluster events, as this does not exists.

[sewenew]

@HHcamel Yes, that's a potential problem. In that case, RedisCluster will throw exception, and you need to update the ip:port manually, i.e. create another RedisCluster. I might try to make it support multiple addresses in the future.

So far, you can try @wingunder's solution, i.e. using a properly configured Redis Sentinel to get the master node address.

Regards

[sewenew]

@wingunder It's a very nice idea by setting the failover timeout long enough, to avoid failover conflict!

This looks like a hidden feature, since no document mentioned this deployment. The only problem is that we have to carefully configure the sentinel to have a longer timeout than Redis Cluster. It should be better, if Redis Sentinel has built-in support for disabling failover.

Regards

[wingunder]

Hi @sewenew,

This looks like a hidden feature, since no document mentioned this deployment. The only problem is that we have to carefully configure the sentinel to have a longer timeout than Redis Cluster. It should be better, if Redis Sentinel has built-in support for disabling failover.

Yes, you're right.

Documenting this and making it official, as well as adding a flag to disable quora-polling and fail-overs, could be a nice enhancement to Sentinel, but this task/request needs to be sent to https://github.com/redis/redis. I have limited time, but will check what I can do.

Regards

[HHcamel]

Hi @sewenew , @wingunder ,

Thank you for your answers and help!

[MattJustMatt]

Following up on this issue to mention native support for multiple initialization addresses would be amazing! It may be possible to handle this in our app logic, but definitely adds some overhead to the initialization process compared to other language Redis clients.

(e.g. the ideal case is I can provide a list of addresses to the cluster client constructor and if one of those nodes happens to be down at init time, redisplusplus will try alternative nodes in the list till it can make the initial connection).

Thank you for considering!