-
Notifications
You must be signed in to change notification settings - Fork 31
Cassandra cluster can not recover if a C* pod and its data are deleted #350
Comments
I'm going to reclassify this as a feature, as storing the UUID on the Pilot resource is a feature we did not previously support, and so when it is lost through some external means, it is expected that it is not recoverable 😄 /kind feature |
Just for clarification 😄
If I got something wrong please correct me |
Thanks @yanniszark. That's an accurate summary 👍 |
I got too busy with the details and forgot to ask the actual question 😄 |
Yep. :-) |
I am interested in both 😄 |
A little follow-up on this: First, in StorageService.java where the join function for a new node is located: As we can see, it calls DatabaseDescriptor.getReplaceAddress() to get the replace address. This is of InetAddressAndPort type, so incompatible from the start, but we'll keep digging in case it the UUID is resolved before reaching us. The DatabaseDescriptor.getReplaceAddress() function: It calls InetAddressAndPort.getByName to retrieve the replace_address. The relevant function is this : Which calls upon HostAndPort.fromString to get the address, which is part of the Google Core Libraries. Description can be found here. Consequently, it would seem that it is not possible to provide host_id to the replace_address option. In the older releases of Cassandra, there was a replace_node option which accepted UUID, but it was deprecated in favor of replace_address. In this case, it seems the way to go is to store the ip address of the node. |
If a C* Pod and its data are deleted, a replacement Pod will be started by the StatefulSet controller, but it will be started with an empty
/var/lib/cassandra
data directory.This will cause a new Cassandra node UUId to be generated, and when the C* node attempts to join the cluster, it will be considered to be a new, rather than a replacement node.
This is much more likely if the cluster is configured to use persistent local storage rather than re-attachable networked PVs.
You can work around this by storing the C* node UUID as a field on e.g. a Pilot resource or centrally on the CassandraCluster resource, so that when a pilot is restarted, it can discover the original C* node UUID and start the Cassandra process with
-Dcassandra.replace-address=<original_uuid>
.See is cassandra-kubernetes-hostid written as part of Improve Cassandra Example , which suggests that you can supply the "hostid" to
-Dcassandra.replace_address
when starting the C* node./kind bug
The text was updated successfully, but these errors were encountered: