K8SSAND-1563 ⁃ error "Exception encountered during startup: The seed provider lists no seeds" when recreating pods #567
Comments
sync-by-unito
bot
changed the title
|
Additional information as result pod didn't have |
|
This doesn't answer the question about what's happening, but I want to ask what is the use case for deleting the pods? Have you considered using the |
|
This should be investigated. At any point for various reasons, Kubernetes can delete and reschedule pods. k8ssandra-operator and cass-operator need to handle that gracefully. It's also worth noting that cass-operator creates a PodDisruptionBudget for each CassadraDatacenter. It only sets the |
In my case it was cluster for test and I used preemptible workers which are automatically deleted |
adejanovski
added
the
zh:Assess/Investigate
|
The problem seems to be that The reason there still are IPs in the seed endpoints when all the pods have been deleted is the |
This is incorrect. Here is the snippet from the method: if labelSeedBeforeStart {
patch := client.MergeFrom(pod.DeepCopy())
pod.Labels[api.SeedNodeLabel] = "true"
if err := rc.Client.Patch(rc.Ctx, pod, patch); err != nil {
return "", err
}
rc.Recorder.Eventf(rc.Datacenter, corev1.EventTypeNormal, events.LabeledPodAsSeed,
"Labeled pod a seed node %s", pod.Name)
// sleeping five seconds for DNS paranoia
time.Sleep(5 * time.Second)
}You are correct that for a single-DC cluster the seed-service and additional-seeds-service will the same endpoints. When the pods get deleted, those endpoints will get removed from the seed-service. The next time k8ssandra-operators performs reconciliation after the endpoints are removed from the seed-service it will delete the endpoints for the additional-seeds-service. At this point cass-operator should proceed with starting a node in each rack. There is obviously some sort of timing issue since @Tom910 mentioned this didn't happen all the time and deleting the pods a second time got things back to a healthy state. For what it's worth, I've had trouble with preemptible worker nodes and cass-operator in the past and avoided them because of it. This is going to require some more investigation. |
|
Please add your planning poker estimate with ZenHub @burmanm |
|
@jsanda You are right that externalSeedPoints := 0
if existingEndpoints, err := rc.GetAdditionalSeedEndpoint(); err == nil {
externalSeedPoints = len(existingEndpoints.Subsets[0].Addresses)
}
labelSeedBeforeStart := readySeeds == 0 && len(rc.Datacenter.Spec.AdditionalSeeds) == 0 && externalSeedPoints == 0
One of the fixes I see here is to exclude endpoints gathered from the DC when refreshing its
for _, seed := range seeds {
// When building endpoints for `dc`, exclude pods gathered from the `dc` itself.
if labels.GetLabel(&seed, cassdcapi.DatacenterLabel) != dc.Name {
addresses = append(addresses, corev1.EndpointAddress{
IP: seed.Status.PodIP,
})
}
}With the fix the sequence becomes such: Additional changes are required in order to avoid for the clusters to become stuck at start-up without seeds in more general case. One way to do that would be to change |
|
As I mentioned in my previous comment, I think there may either be a timing issue or something that fails to trigger reconciliation in k8ssandra-operator sometimes. I say since the error doesn't happen all the time. I have tried to reproduce several times locally with a kind cluster and have been unable to do so. I deployed a 3 node Cassandra cluster. Every time the additional seeds service endpoints get deleted eventually. The pods get recreated and return to the ready state. We could change the logic in seeds.go as you described, but I would still like to determine the root cause. @hvintus how are you reproducing this? |
|
@jsanda I reproduce just by doing In my recent run additional seeds endpoint is deleted 24 seconds after the seed service got cleared. It is less than a second after the first pod transitioned into Starting state (without a seed label). |
adejanovski
added
zh:Assess/Investigate
adejanovski
added
zh:In-Progress
What happened?
When I have many cass clusters and run
kubectl delete --all pods --namespace k8s-cluster-testafter this command 70% of clusters would recover but 30% of clusters would have the errorException encountered during startup: The seed provider lists no seeds.cluster logs
dns check on cass nodes
At the same time, the cluster is repaired if you delete the pods again
Did you expect to see something different?
I want to see 100% recover clusters
How to reproduce it (as minimally and precisely as possible):
Environment
K8ssandra Operator version:
* Kubernetes version information:github.com/k8ssandra/k8ssandra-operator/config/deployments/control-plane/cluster-scope?ref=v1.1.1
* Kubernetes cluster kind:v1.21.5coredns = 1.6.3I have 40 similar clusters like:
Anything else we need to know?:
I found a similar issue - kubernetes/kubernetes#92559 about DNS cache but I don't have oportunity change this parameters right now
┆Issue is synchronized with this Jira Task by Unito
┆friendlyId: K8SSAND-1563
┆priority: Medium
The text was updated successfully, but these errors were encountered: