Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot set cluster_label on memberlist in helm chart #12966

Open
slim-bean opened this issue May 14, 2024 · 1 comment
Open

Cannot set cluster_label on memberlist in helm chart #12966

slim-bean opened this issue May 14, 2024 · 1 comment

Comments

@slim-bean
Copy link
Collaborator

          We have hit 2 issues.
  1. There doesnt appear to be a way to set cluster_label -- Which we have always had to set before to prevent tempo and other grafana products from joining the gossip ring.
  2. The default stateful set alerts from kube-prom-stack alarm about the ingester statefulset.

They are fine AFAIK

Replicas:           1 desired | 1 total
Update Strategy:    OnDelete
Pods Status:        1 Running / 0 Waiting / 0 Succeeded / 0 Failed
❯ kubectl get statefulset -n loki
NAME                   READY   AGE
loki-chunks-cache      1/1     20h
loki-compactor         1/1     20h
loki-index-gateway     2/2     20h
loki-ingester-zone-a   1/1     20h
loki-ingester-zone-b   1/1     20h
loki-ingester-zone-c   1/1     20h
loki-results-cache     1/1     20h
loki-ruler             0/0     20h
KubeStatefulSetUpdateNotRolledOut: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubestatefulsetupdatenotrolledout/

Originally posted by @jseiser in #12506 (comment)

@mogopz
Copy link

mogopz commented May 30, 2024

To give a little context on point 2 (because I'm also seeing the same alert) it's triggering because after updating Loki config only one ingester pod is restarted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants