-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kube-state-metrics keep serving stale metrics after extended apiserver outage #694
Comments
Interesting, thanks for the bug report @bergerx. We depend on Kubernetes client-go reflectors to keep our caches in sync. Reflectors have the option of specifying a resync interval, which we currently disable. I feel like a first step would be to enable the resync interval to have a periodic full resync cycle next to the adhoc diff notifications. In addition we should also add proper instrumentation to the metric store implementation to catch issues like this early. Would this be something you would like to contribute @bergerx? Shouldn't be a big patch, I am happy to guide you along the way. //CC @mgoodness Crossreferencing this Twitter report: https://twitter.com/opsgoodness/status/1098266699382620160 |
I came here from a Google search for the same error and posting here in case they're related. I have a single
and upon viewing logs for that pod I saw similar entries as the above, without the TLS errors
The last log line for my pod is from 5/17, but current date is 5/29. |
that means that that hasn't happened since then, that is a good thing :) watches being renewed is a perfectly normal thing, nothing to be worried about |
Ah, okay. Then I have a different problem I need to track down. Thanks! |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle stale. |
/remove-lifecycle rotten |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle stale We are not hit by this recently, but the original issue doesn't seem to be solved as per @mxinden s first comment in this issue:, the NewReflector is called with kube-state-metrics/internal/store/builder.go Line 336 in 607bf8c
From https://godoc.org/k8s.io/client-go/tools/cache#NewReflector:
|
Resync is really only for external reconciliation, where something external could be modifying the state we're reconciling, but the state that we're reconciling is entirely in-memory state of kube-state-metrics, which is not prone to that. Resync only re-processes items that we already know about, relists already happen automatically on connection resets or similar, which is what the initial report here was about as the apiserver was actually unavailable. |
we hit this issue in ~10 clusters among 45 within ~2 months.
|
Do you run kube-state-metrics in some specific way? With versions pre-1.9.0 you may be experiencing #942, if you're configuring individual namespaces. (note 1.14 is not supported by ksm 1.9.0) Is anyone able to reliably reproduce this issue? |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle stale we're still observing this issue in our clusters. |
@ekesken can you answer the question from Frederic if you are still running into this issue? :) |
we use 10.4.0 version of prometheus helm chart: https://github.com/helm/charts/blame/3944b1421eb039725816c905c00608193226b001/stable/prometheus/Chart.yaml This is our diff for the settings: kubeStateMetrics:
podDisruptionBudget:
enabled: true
priorityClassName: "some-class"
args:
port: 8080
telemetry-port: 8081
namespace: some-namespace,kube-system,argo-cd,default
tolerations:
- key: "some-internal-name.com/dedicated"
operator: "Equal"
value: "cluster-management"
effect: "NoSchedule"
nodeSelector:
unicron.mpi-internal.com/role: "mgmt"
resources:
limits:
cpu: 200m
memory: 256Mi
requests:
cpu: 100m
memory: 200Mi
we're using 1.9.1
yes we do, and as I can see 1.16 is the only supported version for 1.9.5, unfortunately we don't have 1.16 clusters to observe the situation.
we don't have a clue about the reason to reproduce it, but we keep observing the issue in our clusters in AWS EKS with the version |
cc @tariq1890 don't have much experience with the ksm helm chart can you have a look. |
Aha i just came across this in https://github.com/kubernetes/kube-state-metrics/blob/1dfe6681e990623a4078b4e53a2e3c8761e213c6/docs/design/metrics-store-performance-optimization.md:
I didn't check the code but if this optimisation went in, and the deletion of the cached object are now depending on the events, could it be that the stale pod metrics persist if a pod delete event is missed by the ksm while it's trying to reconnecting to api-server restart right after an api-server disconnect for whatever reason. |
It always depended on delete's to remove objects from cache before and after the optimizations. What was optimized was the cache itself. Is anyone still experiencing this with the latest patch release of kube-state-metrics 1.9 and a Kubernetes 1.18 cluster? |
Oh, so you mean ksm has always been edge triggered only, as in level-vs-edge-triggered, not level. I can't comment on your question as we are not yet on Kubernetes 1.18. But here is a scenario that can likely happen for AKS case:
|
@bergerx I tried to reproduce on my cluster and asked our Red Hat OpenShift QE to try to reproduce but we have not had any luck so far. We also did not get any reports from our customers so far that I know of for this bug. Can you possibly provide with exact steps, this would make it easier to fix it. thanks! Note to anyone in this issue, you are welcome to also give it a try in fixing this, PRs are always welcome, we can backport the fix to 1.9 as well as into 2.0! |
I started to think this issue to be mainly happening on to hosted Kubernetes services like AKS and EKS. There are many here using it without any issue, and there is another set of users who are repeatedly experiencing issues. Also see this weird issue stops suddenly stopping for some set of users in a python implementation. Maybe the issue is lying in the way how the apiserver is hosted behind a managed load balancer. @lilic I belive you can reproduce this if you try this on an AKS cluster. Most users complaining here are using AKS/EKS. |
Actually lets do a quick poll among AKS/EKS users, can you vote this entry with:
|
Addresses a bug that causes a gap between `list` and `watch` when kube-state-metrics is sharded (fix for kubernetes#694) Kube-state-metrics does a `list` and then enters a `watch` loop. The intention is to `watch` **all** events after the initial list. The k8s API takes an optional `resource version` parameter which is returned as part of the `list` call and can be forwarded to the `watch` call, in order to fetch all events after the initial `list`. In its sharded version, kube-state-metrics intercepts the returned `list` in order to filter out the events for other shards. It reconstructs the response, but it does not propagate the `resource version` to the modified response. The subsequent `watch` call does not refer to a resource version. When `watch` is called without a `resource version`, it will provide a view consistent with the **most recent** resource version of the `watch` call, missing the events between the `resource version` at `list` call and the most recent one. The k8s documentation captures this as follows: _Get State and Start at Most Recent: Start a watch at the most recent resource version, which must be consistent (i.e. served from etcd via a quorum read). To establish initial state, the watch begins with synthetic "Added" events of all resources instances that exist at the starting resource version. All following watch events are for all changes that occurred after the resource version the watch started at._ Testing: Reproduced the original bug report deterministically by introducing an artificial delay (120s) in list, prior to returning the response, and terminating some pods. Unless the bug is fix, the terminated pods continue to be reported as running by kube-state-metrics
Addresses a bug that causes a gap between `list` and `watch` when kube-state-metrics is sharded (fix for kubernetes#694) Kube-state-metrics does a `list` and then enters a `watch` loop. The intention is to `watch` **all** events after the initial list. The k8s API takes an optional `resource version` parameter which is returned as part of the `list` call and can be forwarded to the `watch` call, in order to fetch all events after the initial `list`. In its sharded version, kube-state-metrics intercepts the returned `list` in order to filter out the events for other shards. It reconstructs the response, but it does not propagate the `resource version` to the modified response. The subsequent `watch` call does not refer to a resource version. When `watch` is called without a `resource version`, it will provide a view consistent with the **most recent** resource version of the `watch` call, missing the events between the `resource version` at `list` call and the most recent one. The k8s documentation captures this as follows: _Get State and Start at Most Recent: Start a watch at the most recent resource version, which must be consistent (i.e. served from etcd via a quorum read). To establish initial state, the watch begins with synthetic "Added" events of all resources instances that exist at the starting resource version. All following watch events are for all changes that occurred after the resource version the watch started at._ Testing: Reproduced the original bug report deterministically by introducing an artificial delay (120s) in list, prior to returning the response, and terminating some pods. Unless the bug is fix, the terminated pods continue to be reported as running by kube-state-metrics (cherry picked from commit c1842eb)
Hello! I am not sure, whether this issue has the similar root casue with the one that we are now facing ( please see #1467 ). We also have AKS, and we have multiple pods (jobs) that run for a short period of time (a couple of seconds for some of them) and then terminate (with reason Completed), and when we make a query with Maybe our issue is also somehow connected to caching? But we don't see any warnings, or timeout, or error in the kube-state-metrics pod's logs. |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
/remove-lifecycle rotten |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close |
@k8s-triage-robot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
I presume that this helps kubernetes/kubernetes#59848 (comment). See also #1499 (comment) |
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
We wanted to check the metrics for some pods and realised we don't have metrics for the pod in question, then check the deployment and other resources, no metrics are there, but some other pod metrics were available.
We checked the metrics exposed from the /metrics endpoint and we saw metrics exposed for pods which are deleted 2+ weeks ago.
So kube-state-metrics were reporting stale metrics for last 3 weeks.
Here are some relevant logs from kube-state-metrics pod:
What you expected to happen:
kube-state-metrics should not be serving stale metrics more than a certain time (I'd say at most couple of minutes for our case).
Seems like the kube-state-metrics didn't try to reconnect to the kube-apiserver after having some issues with the kube-apiserver.
Instead of keep serving stale metrics kube-state-metrics should at least panic out our report unhealthy status.
How to reproduce it (as minimally and precisely as possible):
We didn't reproduce it yet. I'll update if we are able to reproduce.
Anything else we need to know?:
Here is a summary of the deployment status:
Environment:
kubectl version
):v1.11.5
-->Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"753b2dbc622f5cc417845f0ff8a77f539a4213ea", GitTreeState:"clean", BuildDate:"2018-11-26T14:31:35Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
quay.io/coreos/kube-state-metrics:v1.4.0
The text was updated successfully, but these errors were encountered: