-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kubelet does not export metric kubelet_volume_stats_capacity_bytes
#3643
Comments
metric kubelet_volume_stats_capacity_bytes
kubelet_volume_stats_capacity_bytes
This is probably due to #3360 |
Thanks for the updates, can you please detail how? |
A featuregate is enabled currently to disable local storage isolation, because enabling it broke kind on some hosts. There's more discussion in the linked PR and transitively linked issues. You can try this config to see if that PR would solve it: |
Thank you again for the updates. As far as I understand that option was disabled in kind because it broke kubernetes in some scenarios, but I still don't understand the connection with the missing metric because as far as I can see the option is about limiting the ephemeral storage in LimitRange and Pod specs. Am I missing anything? |
I've just tried with this config kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: KubeletConfiguration
localStorageCapacityIsolation: true
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "ingress-ready=true"
- |
kind: ClusterConfiguration
controllerManager:
extraArgs:
bind-address: 0.0.0.0
etcd:
local:
extraArgs:
listen-metrics-urls: http://0.0.0.0:2381
scheduler:
extraArgs:
bind-address: 0.0.0.0
- |
kind: KubeProxyConfiguration
metricsBindAddress: 0.0.0.0
extraMounts:
- containerPath: /var/lib/kubelet/config.json
hostPath: "$HOME/.docker/config.json"
extraPortMappings:
- containerPort: 443
hostPort: 443
- containerPort: 80
hostPort: 80
- role: worker
- role: worker
- role: worker still no metric available! |
kind is shipping kubelet from upstream sources, I was guessing that this metric is gated behind the same code in kubelet because it's related to tracking filesystem stats. We're not modifying kubelet. That config should be equivilant enough because kubeadm currently cluster-scopes kubelet config, but preferably it should be explicitly cluster scoped in case kind fixes this in the future (so kubeadmConfigPatches at the top level not under one of the nodes) It would also be helpful to test this with a minimal configuration for reproducing purposes. Note: I just looked up this metric and it's considered alpha, and related to PVs. https://kubernetes.io/docs/reference/instrumentation/metrics/ The other thought is this may not work with PVs from https://github.com/rancher/local-path-provisioner What's your use case? |
Here you have the reproducer, just run the script and port-forward Grafana in monitoring namespace to be able to see that the metric is missing. Using this reproducer I launched a kube-proxy and then issued this curl on a worker node: to verify that the |
What happened:
What you expected to happen:
Kubelet should export metric
kubelet_volume_stats_capacity_bytes
but it looks like it doesn't.I deployed kind with the following configuration:
I checked with Prometheus and I see many other metrics exported by kubelet but not this, e.g.
![immagine](https://private-user-images.githubusercontent.com/15982257/336132515-ef236a40-5f41-409a-8c33-15efc1645994.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ2MDEzMTQsIm5iZiI6MTcyNDYwMTAxNCwicGF0aCI6Ii8xNTk4MjI1Ny8zMzYxMzI1MTUtZWYyMzZhNDAtNWY0MS00MDlhLThjMzMtMTVlZmMxNjQ1OTk0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI1VDE1NTAxNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTI4MTE3ZTU0YWUyMDI4ZjhkZjgzNGZiZGFlNjA2Yjc1ZjIxOTNiMTI0NjRmNjI1MDZmMDg2MzBjNzFkY2M0ZDQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.P7rZoui4muxkX0zAp5liOrJtlzEyv8QCT8AH5XaPGMA)
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
kind version
): kind v0.23.0 go1.21.10 linux/amd64docker info
,podman info
ornerdctl info
):OS (e.g. from
/etc/os-release
):PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
Kubernetes version: (use
kubectl version
):Client Version: v1.29.5
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0
Any proxies or other special environment settings?:
The text was updated successfully, but these errors were encountered: