-
Notifications
You must be signed in to change notification settings - Fork 81
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CSI in-line volume setup intermittently fails with config error in azstorage [account name not provided]
#1340
Comments
can you use |
@andyzhangx with
|
that's actually the same, you need to specify account name in secret:
and then specify
|
MSI does not need a secret. Note that it is intermittent, not consistently failing. Out of 200 pods, about 10 suffer from this problem. After a few backoffs they mostly succeed. If secret is required it would have failed consistently. |
no, you only need to specify azurestorageaccountname in the secret, that's the way pod inline volume to get the account name, that's for the sake of security. |
That seems to have fixed it. Few clarifying questions:
Thanks for your help. |
This problem is still happening with Kubernetes secrets but at a much lower rate. About once every 200 times. Code is still racy. |
what's current error msg? |
Same: |
@technicianted pls follow this guide to provide csi driver logs on the node: https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/csi-debug.md#case2-volume-mountunmount-failed, and what's current pod config? |
Pod volume config in original post.
|
@technicianted could you provide the csi driver logs on the node:
|
Sorry we moved away from this driver due to instability. Can't provide any more logs. |
Option#3: Inline volume
kubectl create secret generic azure-secret --from-literal azurestorageaccountname=NAME --from-literal azurestorageaccountkey="KEY" --type=Opaque
wget https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/nginx-blobfuse-inline-volume.yaml
#edit nginx-blobfuse-inline-volume.yaml
kubectl create -f nginx-blobfuse-inline-volume.yaml |
Thanks. But I think "performance issue" is not the correct wording. I would use call it unstable. Also, using a secret did not fully resolve the problem, just reduced it significantly but it still happened. I would use stronger wording to completely avoid in-line volumes, or disable it completely. |
What happened:
When starting a pod with in-line CSI volume, it intermittently fails to mount with the error:
After a few kubelet backoffs (no spec changes), it succeeds.
What you expected to happen:
Volume setup should succeed the first time.
How to reproduce it:
Create a deployment with large number of pods to increase the change of happening - perhaps 20+. Use something similar to this:
Anything else we need to know?:
This is highly intermittent. For a large number of pods in a deployment, most of them succeed the first time. Others can take a few retries.
Environment:
kubectl version
): v1.26.12uname -a
): 6.2.0-1019-azureThe text was updated successfully, but these errors were encountered: