-
Notifications
You must be signed in to change notification settings - Fork 537
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pods are getting stuck in ContainerCreating because of EFS CSI driver error #325
Comments
Very similar behavior here one EKS 1.18
|
Is there any relation between this issue happening and recency of fs/access point creation? Or did the same fs already exist 2 weeks before hand? Did this only start happening with |
It has been the same FS the entire time. We are using 1.0.0. We manage our nodes and routinely update the AMIs with a pipeline that handles the |
I also noticed the default logging level seems to have no helpful logs for this. Are we possibly missing a system log that might show more of what is going on. |
I just upgraded the CSI to the latest version (1.1.0) using the default settings in the values.yaml for the helm install. The daemonsets each restarted cleanly. The PV and PVC still show I attempted to restart the pods and they still came up with a It did not reattach until I issued the following:
and then delete/apply the pvc again.
The PVC hangs in a |
I was able to patch the PVC with the following, then delete it:
but after reapplying the pvc the volumes are not attaching.
shows @wongma7 do you have any suggestions? We are unable to use the volume at this point from the cluster. This is the recommended tool from AWS for attaching EFS volumes in EKS but so far it has been very unreliable and the logs show nothing new after setting the logging level higher. |
EFS shouldn't involve an attachment stage, there should be no volumeattachment object, and looking back at the error, I am not sure why there is a FailedAttach object. Can you share the efs CSIDriver object? Could you try this log collector tool? https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/troubleshooting/log_collector.py There should be logs from efs-utils (the app that provides |
The default On my part, let me prioritize getting the fix out into the helm chart |
After the
|
:\ |
@wongma7 I am afraid when the helm upgrade removed the CSIDriver it left my cluster in an unknown state. I still see the
The nodes mentioned are the one the scheduler is trying to run the pods on that mount the disk. I am guessing this is creating a conflict? |
Yes, it is safe to delete them. Now that CSIDriver has been fixed to say attachRequired false, kubelet should not even be attempting to get these volumeattachments |
@wongma7 after removing the volumeattachments mentioned above I am still unable to mount the volumes. I removed the PVC and reset the PV |
Can you share the latest events on the pod? |
Also, I don't see anything useful in the log_collector.py output. Its as though the driver does not see the pv/pvc? |
The driver should at least be logging "NodePublish" requests if it's receiving them. These would be visible from efs-plugin logs like kubectl log efs-node-pod-asdf efs-plugin. If the driver is really not logging anything, then we'll have to look at kubelet logs to ssee why kubelet is not sending requests to the driver. |
|
This log output is the same on all the nodes. |
Could you send me kubelet logs? We can continue on slack to share files if needed. |
No it didn't. Why?
I don't know. I started using EFS CSI very recently with version 1.1.0 Currently we're not using EFS, but I'd like to help so if more info required I'll re-deploy it again, unfortunately the issue happens intermittently, as @2tim said:
|
Apparently there is some undocumented EFS-side "bootstrap" or "cooldown" soon after it is created during which mounts may not succeed. @2tim 's issue we determined was because the csidriver got deleted by helm chart 1.1.0/1.1.1.#338 This meant attachRequired became true, which is wrong, so all Pods got stuck waiting for attach to complete. The solution is to recreate the csidriver with attachRequired false AND (restart kubelet OR reschedule the stuck pods to other kubelets). If your pods' events also say something like "FaileAttach" then it is the same issue. |
E.g.
Roughly, the cluster started 10 minutes later the last mount target was created so the EFS CSI resources were created after that point in time... Do you happen to know for how long we should wait before attempting to use EFS for the first time? |
I am having the same issue with EKS 1.18 |
@khurram-s It doesn't seem so. At least you have the exact error message:
Try to remove it from your manifest |
I'm getting the same timeout and "unrecognized init system" errors (attached) |
issue related to access point creation making an instance unable to mount efs
|
@mikalai-t |
@kbasv not quite:
As you can see there is min. ~10 minutes delay between that point when I can't control Correct me if I'm wrong but even if application tries to use |
Alright, I have some new repro steps to produce the issue, and some log_collector.py dumps from the plugin on the node where it happened. My test setup:
Repro steps:
I went through several successful iterations of these steps until our cluster started hitting ImgPullFailures from our internal Harbor registry from the number of pull requests we were causing. It's possible that pods sitting in an image pull backoff or another failed state is related to causing the issue to appear.
Finally, the log_collector.py results are attached below, run against the efs-node pod on the same node as the pod message above. |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
/remove-lifecycle stale |
+1, getting same issue as above on 1.21 |
Have the same issue on 1.21 and latest helm chart of this driver |
@wmgroot Have the same issue with EKS 1.20 and aws efs csi driver 1.3.3. |
Updates from the last month
Both of these changes combined significantly reduced the number of EFS errors in our deployment. I suspect the issues in our case were a combination of high load average on nodes combined with high load on EFS (I can't be sure because I have no visibility into EFS infrastructure) |
@wongma7 Just to confirm this issue is still seen in EKS 1.20 and aws efs csi driver 1.3.3. |
|
we are seeing the same issue, this issue is intermittent and once a node "goes bad", everything starts to collapse and no more pod can be scheduled to the node. |
The Helm chart does not specify default resources. Our findings suggest that a high load average on worker nodes may cause efs csi driver CPU resource starvation. We have a case where we can replicate this behaviour every time. Setting resource requests for containers seems to solve the problem for us, e.g.:
I'm not entirely sure if |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
Hi Khurram, I had same issue "storage.kubernetes.io/csiProvisionerIdentity not supported " what is the resolution for this. can you please help me |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close |
@k8s-triage-robot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Seeing the same issue on the latest version of CSI driver |
/reopen |
@V0lantis: You can't reopen an issue/PR unless you authored it or you are a collaborator. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/kind bug
Hello.
What happened?
Sometimes, on a new EC2 instance EFS CSI driver can't handle PVC from Pods running on that node and this cause an issue described in subject.
What you expected to happen?
PVC must be executed correctly so Volume can be mounted and attached to the Pod.
How to reproduce it (as minimally and precisely as possible)?
It happens intermittently. Generally the process is following:
Create
EKS
cluster andNode group
(doesn't matter what EC2 instance type to choose - it happened ont3.medium
andm5.large
).Create
EFS
andSecurityGroup
for EFS to allow connection to2049/tcp
fromVPC CIDR
. Say, the EFS ID isfs-33559c06
.Create
EFS AccessPoint
for particular application (e.g. IDfsap-085397e3f5ef75ead
)Create
StorageClass
:Pod
:Anything else we need to know?:
cs-driver-registrar
container (by the moment of writing I lost the exact timestamp):After restarting EFS CSI Pod the error above disappeared, but it didn't help to resolve the issue - the only workaround is to
taint/remove
the node fro the cluster and create a new EC2 instance.Meanwhile logs from the
efs-plugin
before and after restart are not very helpful. It keeps repeating the same sequence every4 minutes 2 seconds
:kubectl get events -n monitoring --sort-by=metadata.creationTimestamp
):Environment
kubectl version
):Installed using Helm chart
aws-efs-csi-driver
version1.1.0
from the repohttps://kubernetes-sigs.github.io/aws-efs-csi-driver
with default values.The text was updated successfully, but these errors were encountered: