-
Notifications
You must be signed in to change notification settings - Fork 904
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
karpenter scale up new node with EFS #5691
Comments
You should use a |
Hey, i tested using startupTaint, forgot to mention. with startupTaints: Is not working, but i think it's more ticket to EFS. if there is no way to spawn node before limit is reached. |
Was the pod being scheduled? Was the EFS driver removing the startupTaint? If there's any more details about this race you could give, hopefully we can definitively say if it's on Karpenter or the efs-csi-driver. |
"Was the EFS driver removing the startupTaint?" i assume yes, cuz it is set in nodeclaim, and right now node don't have any taints. kubectl get pods
describe
kubectl get nodes -o json | jq '.items[].spec.taints' kubectl get nodeclaim hxps-66ae7155-6992-40d7-a887-ae5a746c5c25-dev-mseries-btnkm -o yaml
|
If a node is being provisioned and the pod is being scheduled I'm going to wager the problem is on the EFS side. Without pod logs it's of course impossible to tell for sure, may be good to take up with the EFS folks. |
@jmdeal and it seems that efs/karpenter removed this taint to early.
|
Karpenter is not responsible for removing the taint, at that point it seems like it would be an aws-efs-cis-driver issue. I see you already opened an issue there, I'm going to close out this one for the time being. Feel free to reopen if something points it back to Karpenter. |
Description
Observed Behavior:
We have cluster with 1 NODE and full limit pods 58/58(but i believe same problem will be with more nodes)
We are installing additional pods with EFS, but pod is failing cuz of some race problem EFS on new nodes, related ticket- kubernetes-sigs/aws-efs-csi-driver#1069
This bug is already "workarounded" by EFS using taints for nodes link to EFS docs
https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/docs/README.md#configure-node-startup-taint
So i added to karpenter nodepool this taint.
And if using this taint.
karpenter will not scale up new node, cuz no pods is in state pending
Pod is waiting with failedscheduling
![image](https://private-user-images.githubusercontent.com/34127303/306072162-40c0479a-6fa3-422b-b951-fa1ade8ae10d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0NjcyMzQsIm5iZiI6MTcyNDQ2NjkzNCwicGF0aCI6Ii8zNDEyNzMwMy8zMDYwNzIxNjItNDBjMDQ3OWEtNmZhMy00MjJiLWI5NTEtZmExYWRlOGFlMTBkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI0VDAyMzUzNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTVkMGVmMDVjYmUzYjgwM2YwODQ4ODU2YjJlOWVjYjAxZmE4ZjQxMzU1YWI2ZTIzZTA0ZDBhMTM2NDQzMzRkZTkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.3Ad6pNjjp81psHZ_ElSpvKNCsIGJFVRBXnUgS4v926o)
Expected Behavior:
Some way to trigger creation of extra nodes, before max limit of number pods pods.
Or some other idea.
Reproduction Steps (Please include YAML):
Versions:
kubectl version
):The text was updated successfully, but these errors were encountered: