-
Notifications
You must be signed in to change notification settings - Fork 203
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SGX-enabled pods sometimes get created without SGX device mounted #1695
Comments
@Feelas can you check the pod's resources when the error happens? I believe you only get
and not
which then suggests the operator has failed to mutate the pod. you could check the API server logs for more details why the operator/webhook had failed |
Not sure if it's related, but we've seen somewhat similar with e2e fpga. Seems like a Pod is scheduled with the generic resource name (Pod goes to Pending) and when the operator comes fully online it doesn't detect the Pending pod. Pod is left in Pending state. |
@mythi we've managed to trigger the error again. As a result:
Given my limited knowlede of webhooks:
If yes, it looks like the webhook might not handle "retroactively" mutating Pods w/ SGX requests that were created on the cluster before the webhook itself is created and registered, unless of course I am missing something about how the webhooks work. |
I'd like to understand why the mutation gets missed on CREATE because that's what starts CrashLoopBackOff. |
Same, the only idea that would make sense would be if Retries for spawning the Pod wouldn't throw a CREATE. So the timeline would look like this, in my mind: And by this point it would boil down to the question "when can we be sure that the CREATE hook is already up"? |
Describe the issue
Not entirely sure if that is a bug in IDP, perhaps it just needs some explanation on how to properly ensure that sgx.intel.com/epc is available already or an upstream K8s/kubelet issue.
We are seeing a randomly occuring situation where Pods with "requests: sgx.intel.com/epc: 1Gi" defined in the Pod Spec get created without the necessary /dev/sgx_enclave volume mount. This causes the workload to fail due to lack of the SGX device (ENOENT error in Gramine).
My (intuitive) understanding is that until "sgx.intel.com/epc" is registered on the node, the Pod should not get created (due to being unschedulable for insufficient resources)?
If that intuition is wrong, what is the correct way to make sure that the Pod creation waits for "sgx.intel.com/epc" availability? Since this process is done as part of an automated flow (ArgoCD), we are not currently fully sure in what order the resources will get created.
To Reproduce
No clear reproduction scenario, happens randomly. We did not find a correlation with specific OCP version.
Expected behavior
Pod is created with the necessary volume mounts with /dev/sgx_enclave each time.
System:
Additional context
It might be an edge-case similar to the issue noticed for nVidia GPU (this specific comment): NVIDIA/k8s-device-plugin#291 (comment).
Similar as in the linked issue, the Pod somehow gets started before sgx.intel.com/epc can be succesfully allocated and mounted. Very curious why that happens.
The text was updated successfully, but these errors were encountered: