-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Seeing flakey testing where default
namespace doesn't exist
#2626
Comments
/kind support |
Probably as envtest first deploys CRDs and then waits for them to be available. Which client call gives you this error? |
Yeah, this is what I was suspecting. There must be enough time in there that it's able to bring-up the default namespace.
It's a call to create a pod with no namespace specified. |
@jonathan-innis I'm fine with adding a "waitForDefaultNamespace" call here: https://github.com/kubernetes-sigs/controller-runtime/blob/21779fbe6419e03ae1cd86a69b35d2725c6f7558/pkg/envtest/server.go#L273C1-L273C1 |
Perfect, I'll try to put-up a PR to add this in shortly 🎉 Thanks for the help! |
@sbueringer Opened a PR with the wait mechanism. Frustratingly, I've been trying to repro this locally but no dice, so it's hard for me to tell if this is actually going to resolve our issue. I'd suspect that it will, but it's hard to say without being able to prove it. Let me know if you have any thoughts on ways I could force this race. |
Currently, we are running
envtest
in Karpenter where we are starting up all of the binaries and then applying pods to thedefault
namespace. Every now and then, we are seeing failures in our pod metric testing (seen here: https://github.com/kubernetes-sigs/karpenter/actions/runs/7251475446/job/19753909305?pr=885). When this happens, it always contains the same errorThis error indicates that the
default
namespace doesn't exist at the apiserver when we are applying the object; however, I would generally expect this namespace to exist once the apiserver starts up.It's worth noting that we don't see this error in other testing where we are running
envtest.Environment
with theCRDs
option set. I suspect that this is due to the fact that this gives the setup a little more time to populate this namespace at the apiserver before it returns.Is there a general recommendation here? Obviously we could wait for the
default
namespace to exist in our own code, but it would be nice if we could be assured that it would always exist when we start testing from upstream.The text was updated successfully, but these errors were encountered: