HNC: subnamespaces are deleted because we don't realize CRDs are being deleted

[adrianludwin]

The "should not delete subnamespace" test in delete_crd_tests.go is flaky and fails maybe 20% of the time. This is fairly important because this is our only line of defense when uninstalling HNC - when you delete the CRDs, the subnamespaces are not supposed to be deleted.

According to the logs, HNC simply fails to notice that the CRDs are being deleted. A subsequent reconcile about 0.1s later does notice that the CRDs are being deleted, but by then it's too late - we've already started deleting the subnamespace. In other words, there's a race condition.

I'm asking some apiserver experts if there's a better way to solve this.

[adrianludwin]

/assign @GinnyJI

Ginny, can you please look into this?

Per the controller-runtime folks, while the default client uses the cache, we can create our own controller-runtime client that doesn't use the cache. And per @liggitt , that's the best way we have to determine if a CR's being deleted due to its CRD being deleted.