Leverage finalizer for Package deletions

[mjnagel]

Describe what should be investigated or refactored

In the current state of things deletions happen in a watch in our pepr code. As we manage additional "external resources" (such as keycloak clients), blocking on that deletion is critical to prevent errors when a rapid delete + create process happens. Kubernetes finalizers feel like the right solution to ensure that someone cannot reuse a Package name before the previous one is fully cleaned up.

Links to any relevant code

This block of code is the most relevant - note that these are two separate flows and there is currently no "block on delete" in here:

uds-core/src/pepr/operator/index.ts

Lines 43 to 60 in d71d83e

	// Watch for changes to the UDSPackage CRD and cleanup the namespace mutations
	When(UDSPackage)
	.IsDeleted()
	.Watch(async pkg => {
	// Cleanup the namespace
	await cleanupNamespace(pkg);
	// Remove any SSO clients
	await purgeSSOClients(pkg, []);
	});
	// Watch for changes to the UDSPackage CRD to enqueue a package for processing
	When(UDSPackage)
	.IsCreatedOrUpdated()
	// Advanced CR validation
	.Validate(validator)
	// Enqueue the package for processing
	.Reconcile(packageReconciler);

Additional context

https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/

[mjnagel]

Linking Pepr issue/feature suggestion for native finalizer support - defenseunicorns/pepr#1022

[mjnagel]

May put this on hold to see what shakes out from the Pepr native support feature request.