Reinstalled flux behaves as if it wasn't uninstalled

[philenzsky]

I created two plain kubectl manifest files, to install two deployments, in a folder in a repo watched by flux.
Unfortunately, I accidentally included declarations to create a namespace in each manifest file.
The namespace folder already contained many other deployments.

Firstly, the reconciliation of one of these new manifests failed, presumably as both were trying to create the same namespace. I didn't really understand the message from "flux get all -A".
Apologies I didn't record it either... something about a reference to a directory not a file.

Secondly, as expected, when I deleted both manifests from the watched folder, the namespace and all of its contents was removed.
I didn't think about that until everybody started asking where their applications had gone!!!

Now the bit I don't understand...
I cleaned things up by creating the namespace once more and uninstalling/installing flux again.
I used the same git repo for the install... using flux bootstrap gitlab and the same watched folder.
I then copied the two updated manifests (with no declaration to create namespace) to the folder for flux to synch.

But the result was that flux again deleted the namespace.
I was definitely not expecting that!!!

Any idea why?
Did flux not cleanly uninstall?
Should I have used a new gitlab repo so there was no historical versions of the manifests with the create manifest declaration?

[kingdonb]

Without a few more details, like the actual manifest that you used, I'm not sure anyone can tell why Flux deleted the namespace. But it does sound like when you repeated the same steps, you got the same outcome. So that at least suggests the possibility of holding it wrong. There could also be a guard rail missing, or other UX enhancement possible that would make this mode of catastrophic failure less likely. I really cannot tell you without the manifest from your repo.

It sounds like you are not bootstrapping flux the normal way. When you say you create the namespace and apply the manifest again, how does this differ from bootstrapping exactly and why are you doing it this way, instead of running bootstrap according to the instructions?

Please provide more details about any specific guides you followed, and include as many details as you can about the reasons why you may have needed to diverge from the guide as it is written. These issues may complicate support but without more information, I cannot be of any help except to balk and say "sounds like you got a repro there, at least huh huh"

I'm sorry, I don't mean to make light of a serious issue, (but please don't go and do this a third time just to make a point!) we can certainly help you piece together exactly what you did to make sure others know not to do that.

It may be expected behavior. Running flux uninstall is expected to delete all the custom resources and the flux-system namespace. If you have anything installed in there, that may come as a surprise, so you should always test these things on a staging instance before running especially destructive commands in production.

If you delete the flux-system namespace before running flux uninstall, for example, that can easily result in all hell breaking loose. That is not the recommended process for uninstalling flux in any guide we host, though, so hopefully you didn't get that specific advice from anywhere in the flux docs! (Let us know...)

[philenzsky]

Thanks for getting back to me so promptly. Apologies for the vagueness. Hopefully this step-by-step list is more useful:

• I installed flux in a test environment using this guide from Gitlab: https://docs.gitlab.com/ee/user/clusters/agent/gitops/flux_tutorial.html

• I needed to install in a directory other than flux-system though: flux bootstrap gitlab --owner=mycompany/mygroup/ci --repository=flux-dev --branch=main --path=clusters/company-dev --deploy-token-auth -n ops-devtest

• I then used git to add/commit/push two manifest files to the repository flux-dev, main branch, clusters/company-dev path.

• The manifests contained deployments that needed to be deployed to another namespace apps-devtest.

• The apps-devtest already existed and contained several deployments already.

• A sample of one of the manifests, except not deploying nginx in reality...

•     apiVersion: v1
      kind: Namespace
      metadata:
         name: apps-devtest
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
          name: nginx-deployment
          namespace: apps-devtest
          labels:
             app: nginx
          spec:
             etc...
  

• The second deployment similar, with the same namespace kind at the top... which I really didn't want in there!

• Once flux went to reconcile things, it installed one of the manifests, and then errored on the second one.

• I realised the manifests should not have had the namespace kind section, so used git to rm/commit/push to get rid of the manifests.

• I didn't think that through, as, of course, flux reconciled and attempted to remove the apps-devtest namespace and all of its contents. So, my mistake in the first place.

• I then decided to uninstall flux using flux uninstall --namespace=ops-devtest --keep-namespace=true

• And I also need to remove finalizers from the apps-devtest namespace as it had been left in a terminating state, and then removed that and create a new apps-devtest namespace.

• I then used git to remove the flux kustomization files from my flux-dev git repository (under ops-devtest folder).

• I then installed flux again, using exactly the same command as the first time.

• I then used git to add/commit/push new versions of my two deployment manifests - with no namespace kind definitions at the top to the flux-dev repository, main branch, clusters/company-dev path.

• When flux attempted to reconcile, I got the unexpected result: flux again decided to delete the namespace and its contents and seemed to get stuck trying to reconcile.

• So, it is as if flux somehow used information from the old versions of the manifests with the namespace kind section still in existence.

• I've uninstalled flux again and recreated the apps-devtest namespace again (as lots of other people need to deploy things [using kubectl] to that namespace).

• I was going to try two more things from here...

1. Install flux using a new repository than the flux-dev one I've used so far. Perhaps somehow flux is looking at an old commit of the manifest files containing the namespace section.
2. Instead of committing the manifest files into the common flux path clusters/company-dev, create flux GitRepository source manifests and Kustomization manifests in there pointing to each of the repositories and kustomize folders from which the manifest files were created. The only reason for not using this approach so far was to keep things as simple as possible for development teams seeing flux and gitops for the first time.

• But before doing either of those things, I wanted to see if you had any advice on best way to proceed.
• Whatever I do, I'll ensure I backup the apps-devtest namespace first.

[kingdonb]

I don't see exactly what went wrong in your described steps. One other detail that might help is your settings for timeout and interval.

Did you make any adjustments to the default settings? How many Kustomization resources are used, and do any of them overlap?

It could be something about your layout, if you have been able to reproduce the issue with a test deployment perhaps you can also publish a repo with the minimum viable repro where we can follow your steps to reproduce the issue?

I know I said more details are great, and this is great, but it will also be very helpful if you can include in one published public repo the specific manifests that actually were in the failure scenarios. I'm looking for details to find out if prune (garbage collection) was enabled, what resources got their garbage collectors triggered and how, if not only the apps ns did.

If you're sure you know how it happened and it won't happen again, we can close this issue any time now. But if there is a specific UX enhancement we can suggest so this won't happen, or a strategically placed note in the docs you followed warning people not to make the same mistake, somehow concisely enough to make sense to a reader, we will definitely make the change. I will be glad to follow this up later. Thanks for the further reports.

[kingdonb]

The timeout and interval settings are important because an incomplete apply can get rolled back when the health check fails. This should only happen when timeout is less than interval. It would only have an impact if some resources were applied successfully and other resources failed their health check but the whole Kustomization could not be applied together.

This would result in the whole change being reverted, in the server side apply step (after some deployment may already have came online and is serving traffic) and if namespaces are mixed with other resources, it could result in the surprise deletion of a namespace and all the contents. The defense against this is keeping your manifest bundles small and declaring the namespaces in a separate Kustomization, perhaps even with prune disabled. I'm not sure how to make this advice concise or where to place it for the necessary impact. This also may not be exactly what our readers (or you) needed to hear. Thanks again, do let us know please.

Other factors like dependency order and paths overlapping could be further confounding. The published repo with repro steps will help us know more about it, if you can provide one.

[philenzsky]

Thanks very much for the help and advice.
I'm happy for this one to be closed.
More learning on our end!

I don't believe this had anything to do with Flux. More to do with the fact that I needed to force delete the namespace (due to external pressure to get everything running again asap in a namespace with the same name!) which is never a good thing to do in Kubernetes. 

I think your summary is great advice... "... and if namespaces are mixed with other resources, it could result in the surprise deletion of a namespace and all the contents. The defense against this is keeping your manifest bundles small and declaring the namespaces in a separate Kustomization, perhaps even with prune disabled.

We will definitely be moving to individual kustomizations, and probably disabling prune to start with!

Thanks so much.