Awx-operator Helm upgrade fails due to version labels in selector

[andriktr]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

Hi,
If we try to run helm upgrade for awx-operator for example from 0.28.0 to 0.29.0 version update is failing with the following error:

Error: UPGRADE FAILED: release awx-operator failed, and has been rolled back due to atomic being set: cannot patch "awx-operator-controller-manager" with kind Deployment: Deployment.apps "awx-operator-controller-manager" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"control-plane":"controller-manager", "helm.sh/chart":"awx-operator-0.29.0"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

The reason for this is a version label used in deployment selector:

spec:
  replicas: 1
  selector:
    matchLabels:
      control-plane: controller-manager
      helm.sh/chart: awx-operator-0.29.0
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/default-container: manager
      labels:
        control-plane: controller-manager
        helm.sh/chart: awx-operator-0.29.0

Deployment selector labels are immutable and can't be changed during upgrade https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#label-selector-updates
In our case this selector changes from one version to another

AWX Operator version

0.28.0

AWX version

21.5.0

Kubernetes platform

kubernetes

Kubernetes/Platform version

1.23.5

Modifications

no

Steps to reproduce

run helm upgrade for awx-operator

Expected results

deployment upgraded

Actual results

upgrade fails with

Error: UPGRADE FAILED: release awx-operator failed, and has been rolled back due to atomic being set: cannot patch "awx-operator-controller-manager" with kind Deployment: Deployment.apps "awx-operator-controller-manager" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"control-plane":"controller-manager", "helm.sh/chart":"awx-operator-0.29.0"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Additional information

No response

Operator Logs

No response

[jeffb4]

I ran into this problem as well. Not that it should be required, but you can kubectl -n awx delete deployment awx-operator-controller-manager and continue with the Helm chart upgrade.

[andriktr]

@jeffb4 Yea sure we can remove existing awx-operator deployment and run helm upgrade, but this is not how it should be :)
Selector label should not contain helm.sh/chart: awx-operator-0.29.0 or any other label which may change from version to version. This especially important if you have automated deployment via pipelines.

[FlorianLaunay]

@andriktr thanks for this issue!

Yeah, very frustrating problem 😓

[trippinnik]

Same 0.29.0 to 0.30.0

[andriktr]

The only way I see how to fix it is to remove helm.sh/chart: awx-operator-x.xx.x selector and point it as a breaking change in the release.
Currently as workaround I just changing a version number in the selector to the same which was set during the first awx-operator helm installation.

[FlorianLaunay]

I have started to work on a local branch to fix this. I will keep you in touch and open a PR when the fix will be ready.

[FlorianLaunay]

After investigations, this problem seems to be caused by kustomize during the helm chart generation. This one is managed in Makefile by the helm-chart-generate step (Makefile#L295) using kustomize edit set label to add commonLabels field to kustomization.yaml containing the helm.sh/chart label with the operator version.

Unfortunately kustomize use commonLabels arg to set the metadata.labels field but also spec.selector.matchLabels and the community seams to disagree with this behavior : kubernetes-sigs/kustomize#1009 😅 . Indeed this field is now immutable for deployments in recent versions of Kubernetes (https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#label-selector-updates). So running a helm upgrade with a chart containing a deployment template with a different content in the spec.selector.matchLabels field is not allowed and cause this error.

The only solution I see is to use the labels fiels in kustomization.yaml instead of commonLabels (kubernetes-sigs/kustomize#3743). But unfortunately again, kustomize edit set label don't manage this new field 😢. But it seems well-documented (kubernetes-sigs/kustomize#3756) (https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/labels/) 🤩 .

Using this new field means stop using kustomize edit set label and add labels field in kustomization.yaml with yq. I need to modify the Makefile to do that and run some test builds before submitting a PR.

[FlorianLaunay]

@andriktr I have a working fix with #1114

Waiting to be reviewed 😇

[andriktr]

@FlorianLaunay sounds great.
Thanks for your effort ;)

[FlorianLaunay]

@andriktr I have a working fix with #1114

Waiting to be reviewed 😇

Got an interesting review from the AWX team. Another solution is now being considered. Awaiting for their response to push 😉