Preview local changes with flux build/diff kustomization

[souleb]

Signed-off-by: Soule BA soule@weave.works

fixes #2142

Allow users to preview changes to their Kustomize overlays without committing changes to upstream

• flux build kustomization my-app --path ./path/to/local/manifests. This can be tested by pulling this pr.
• flux diff kustomization my-app --path ./path/to/local/manifests

The diff output is formatted with https://github.com/homeport/dyff

Tests

build kustomization

With the following tree:

⋊> ~/f/p/kustomize on master ⨯ pwd                                                                                                                                                                  11:29:56
/Users/souleb/flux/podinfo/kustomize
⋊> ~/f/p/kustomize on master ⨯ tree                                                                                                                                                                 11:29:57
.
├── deployment.yaml
├── hpa.yaml
├── kustomization.yaml
├── secret.yaml
├── service.yaml
└── token.encrypted

0 directories, 6 files

⋊> ~/f/p/kustomize on master ⨯ cat kustomization.yaml                                                                                                                                               11:30:01
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- hpa.yaml
- deployment.yaml
- service.yaml
- secret.yaml
secretGenerator:
- files:
  - token=token.encrypted
  name: podinfo-token
- literals:
  - username=admin
  - password=1f2d1e2e67df
  name: db-user-pass

# secret used to test a sops encrypted secret diff
⋊> ~/f/p/kustomize on master ⨯ cat secret.yaml                                                                                                                                                      11:32:23
apiVersion: v1
data:
  token2: KioqU09QUyoqKgo=
kind: Secret
metadata:
  name: podinfo-token-4c8c578kkk
type: Opaque
⋊> ~/f/p/kustomize on master ⨯ echo "KioqU09QUyoqKgo=" |base64 -d                                                                                                                                   11:32:26
***SOPS***

and a kustomization:

⋊> ~/flux cat fleet-infra/clusters/my-cluster/podinfo-kustomization.yaml                                                                                                                            11:30:41
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 4m0s
  path: ./kustomize
  prune: true
  sourceRef:
    kind: GitRepository
    name: podinfo
  validation: client
  decryption:
    provider: sops
    secretRef:
      name: sops-age
  targetNamespace: flux-system
  patches:
    - patch: |-
        apiVersion: autoscaling/v2beta2
        kind: HorizontalPodAutoscaler
        metadata:
          name: podinfo
        spec:
          minReplicas: 2
      target:
        name: podinfo
        kind: HorizontalPodAutoscaler

The build command renders:

⋊> ~/flux /Users/souleb/go/src/github.com/fluxcd/flux2/bin/flux build kustomization podinfo --path ./podinfo/kustomize                                                                              12:03:23
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  labels:
    kustomize.toolkit.fluxcd.io/name: podinfo
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: podinfo
  namespace: flux-system
spec:
  maxReplicas: 4
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 99
        type: Utilization
    type: Resource
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    kustomize.toolkit.fluxcd.io/name: podinfo
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: podinfo
  namespace: flux-system
spec:
  minReadySeconds: 3
  progressDeadlineSeconds: 60
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app: podinfo
  strategy:
    rollingUpdate:
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/port: "9797"
        prometheus.io/scrape: "true"
      labels:
        app: podinfo
    spec:
      containers:
      - command:
        - ./podinfo
        - --port=9898
        - --port-metrics=9797
        - --grpc-port=9999
        - --grpc-service-name=podinfo
        - --level=info
        - --random-delay=false
        - --random-error=false
        env:
        - name: PODINFO_UI_COLOR
          value: '#34577c'
        image: ghcr.io/stefanprodan/podinfo:6.0.3
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - podcli
            - check
            - http
            - localhost:9898/healthz
          initialDelaySeconds: 5
          timeoutSeconds: 5
        name: podinfod
        ports:
        - containerPort: 9898
          name: http
          protocol: TCP
        - containerPort: 9797
          name: http-metrics
          protocol: TCP
        - containerPort: 9999
          name: grpc
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - podcli
            - check
            - http
            - localhost:9898/readyz
          initialDelaySeconds: 5
          timeoutSeconds: 5
        resources:
          limits:
            cpu: 2000m
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 64Mi
---
apiVersion: v1
kind: Service
metadata:
  labels:
    kustomize.toolkit.fluxcd.io/name: podinfo
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: podinfo
  namespace: flux-system
spec:
  ports:
  - name: http
    port: 9899
    protocol: TCP
    targetPort: http
  - name: grpc
    port: 9999
    protocol: TCP
    targetPort: grpc
  selector:
    app: podinfo
  type: ClusterIP
---
apiVersion: v1
data:
  token2: KioqU09QUyoqKgo=
kind: Secret
metadata:
  labels:
    kustomize.toolkit.fluxcd.io/name: podinfo
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: podinfo-token-4c8c578kkk
  namespace: flux-system
type: Opaque
---
apiVersion: v1
data:
  token: KipTT1BTKio=
kind: Secret
metadata:
  labels:
    kustomize.toolkit.fluxcd.io/name: podinfo
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: podinfo-token-4c8c578kkk
  namespace: flux-system
type: Opaque
---
apiVersion: v1
data:
  password: MWYyZDFlMmU2N2Rm
  username: YWRtaW4=
kind: Secret
metadata:
  labels:
    kustomize.toolkit.fluxcd.io/name: podinfo
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: db-user-pass-bkbd782d2c
  namespace: flux-system
type: Opaque

We sanitize the multi-doc yaml before printing, it. Every value of the data object in secrets are base64 decoded, and if it's sops data, we replace the whole value with a mask.

Test diff kustomization

[stefanprodan]

@souleb we should mask only the secret values encrypted with SOPS. I propose we change the mask to **SOPS**, the secrets that are in plain text in the repo they should be rendered as they are.

[souleb]

@stefanprodan my test diffing on plain kubernetes secrets always show changes in all values, when a secret is drifted. That's because the ssa.Diff code mask values with ****** and *****:
https://github.com/fluxcd/pkg/blob/cd50d1697c5e59b767b3debd52963941defddb3b/ssa/manager_diff.go#L54

Can we have different mask only if the underlying value is different?

[souleb]

This is no longer a draft.

It can be thoroughly tested by humans 😸
@stefanprodan @hiddeco @kingdonb @somtochiama @Nalum @phoban01

[stefanprodan]

@souleb can you please update the PR description with up to date examples, I don't see any labels in the build output, if that's the case then this is a bug, as build should add the ownership labels to all objects.

[souleb]

Yes going to update that.

@souleb can you please update the PR description with up to date examples, I don't see any labels in the build output, if that's the case then this is a bug, as build should add the ownership labels to all objects.

Is it? We set the ownership label just before the dry-run in the diff command. It never appears in the output.

flux2/internal/kustomization/diff.go

Line 61 in b6c42b6

resourceManager.SetOwnerLabels(objects, b.kustomization.GetName(), b.kustomization.GetNamespace())

[stefanprodan]

@souleb the build command should have the same output as in the controller.

[souleb]

@souleb the build command should have the same output as in the controller.

The labels now appear in the build.

[bjhaid]

Thanks for doing this work, I have similar interests and had done PoC, before finding this, I left 2 comments... My other concern is there a plan to ensure the logic here stays in sync with the Reconcile and finalize methods from the kustomize-controller...

[stefanprodan]

@souleb I found several bugs in the dry-run, it doesn't ignore objects with kustomize.toolkit.fluxcd.io/reconcile: disabled and it reports bogus HPA metrics. I'll fix these in fluxcd/pkg/ssa.

[stefanprodan]

@souleb please update ssa to v0.11.0 and add the kustomize.toolkit.fluxcd.io/reconcile: disabled to the DiffOptions.

[stefanprodan]

@souleb I've run the latest commit and works fine now on my test cluster. Tested this with all sorts of edits, I think we can release this as an "experimental" feature.

[stefanprodan]

LGTM

Thanks @souleb 🥇