Support fro git+ssh resources in kustomization.yaml

[vladimirfx]

Please add ssh package to kustomize-controller to provide fully functional git support in kustomization.yaml. We build our own images but it can be useful for other users.

[stefanprodan]

Git is installed https://github.com/fluxcd/kustomize-controller/blob/main/Dockerfile#L39

[vladimirfx]

For urls such ssh:git@somehost.com/repo.git ssh client is required. It is used by git to establish secure connection and (what is more important) to authorize user.
So we just 'inject' keys in kustomize-controller pod to support private repositories in transparent way. Usually same secret that source-controller is using.

Can I provide a PR?

[stefanprodan]

Yes please open a PR. Thanks

[vladimirfx]

Yes please open a PR. Thanks

Thank YOU for such amazing tool!

[LLlnak]

For urls such ssh:git@somehost.com/repo.git ssh client is required. It is used by git to establish secure connection and (what is more important) to authorize user.
So we just 'inject' keys in kustomize-controller pod to support private repositories in transparent way. Usually same secret that source-controller is using.

Hello Vladimir,
I'm wondering how you make it work. :)
I'm trying to reference in Flux-Kustomization on a source with kustomization.yaml which has...

bases:
...
  - ssh://git@somehost.com/repo.git

...but all I got is "kustomize build failed: accumulating resources: 2 errors occurred:
* accumulateFile error: "accumulating resources from 'ssh://git@somehost.com/repo.git': open /tmp/Kustomization-name851797623/path-to-kustomization.yaml/ssh:/git@somehost.com/repo.git: no such file or directory"
* loader.New error: "git cmd = '/usr/bin/git fetch --depth=1 origin HEAD': exit status 128""

Frankly speeking, I'm also big fan of Flux, but we can't use it without referencing across repositories in Kustomization.yaml.
It would be really nice if you could share some example when you succeed with this.
Thanks!

[vladimirfx]

@LLlnak We implement key-based ssh auth by attaching secret volume to kustomize-controller.
Working example:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/instance: flux-system
    app.kubernetes.io/version: latest
    control-plane: controller
  name: kustomize-controller
  namespace: flux-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kustomize-controller
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      labels:
        app: kustomize-controller
    spec:
      containers:
      - args:
        - --events-addr=http://notification-controller/
        - --watch-all-namespaces=true
        - --log-level=info
        - --log-json
        - --enable-leader-election
        env:
        - name: RUNTIME_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        image: ghcr.io/fluxcd/kustomize-controller:v0.5.3
        imagePullPolicy: IfNotPresent
        livenessProbe:
          httpGet:
            path: /healthz
            port: healthz
        name: manager
        ports:
        - containerPort: 8080
          name: http-prom
        - containerPort: 9440
          name: healthz
          protocol: TCP
        readinessProbe:
          httpGet:
            path: /readyz
            port: healthz
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 64Mi
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
        volumeMounts:
        - mountPath: /tmp
          name: temp
        - mountPath: /home/controller
          name: ssh
        - mountPath: /keys
          name: ssh-credentials
        lifecycle:
          postStart:
            exec:
              command:
                - sh
                - "-c"
                - mkdir -p ~/.ssh && cat /keys/identity > ~/.ssh/id_rsa && chmod 600 ~/.ssh/id_rsa && cat /keys/identity.pub > ~/.ssh/id_rsa.pub && cat /keys/known_hosts > ~/.ssh/known_hosts
      nodeSelector:
        kubernetes.io/arch: amd64
        kubernetes.io/os: linux
      securityContext:
        fsGroup: 1337
      terminationGracePeriodSeconds: 60
      volumes:
      - emptyDir: {}
        name: temp
      - emptyDir: {}
        name: ssh
      - name: ssh-credentials
        secret:
          secretName: ssh-credentials
          defaultMode: 0755

[LLlnak]

@vladimirfx this approach works well for me too - thanks a lot!
May be will help for somebody to not edit Flux's manifests for every upgrade - re-worked a bit with Kustomize-patching and secrets with env-variables:
kustomization.yaml

namespace: flux-system
resources:
  - gotk-components.yaml # generated with "flux install --version=latest --arch=amd64 --export > ./gotk-components.yaml"
patchesStrategicMerge:
  - patch-deployment-kustomize-controller.yaml

patch-deployment-kustomize-controller.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustomize-controller
  namespace: flux-system
spec:
  template:
    spec:
      containers:
      - name: manager
        env:
          - name: IDENTITY
            valueFrom:
              secretKeyRef:
                name: bitbucket.ssh-credentials
                key: identity
          - name: IDENTITYPUB
            valueFrom:
              secretKeyRef:
                name: bitbucket.ssh-credentials
                key: identity.pub
          - name: KNOWNHOSTS
            valueFrom:
              secretKeyRef:
                name: bitbucket.ssh-credentials
                key: known_hosts
        lifecycle:
          postStart:
            exec:
              command:
                - sh
                - "-c"
                - mkdir -p ~/.ssh && printf '%s\n' "$IDENTITY" > ~/.ssh/id_rsa && chmod 600 ~/.ssh/id_rsa && printf '%s\n' "$IDENTITYPUB" > ~/.ssh/id_rsa.pub && printf '%s\n' "$KNOWNHOSTS" > ~/.ssh/known_hosts
        volumeMounts:
        - mountPath: /tmp
          name: temp
        - mountPath: /home/controller
          name: ssh
      volumes:
      - emptyDir: {}
        name: temp
      - emptyDir: {}
        name: ssh

[stefanprodan]

@LLlnak thanks for providing this example, this is the way to go until we implement fluxcd/flux2#326 (comment)

[vladimirfx]

@LLlnak thanks for providing this example, this is the way to go until we implement fluxcd/flux2#326 (comment)

IMO multiple merged GitRepository do not solve encapsulation and transparency issues.
Composition of repos through submodules is transparent for developers and very flexible composition mechanism.

Some application projects use distinct repo for common k8s configs and attach it through submodule into application repo. Some inline k8s folder in application repo. And ops... never mind at all. We can switch layouts on the fly and all works.
Submodule is the solution for repository composition from SCM vendor and it woks fine (more or less). Emulation of that mechanism by means of CRD never replaces original one in term of flexibility and transparency.

Maybe extend source-controller with direct support of submodules (through config switch) with ability to extend available secrets? That way we will get best of both approaches - caching + flexibility.

@LLlnak Thank you! We used something similar (without env) already but your approach is better.

[achetronic]

Just for helping future people, I will publish an easier method to do the same using only volumes (it seems cleaner):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- flux-system
- infrastructure
- applications

patches:
  - target:
      kind: Deployment
      name: kustomize-controller
      namespace: flux-system
    path: overlays/patchFluxKustomizeController.yaml
---
# overlays/patchFluxKustomizeController.yaml

# Kustomize Controller implements the remote resources from Git repositories in Kustomization manifests
# but can NOT import the SSH keys using a flag or anything. This way it only works for public Git repositories and not
# for private ones.
# The solution is to patch the Deployment to mount the SSH identity that Flux creates on bootstrap process as a volume
# inside the container

# Create a volume with the 'flux-system' Secret content
# Will result in empty directory if the associated Secret doesn't exist, rather than blocking pod startup
- op: add
  path: /spec/template/spec/volumes/-
  value:
    name: ssh
    secret:
      optional: true
      secretName: flux-system
      items:
        - key: identity
          path: id_rsa
        - key: identity.pub
          path: id_rsa.pub
        - key: known_hosts
          path: known_hosts

# Mount the volume inside the controller container
- op: add
  path: /spec/template/spec/containers/0/volumeMounts/-
  value:
    name: ssh
    mountPath: "/.ssh"

# Add an annotation to make Stakater Reloader rotate the pods when the Secret changes
- op: add
  path: /metadata/annotations
  value:
    secret.reloader.stakater.com/reload: "flux-system"

@stefanprodan @hiddeco
Would you like me to add this to the official documentation?

[kingdonb]

No, I think the direction of the project is towards discouraging "source acquisition" tasks from happening in Kustomize controller. We had a conversation in a recent Flux developer meeting about disabling remote bases in Kustomize altogether, but it would be likely have to be implemented as an opt-in, since otherwise this is a breaking change.

There are a number of benefits that Source Controller provides that Kustomize Controller cannot hope to supplement with further downstream duplication of this same functionality. Not just authentication, but caching, authentification, and cryptographic signature verification, and so network as well as runtime performance will be impacted by taking this end-around the Source controller in Kustomize. I don't think we want to provide instructions or leading guidance in the direction of telling about how to make that end-around.

It's fine if you do this in your own Flux installations, so long as you understand what you're doing, and I don't think we're intent on stopping you if you understand the cost involved, as some providers may pay a metered cost for every bandwidth usage, and GitHub is an example that would almost definitely take you outside of your account boundary, this has potential to cost real money... it is something that cluster operators probably would want to prevent and disable you from doing it.

Especially not only for cost but also security reasons. It can be that your cluster operator sets up policies in a policy enforcement engine as admission webhook, about what URL spec regex the sources in your Flux API can use, and which domains they are allowed to come from.

If you can make these choices in the kustomize build, then you can escape these protections and we think that it is likely cluster operators will need an option to disable this type of behavior, it should be coming in future versions of Flux I hope.

I think we prefer to use more Sources and promote the use of the spec.include feature in GitRepository sources, so that caching and verification features can be engaged.

[rodrigoscferraz]

Just for helping future people, I will publish an easier method to do the same using only volumes (it seems cleaner):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- flux-system
- infrastructure
- applications

patches:
  - target:
      kind: Deployment
      name: kustomize-controller
      namespace: flux-system
    path: overlays/patchFluxKustomizeController.yaml
---
# overlays/patchFluxKustomizeController.yaml

# Kustomize Controller implements the remote resources from Git repositories in Kustomization manifests
# but can NOT import the SSH keys using a flag or anything. This way it only works for public Git repositories and not
# for private ones.
# The solution is to patch the Deployment to mount the SSH identity that Flux creates on bootstrap process as a volume
# inside the container

# Create a volume with the 'flux-system' Secret content
# Will result in empty directory if the associated Secret doesn't exist, rather than blocking pod startup
- op: add
  path: /spec/template/spec/volumes/-
  value:
    name: ssh
    secret:
      optional: true
      secretName: flux-system
      items:
        - key: identity
          path: id_rsa
        - key: identity.pub
          path: id_rsa.pub
        - key: known_hosts
          path: known_hosts

# Mount the volume inside the controller container
- op: add
  path: /spec/template/spec/containers/0/volumeMounts/-
  value:
    name: ssh
    mountPath: "/.ssh"

# Add an annotation to make Stakater Reloader rotate the pods when the Secret changes
- op: add
  path: /metadata/annotations
  value:
    secret.reloader.stakater.com/reload: "flux-system"

@stefanprodan @hiddeco Would you like me to add this to the official documentation?

Look's good but from our side we also need to add a volume mount using the same Volume created from the secrets pointing to the /home/controller/.ssh folder

      # Mount the volume inside the /home/controller/ folder
      - op: add
        path: /spec/template/spec/containers/0/volumeMounts/-
        value:
          name: ssh
          mountPath: "/home/controller/.ssh"

Apparently the kustomize controller user on Flux v0.20.01(the version that we are using) is "controller", we had another kustomize that get a file from another git repository(private) using ssh

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- kube-system/efs.yaml
- overprovisioning/manifests.yaml
- ssh://git@github.com/teste/kubernetes-addon-external-secrets.git//kustomize/external-secrets?ref=v1.1.2
- ssh://git@github.com/teste/kubernetes-addon-external-dns.git//kustomize/external-dns?ref=v1.1.2

If we didn't mount the secret on the /home/controller path the kustomize controller can't connect to the github repo.

So maybe this can also help someone else.

[achetronic]

No, I think the direction of the project is towards discouraging "source acquisition" tasks from happening in Kustomize controller. We had a conversation in a recent Flux developer meeting about disabling remote bases in Kustomize altogether, but it would be likely have to be implemented as an opt-in, since otherwise this is a breaking change.

There are a number of benefits that Source Controller provides that Kustomize Controller cannot hope to supplement with further downstream duplication of this same functionality. Not just authentication, but caching, authentification, and cryptographic signature verification, and so network as well as runtime performance will be impacted by taking this end-around the Source controller in Kustomize. I don't think we want to provide instructions or leading guidance in the direction of telling about how to make that end-around.

It's fine if you do this in your own Flux installations, so long as you understand what you're doing, and I don't think we're intent on stopping you if you understand the cost involved, as some providers may pay a metered cost for every bandwidth usage, and GitHub is an example that would almost definitely take you outside of your account boundary, this has potential to cost real money... it is something that cluster operators probably would want to prevent and disable you from doing it.

Especially not only for cost but also security reasons. It can be that your cluster operator sets up policies in a policy enforcement engine as admission webhook, about what URL spec regex the sources in your Flux API can use, and which domains they are allowed to come from.

If you can make these choices in the kustomize build, then you can escape these protections and we think that it is likely cluster operators will need an option to disable this type of behavior, it should be coming in future versions of Flux I hope.

I think we prefer to use more Sources and promote the use of the spec.include feature in GitRepository sources, so that caching and verification features can be engaged.

Sorry for the long delay on answeting, but we have had a lot of tasks in the middle. I understand your point but we have a strong reason to do this.
I would prefer to talk about this in private to discuss the use case and not to promote it here until we get some conclusions 💪🏻👨‍💻

[vdksystem]

Will add my 5cents to this thread, as I ended up here, looking for some "obvious" feature.
In my case I'm trying to convince team to switch from hundreds of helm chart with centralised library chart to something more easy to maintain and still flexible (as we use helm charts only intrnally).
In this case we switch from helm-library -> helm chart -> helmrelease to base kustomization -> kustomization in gitops repo -> flux kustomization. It is very strait forward to use remote base.
At the very beginning I decided to go with flux, because it is just a wrapper on top of existing tooling. So if I was using helm and want to use flux, I can just add it into existing deployments. If I want to remove flux, I continue to use helm without flux.
I was expecting to have same behaviour with kustomize.
To follow suggested "best practices" I need to use either include, or submodules. I tried - but very simple kustomize with remote base thing now is totally not simple with either hundreds of include in flux-system git source or hundreds of git sources with flux-system include and hundreds of flux kustomizes (comparing to single helm release file)

In our case I would like to use https, su just added gitconfig to kustomize-controller
.gitconfig

[url "https://api:github_pat_TOKEN@github.com/"]
    insteadOf = https://github.com/

patch.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustomize-controller
spec:
  template:
    spec:
      containers:
      - name: manager
        volumeMounts:
        - mountPath: /.gitconfig
          name: gitconfig
          subPath: .gitconfig
      volumes:
      - name: gitconfig
        configMap:
          name: gitconfig

kustomization.yaml

configMapGenerator:
  - files:
      - .gitconfig
    name: gitconfig
    namespace: flux-system
generatorOptions:
  disableNameSuffixHash: true
patches:
  - target:
      version: v1
      group: apps
      kind: Deployment
      name: kustomize-controller
      namespace: flux-system
    path: patch.yaml