Private remote target not working

[wolfmah]

flux version: v0.16.1
ghcr.io/fluxcd/kustomize-controller: v0.13.2

Our setup is that each application have their Kustomize configuration in their own repo, residing along the actual code. Some of the configuration (mostly labels) are pretty much the same across our fleet. In an effort to turn our Kustomize configuration more DRY, I was trying to implement remote target for common Components that apply mostly patches. Those Components resides in a private repository, which also contains all our Kustomize infrastructure applications.

I was testing locally and came up with the design by manually calling kustomize build ... and made it work. But once inside kustomize-controller, it was no longer working. I searched for solution or insight through the documentation, but I haven't found anything. Though I came across this quote from an issue of flux2: fluxcd/flux2#1445 (comment)

[...] you need to use Flux GitRepository as kustomize-controller doesn't know how to handle SSH auth.

It would be awesome if this feature, available to Kustomize, would be made available to kustomize-controller. It could reuse the same secret created for GitRepository.

As a work around, I have tried using GitRepository.spec.include and it works. You just need to bind the two repositories, and add the inclusion to the application GitRepository. What is a deal breaker is that you loose the ability to do local (or via CI) kustomize build ..., because the injected folder is never to be found in the git repository.

Seeing that, I tried to create a symbolic link at the designated target folder inside the application repository (spec.include[*].toPath). Locally, it work, but once the symbolic link is commit in the repository, the application GitRepository doesn't like it and error out instead of trying to replace the existing path with the intended folder coming from the inclusion.

Is their any alternative to using remote target on private repositories, that I haven't thought of, that would still allow us the use of local kustomize build ... while also working with Flux?

[JaneLiuL]

Good issue!
If you want to use kustomize build.., then you need to write another kustomization.yaml (below is an example) to test remove gitrepository, but it would not work with kustomize-controller.

resources:
- git@gitlab.com:someName/test.git/bases/?ref=test

Please correct me if I am wrong.

[wolfmah]

Haven't thought o that one: having two kustomization.yaml, one for Flux, one for local/CI. I haven't had the time to test it, but I'm certain it would work. And the way I see it, it could still be relatively DRY, in the sense that the local kustomization could reference the other one.

But I want to emphasize, that this is still only a workaround. If kustomize-controller could understand SSH/HTTPS authentication, thus allowing the full Kustomize featureset, it would be a lighter configuration, easier to understand/use and easier to maintain.

I'll most likely end up using it because having DRY code is more appealing to me than the complexity of making it work.

I'll try to find time this week to test this out. I'll post back the result.

[stefanprodan]

This controller is not meant to be used standalone, it’s part of Flux and depends on source-controller for manifests acquisition, origin validation (pgp) and caching. To kustomize resources you have 3 options: in-line patches, source includes and git submodules.

[wolfmah]

To kustomize resources you have 3 options: in-line patches, source includes and git submodules.

The way I see it (and, of course, all this is base on how we organized our kustomization):

• In-line patches are not ideal for our use case. It would require duplicating a lot of stuff in each repositories. Not ideal when you have to update one common annotation across the entire fleet.
• Source includes is a Flux solution that breaks a standalone kustomize build. We still need to retain the ability to run kustomize build for our CI: we validate the Kustomize code before merging into the master branch.
• Git submodules could work, if you are disciplined and know what you are doing in git. Unfortunately, at my current work place, that would quickly turn into a disaster with a lot of wasted time trying to coach all the devs.

I didn't go through with what was outlined above. Instead, I found this gem: injecting the git ssh credentials inside kustomize-controller make private remote targets work.

Thanks to this issue comment: #158 (comment)

As far as I'm concern, that's a clean solution to my problem.

[BarthV]

Injecting ssh credentials in kustomize-controller homedir is now broken since v0.20.0 release 3 days ago.

Flux now deploy controller without root access, using a read-only FS with low priviledged "nobody" account.

This breaks a bit this workaround that I'm also using on a daily basis :(

EDIT : solved by creating a /.ssh emptyDir & using it as nobody ~/.ssh (since / is useraccount homedir)