Helm Operator - Allow Operator level Global Values Secrets

[captncraig]

I have one repo I intend to use for multiple clusters with a layout something like this:

base-cluster/helm-releases
    --ingress.yaml
    --cert-manager.yaml
    --dashboard.yaml
clusterA/helm-releases
    --serviceA.yaml
    --serviceB.yaml
clusterB/helm-releases
    --serviceC.yaml
    --serviceD.yaml

The intent is to have a flux operator running in each cluster, each of them having a --git-path that includes base-cluster, as well as it's own cluster-specific subdirectory. This could even be extended to have a deeper hierarchy of shared folders for different classes of clusters.

Many of my charts depend on a few different values, and it would be nice to have a few things in the shared base charts modifiable on a per-cluster basis. Things like:

• .Values.global.cluster is used in most charts for urls and things.
• .Values.ingress.image.version may have a default value in the base charts, but we may want to override in a particular cluster to test an upgrade.
• .Values.experimentX.enable may be useful to gate features in some clusters but not others.

There's more examples as well.

My current options for getting values into a chart are:

1. Explicitly put them in the values in a helmRelease object. This is not usually feasible for shared charts without copying them for each cluster, or somehow templating them or something.
2. Specify a values secret. This is limited in that it must be in the same namespace as the individual helm release. So if I have a central values secret, it would need to be copied into each release namespace and added to each helmRelease object.

My proposal is to allow one or more values secrets to be given to the helm operator that will serve as the base for all helm releases handled by the operator. The secret(s) can (or must) be in the same namespace as the operator itself. In my case I would likely create a secret alongside the helm operator as part of my cluster bootstrapping process. I would want all releases created by that operator to use those values as the base, with any explicit values or secret-imported values overriding them for individual releases.

These secrets can be specified to the operator with a command-line flag, as they should not change often.

Does that seem like a feasible approach? I can work on a PR if so.

[captncraig]

One downside might be that "magic" global values might not be expected by chart authors, and hard to debug for someone unaware that they exist. So if some global value interacts poorly with a public chart or something it could get confusing.

But hopefully anyone using this feature would be careful to use well-namespaced value keys, and use global values sparingly if at all.

This would not be a breaking change, since you'd need to opt-in by adding command line arguments.

[squaremo]

Thanks @captncraig for this well-explained feature request. I think it's a reasonable thing to add -- I understand the requirement, and I can't see a better way to accomplish it.

It's worth mentioning that #1468 was partly aimed at this use case, but (to avoid a similar magic problem to that you mention) it ended up using per-release values. This use case differs enough, I think, to be worth its own implementation.

[stefanprodan]

This can be accomplished by using the cross-namespace support #2334

You would create the secrets and HelmReleases in the flux namespace and helm-op will install the charts according to the targetNamespace field.

[stealthybox]

I think the cross-namespace feature for this might be unrelated to the request.

For sharing base values in the HelmReleases of different git.paths, you want to use manifest generation. This is a generic way for you to run any logic. You can factor your repository structure in the way you want.

I would use kustomize like this:

---
# ClusterA helm-releases
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base-cluster/helm-releases
- myapp-helm-release.yaml
patchesJson6902:
- target:
    group: flux.weave.works
    version: apiextensions.k8s.io/v1beta1
    kind: HelmRelease
    name: myapp
  path: ../../base-cluster/helm-values.patch.yaml
patches:
- myapp-value-overrides.patch.yaml
- cert-manager-value-overrides.patch.yaml

In this example ../../base-cluster/helm-values.patch.yaml modifies the values object of the myapp helm-release for ClusterA.

Note you'll need to use the kustomize build --load_restrictor=none flag to load patches from a parent directory. (explanation)
The default releases can be loaded from a different kustomize base without the load_restrictor flag.

[kingdonb]

In Flux v2, load_restrictor=none is the default behavior of Kustomize controller and many workflows like this are possible with the new documentation supporting manifest generation outside of Flux, in CI

There are many approaches that will allow more granular control over global values and filter down to releases through different processes. Secret generation capabilities are intentionally limited in Flux v2, if Mozilla SOPS support does not meet your needs, please open a discussion in the new repo.

I will close the issue for now as it is stale. Flux v1 remains in maintenance mode.

We can provide support to Flux v1 users especially with respect to migrating to the new Flux version, Flux v2, which has some breaking changes but is also a total rewrite and also brings many improvements. I'm closing the issue but you are most welcome to write back on this issue, or open another issue. Thanks for using Flux.