Specify cluster specific values for nginx-ingress controller (e.g. public IP)?

[dev-samples]

I am using flux2 base on this example:
https://github.com/fluxcd/flux2-kustomize-helm-example

to maintain a set of AKS k8s clusters. I am not sure where to put my cluster specific nginx-controller values though.

In the above example nginx values lives in:

infrastructure/
 - nginx/
       release.yaml

meaning it will be applied across all cluster.

But I need to have cluster specific information/values for nginx like:

  values:
    service:
      loadBalancerIP: '20.x.y.z'
      annotations:
        service.beta.kubernetes.io/azure-load-balancer-resource-group: cluster-01-public-ip-rg

Since I am storing public IPs for nginx in separate aks resource groups for each cluster.

I would still like to share common configuration across cluster like:

spec:
  releaseName: nginx-ingress-controller
  chart:
    spec:
      chart: nginx-ingress-controller
      sourceRef:
        kind: HelmRepository
        name: bitnami
        #namespace: nginx
        namespace: flux-system
      version: "5.6.14"
  interval: 2m0s
  install:
    remediation:
      retries: 3
  # Default values
  # https://github.com/bitnami/charts/blob/master/bitnami/nginx-ingress-controller/values.yaml
  values:
    service:
      type: LoadBalancer  

Right now I am considering to move nginx configuration/values to the apps folder/files structure since it has the concept of base folder/configuration and cluster specific configuration.

But is there a similar way to configure common and cluster specific configuration for infrastructure?

[stefanprodan]

Please see the docs on Ways of structuring your repositories

TLDR: using a kustomize base and overlays for the infrastructure items is the right approach

[dev-samples]

Thanks, based on: https://fluxcd.io/docs/guides/repository-structure/#repository-structure for the infrastructure part I now have:

├── apps
├── clusters
├── infrastructure
│   ├── base
│   │   ├── cert-manager
│   │   │   ├── kustomization.yaml
│   │   │   ├── namespace.yaml
│   │   │   └── release.yaml
│   │   └── nginx
│   │       ├── namespace.yaml
│   │       └── release.yaml
│   ├── dev
│   │   └── nginx
│   │       ├── kustomization.yaml
│   │       └── values.yaml
│   ├── kustomization.yaml
│   ├── sources
│   │   ├── bitnami.yaml
│   │   ├── jetstack.yaml
│   │   ├── kustomization.yaml

With content for the files below:

infrastructure/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- sources
- base/cert-manager
- dev/nginx

/infrastructure/dev/nginx/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: nginx
resources:
  - ../../../base/nginx/namespace.yaml
  - ../../../base/nginx/release.yaml
  - values.yaml  

infrastructure/dev/nginx/values.yaml

spec:
  # Default values
  # https://github.com/bitnami/charts/blob/master/bitnami/nginx-ingress-controller/values.yaml
  values:
    service:
      loadBalancerIP: '20.x.y.z'
      annotations:
        service.beta.kubernetes.io/azure-load-balancer-resource-group: cluster-01-public-ips

infrastructure/base/nginx/release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: nginx
spec:
  releaseName: nginx-ingress-controller
  chart:
    spec:
      chart: nginx-ingress-controller
      sourceRef:
        kind: HelmRepository
        name: bitnami
        namespace: flux-system
      version: "5.6.14"
  interval: 2m0s
  install:
    remediation:
      retries: 3
  values:
    service:
      type: LoadBalancer    

One of the errors I now get is:

...infrastructure/dev/nginx/values.yaml' must be a directory to be a root, get: invalid source string: values.yaml\\\"\\n\\n\"\n\n"}

So thats not the way to feed it cluster specific values.

Any suggestions on what needs to be corrected?

[stefanprodan]

you’re missing patchesStrategicMerge for values, see this example: https://github.com/fluxcd/flux2-kustomize-helm-example/tree/main/apps/production