Best Practies FluxCD Performance

[airluke19]

Hello guys,
I am writing to get information about your experience.

We are currently managing an eks 1.28 cluster with fluxcd

❯ flux version
flux: v2.2.3
distribution: flux-v2.2.3
helm-controller: v0.37.4
kustomize-controller: v1.2.2
notification-controller: v1.2.4
source-controller: v1.2.4

We currently have a single repository with branches that identify the environment

With fluxcd we have only one gitRepository and n. kustomization for every application related to each namespace. For example:

namespace1-app1-kus.yaml
namespace1-app2-kus.yaml
namespace1-app3-kus.yaml
namespace1-app4-kus.yaml
namespace1-app5-kus.yaml
namespace1-app6-kus.yaml
...

namespace2-app1-kus.yaml
namespace2-app2-kus.yaml
namespace2-app3-kus.yaml
namespace2-app4-kus.yaml
namespace2-app5-kus.yaml
namespace2-app6-kus.yaml
...

namespaceX-app1-kus.yaml
namespaceX-app2-kus.yaml
namespaceX-app3-kus.yaml
namespaceX-app4-kus.yaml
namespaceX-app5-kus.yaml
namespaceX-app6-kus.yaml
...

We currently manage these objects:

The infrastructure is growing and fluxcd becomes slower and slower when it has to reconcile after a commit to the repository

❯ flux tree kustomization flux-system | wc -l
 1444

We performed tuning as per fluxcd documentation
https://fluxcd.io/flux/installation/configuration/vertical-scaling/

This is our fluxcd manifest

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patches: |
 apiVersion: apps/v1
 kind: Deployment
 metadata:
 name: all
 specs:
 templates:
 metadata:
 annotations:
 cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
 specs:
 affinity:
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: workload
 operator: In
 values:
 - mgm
 containers:
 - name: manager
 resources:
 limits:
 cpu:
 memory:
 target:
 kind: Deployment
 labelSelector: app.kubernetes.io/part-of=flux
 - patches: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --concurrent=10
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --requeue-dependency=5s
 target:
 kind: Deployment
 name: "(helm-controller|source-controller)"
 - patches: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --concurrent=30
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --requeue-dependency=5s
 - op: replace
 path: /spec/template/spec/volumes/0
 value:
 name: temp
 emptyDir:
 medium: Memory
 target:
 kind: Deployment
 name: kustomize-controller
 - patches: |
 - op: add
 path: '/spec/template/spec/volumes/-'
 value:
 name: persistent-data
 persistentVolumeClaim:
 claimName: gotk-pvc
 - op: replace
 path: '/spec/template/spec/containers/0/volumeMounts/0'
 value:
 name: persistent-data
 mountPath: /data
 target:
 kind: Deployment
 name: source-controller

However, we have noticed that increasing the concurrent value takes up a lot of memory and cpu usage and we do not want to raise it further

When we execute the reconcile command it takes a long time
flux reconcile ks kustomization_name --with-source

Are there best practices we can apply?
In particular, I wonder if making multiple gitrepositories (which always point to the same application repository) for each namespace, therefore with the relative kustomizations, could help to have shorter reconciliation times.

If I create n gitrepositories for each namespace and the kustomizations will refer to the gitrepository of the right namespace, when I run the command flux reconcile ks kustomization_name --with-source I imagine it will be faster since the kustomizations of only that namespace will be reconciled, correct?

[stefanprodan]

You can either split the monorepo per tenant/app or switch to OCIRepository and push each tenant/app as a dedicated artifact. Based on the benchmark, Flux can reconcile 1K OCIRepo/Kustomization pair in 4m https://github.com/fluxcd/flux-benchmark/blob/main/RESULTS.md