SOPS AZKV getDefaultAzureCredential triggers API rate limit

[apeschel]

When the Kustomize controller comes up, the getDefaultAzureCredential function will hammer the Azure API with login requests and get rate limited for a few minutes, resulting in failed Kustomize runs.

It would be preferable to cache the auth token and reuse it, rather than retrieve a new one on each SOPS decryption.

https://github.com/fluxcd/kustomize-controller/blob/main/internal/sops/azkv/keysource.go#L216

[kingdonb]

I don't know if we can cache the auth token and reuse it across Kustomizations without violating multi-tenancy. You can set retryInterval to a shorter value than interval if you don't want to wait for the whole interval (and since you've probably already set it longer than the default interval to avoid the rate limiting as much as possible.)

There may be some improvement that's possible to handle the rate limiting behavior a bit more cleanly.

[Maxcodesthings]

So for every kustomization that uses SOPS it will decrypt the secret each interval? That does sound overkill. I am also nearing a limit for the number of calls SOPS is making to AKV.

Is the feature gate CacheSecretsAndConfigMaps supposed to help with this?

[souleb]

This is being worked on in fluxcd/pkg#766