Flux is flooding bitbucket server with ESTABLISED SSH connection

[vudzero]

Describe the bug

Hi all,

After using flux for a couple of days now, we found out that our bitbucket server was flooded with ESTABLISED connections from our flux installations(2 clusters). We found out because some of our users where not able to execute SSH operations since the maximum concurrent connections was reached on the server (our max is 250 concurrent connections). It turns out that 240 of those connections where bound to our kubernetes ip addresses.

Using tcpdump, we found out that transactions were happening on all of those connections.

When we execute ssh transaction on our local machines, the ESTABLISHED connections only lives for about 1 second on the server. What could explain that the flux's ones seems to be kept alive forever?

-Our kubernetes cluster is hosted in AKS
-The reconciliation period is every 1min

Thanks everyone for your support!

Steps to reproduce

Flux was bootstrap with the following command

flux bootstrap git \
 --components-extra=image-reflector-controller,image-automation-controller \
 --context=aks-<ENV> \
 --branch=main \
 --url=ssh://git@<our_git_ssh_address>:7999 \
 --path=clusters/<ENV> \
 --private-key-file=/<path to certificate>
 --silent

Expected behavior

Expect flux reconciliation to happen and then have the ssh connection closed with the server

Screenshots and recordings

No response

OS / Distro

Ubuntu 20.04

Flux version

V0.31.1

Flux check

► checking prerequisites
✔ Kubernetes 1.22.6 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.22.1
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.23.2
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.19.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.26.1
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.24.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.25.5
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta2
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta1
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta1
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed

Git provider

Bitbucket server

Container Registry provider

Private

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[aryan9600]

Hi @vudzero, this is a known issue (ref: fluxcd/image-automation-controller#334). Please upgrade your flux installation, the fix for this issue was released as a part of flux2 v0.31.2.

[vudzero]

Thanks @aryan9600 , the update does fix the issue. Thanks for your help!