-
Notifications
You must be signed in to change notification settings - Fork 69
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Goroutine leak in image-automation-controller with EXPERIMENTAL_GIT_TRANSPORT #334
Comments
(since writing this, I've upgraded to v0.28.2 and will report back if that makes a difference) |
Related to fluxcd/source-controller#636 |
For now, this doesn't seem to be happening after the v0.28.2 upgrade, I'll re-open if it reoccurs:
|
This seems to be still ongoing, with
set on the pod
|
@jiphex do you mind checking whether you can still reproduce this with out latest release candidate? This RC is based on #369, therefore you no longer need to set the environment variable to enable the experimental transport, as Managed Transport will be enabled by default. |
We have a release candidate version with the fix: |
This version of source-controller introduces a fix for the SSH connections leak issue reported at: #334 Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
This version of source-controller introduces a fix for the SSH connections leak issue reported at: #334 Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
This version of source-controller introduces a fix for the SSH connections leak issue reported at: fluxcd#334 Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
This version of source-controller introduces a fix for the SSH connections leak issue reported at: #334 Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
This version of source-controller introduces a fix for the SSH connections leak issue reported at: #334 Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
This version of source-controller introduces a fix for the SSH connections leak issue reported at: #334 Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
This version of source-controller introduces a fix for the SSH connections leak issue reported at: #334 Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
This version of source-controller introduces a fix for the SSH connections leak issue reported at: #334 Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
This version of source-controller introduces a fix for the SSH connections leak issue reported at: fluxcd#334 Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Before v0.21.0, there seemed to be an issue with Image Automation that meant that the
source-controller
ended up leaking goroutines which would have working (responding toClientAliveInterval
pings), active connections to the SSH server defined in theGitRepository
source for the automation. One of the reasons I started to look intoEXPERIMENTAL_GIT_TRANSPORT
was to see if it'd fix this, because these connections remain open forever on our Gitlab server and eventually stop SSH checkouts from working.With
EXPERIMENTAL_GIT_TRANSPORT=true
, it seems like this goroutine leak has now moved to theimage-automation-controller
. Here's a Prometheus graph ofgo_goroutines
for the image-automation-controller:At this point, the image-automation-controller pod has been up for 20 hours and there's 95 open SSH connections to the Gitlab server. To prove this is the culprit, I've restarted the image-automation-controller pod:
The resources (imageautomation,gitrepository) are the same as my previous issue, and available here: https://github.com/fluxcd/image-automation-controller/files/8333914/image-repo.yaml.txt
The image-automation-controller deployment is as follows:
The text was updated successfully, but these errors were encountered: