Github Enterprise SSH handshake failing #652
Comments
|
It seems that the bug was introduced in v0.22.0 of the sourcecontroller. Should i open the bug there? |
|
@steveizzle thank you for reporting this. Do you mind sharing what |
|
Can you also confirm whether this is consistent and every single reconcile yields the error mentioned, or is intermittent and happens every so often? Would you mind running the latest version of |
I used go-git and tried now libgit2. With libgit2 it is working.
Sure, i will give it a try and report back |
The error seems to be consistent. As soon as i enable the experimental transport neither libgit2 nor go-git is working: We found out that one repository is using an ED25519 Key which seems to be still working. Normally we use RSA keys. |
|
Thanks for that. Do you mind sharing the errors you get when using |
|
We are getting the same error suddenly after redeploying a test cluster that we deploy often (almost every day) and doesn't give any errors with Flux. Is there any resolution for this, or just keep trying until it works? |
|
I currently pinned an old version, but will spent some time on getting more debug Infos the next days |
|
@steveizzle @campanellaf this may have to do with some of the crypto level settings that may have changed upstream. We have just released a RC with some improvements specially around the |
|
@pjbgf Regarding the rc, Unfortunately the rc didnt fix the problem: This is a example of a git repository in my test cluster: |
|
I also checked out the algorithms for the known_hosts and got the following: Working:
Not Working:
So i would say its not related to the known_hosts.. And still everything is working when i switch the source controller back to v0.21.2 |
|
I did not set the Experimental env Variable in the test with the rc. Is this still necessary with the rc? |
|
@steveizzle we have released source-controller yesterday with managed transport enabled by default. The official image is here: https://github.com/fluxcd/source-controller/pkgs/container/source-controller/23808902?tag=v0.25.0 Please test it with go-git and libgit2, no env var needed. |
|
Unfortunately we have got the same problem with v0.25.0 :( |
|
I also let it ran with loglevel debug, but i think stacktrace doesnt help much: |
|
@steveizzle the repositories that are not working, are all of them on the same Git server? Also, I noticed that the failing repos are using ssh-rsa for Auth-N, can you confirm that is enabled on the server? We have released another release candidate that further improves on the connection management and observability.
|
|
About the auth-N: That should be a yes. And the rsa key is still working with the old version of the source controller.. Regarding the new rc: |
|
i meet same issue. My flux env : |
|
@Kevinwoolworth @steveizzle would it be possible for you to post the description of the secret that the |
|
@aryan9600 Sure, we switched to working authentications, but i surely will help to assist on this issue: |
|
Hi is there any update or potential timeline for this issue? |
|
Having this issue on a private GitLab instance as well. I am running source-controller v0.25.11 and still having this issue: |
|
@cdenneen Can you share more details about your GitLab instance? I'm running GitLab 15.2.2-ee and it's working fine. |
|
I did these steps, using AWS EKS and CodeCommit, from the scratch: and I got this error: I am using:
|
|
@anetogit are you sure that your ssh key-pair isn't protected by a password? if it is, then you could create a secret using |
|
@anetogit this issue is about GitHub not AWS… For AWS CodeCommit please read the docs: https://fluxcd.io/docs/use-cases/aws-codecommit/ |
@aryan9600 No, my key is not protected with a password, there is no passphrase. I was using private key because of the CodeCommit of AWS. I am going to try |
@stefanprodan Yes. The tutorial works with the HTTPS: but not with SSH, because o the issue: thanks. |
|
I hope this is placed in the correct thread. I am facing a similar issue with Github Organizations where I can successfully connect to private repositories via ssh, but after roughly 5-10 minutes 'flux get source git' will show failure with error message: failed to checkout and determine revision: unable to fetch-connect to remote 'https://github.com/{org}/{repo}': unhandled HTTP error 401 Authorization Required. I can run 'flux create source {flags}' right after and it will re-establish the connection without issue, then repeat failure. |
|
This is still happening, on public Github, non-enterprise: |
|
@LockedThread given this issue is kind of stale, can you please share your source-controller version? |
The version does show in my comment but here is the output of |
|
I also just decided to update to see if the issue fixed itself, still broken: |
|
This is the bootstrap command I am using: |
|
Here's more context from flux events: |
|
Can you please try to rotate the credentials (and things like |
I really appreciate the quick response. That worked. You're really saving me right now. |
|
You're welcome! If that worked (and based on the error), I suspect the deploy key got deleted from the repository. As from the top of my head, a |
The deploy key wasn't deleted and it was being used up until 6 days ago when it randomly broke. This was in our production cluster and I was freaking out why my changes weren't getting deployed. |
|
Did your PAT expired 6 days ago? The deploy key expires when the GH token used for bootstrap expires. If you use PATs with an expiry date, you need to make sure to rotate the key before they expire. |
None of the keys or PATs expired |
|
I had the same issue was driving me crazy. Turned out that I specified the URL wrong for SSH: - url: ssh://MY-ORG@github.com/MY-REPO.git
+ url: ssh://git@github.com:22/MY-ORG/MY-REPO |
Describe the bug
We use flux2 with our github enterprise instance. This worked fine until the 0.28.3 Release. Since the Update we cannot fetch our repos anymore and get the following error message:
k get gitrepositories -n flux-system NAME URL AGE READY STATUS flux2-sync ssh://git@foo/abc/def 64d False failed to checkout and determine revision: unable to clone ' ssh://git@foo/abc/def': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remainWith version 0.27.0 everything is fine again, without any other change to the environment (same keys and so on):
If you need more information, please let me know. Unfortunately the source-controller log also doesnt give me more than the above error message
Steps to reproduce
We installed flux with the community helm charts without any relevant change to the values.yaml
Expected behavior
It is working again ^^
Screenshots and recordings
No response
OS / Distro
N/A
Flux version
N/A
Flux check
► checking prerequisites
✗ flux 0.24.1 <0.28.5 (new version is available, please upgrade) //cli is not used
✔ Kubernetes 1.21.5-eks-bc4871b >=1.19.0-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.18.2
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.21.1
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.17.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.22.2
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.23.1
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.22.3
✔ all checks passed
Git provider
GitHub Enterprise
Container Registry provider
No response
Additional context
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: