gpg-agent does not support SSH certificates, tsh should be aware of this

[webvictim]

What happened:

• Yubikeys use gpg-agent to hold keys/certificates
• gpg-agent does not support SSH certificates (https://dev.gnupg.org/T1756)
• When gpg-agent is running and you log into a Teleport cluster, tsh adds stuff to the agent that it doesn't understand
• This causes issues for people using Yubikeys

What you expected to happen:
1)tsh should have a command-line flag which disables any attempt to write to a running agent
2) Ideally, tsh would be able to detect when gpg-agent is running rather than ssh-agent and deliberately avoid writing its keys there.

This would enable much smoother Teleport operations for Yubikey users. tsh still writes its certificates to ~/.tsh/keys/<cluster>, which means that people can use these for their Teleport operations instead.

How to reproduce it (as minimally and precisely as possible): Run gpg-agent, log into a Teleport cluster using tsh then observe that what's in the agent is not a valid certificate (error fetching identities: Invalid key length)

Environment:

• Teleport version (use teleport version): Teleport Enterprise v4.1.4git:v4.1.4-0-gc487a75c go1.13.2
• Tsh version (use tsh version): Teleport v4.1.4 git:v4.1.4-0-gc487a75c go1.13.2
• OS (e.g. from /etc/os-release): Fedora 30

[russjones]

In a recent discussion with a customer:

Sure. gpg-agent works with our YubiKeys so when we have individual users SSH keys,
they would be stored on a yubikey. gpg-agent is the agent that connects between the
YubiKey and acts as an ssh agent. The normal ssh-agent doesn't have such a capability.
So, we would switch ssh-agent out for gpg-agent. When we do that, when tsh adds the
ssh certificate to gpg-agent, it doesn't know how to handle it (It adds the cert, but
then will no longer serve up any identities, effectively killing the gpg-agent)

[thardie]

Interestingly, on the YubiKey side, at least 1 other user claims there is a way to make this work: drduh/YubiKey-Guide#43

[benarent]

This came up again with Reliability Improvements for E- Adding other customer to the issue.

[aelkugia]

Had another conversation with a customer interested in this, who has some engineers using gpg-agent/yubikey.

[cpanato]

+1

[cnelson]

Also ran into this. currently working around it with alias tsh='SSH_AUTH_SOCK= tsh'

[awly]

Fix is in review.

FWIW, it appears that OpenSSH 8.2+ supports security keys now: https://buttondown.email/cryptography-dispatches/archive/cryptography-dispatches-openssh-82-just-works/.
I haven't tried it myself, but might be interesting for folks in here.

[Chili-Man]

Is there anyway to disable tsh login from generating an SSH key when we only use teleport for kubernetes access?

And for those wondering how to resolve the error by removing the offending keys; they can be manually removed from within the ~/.gnupg/private-keys-v1.d directory

[webvictim]

There's not currently a way to stop SSH certs from being generated completely, but you can prevent tsh from attempting to add them to the agent by using tsh login --no-use-local-ssh-agent or setting the TELEPORT_USE_LOCAL_SSH_AGENT environment variable to false.

[Chili-Man]

Even using tsh login --no-use-local-ssh-agent doesn't seem to help; after I login, and then I try to use the loaded kubeconfig provided through that command, by invoking any kubectl command, its added back into the ssh agent when kubectl tryies to get the credentials with the command in kubeconfig tsh kube credentials --kube-cluster=teleport.example.com --teleport-cluster=teleport.example.com

[Chili-Man]

After further testing, it appears that if you manually add the --no-use-local-ssh-agent as part of the command it uses in the kubeconfig to get the credentials, it won't add the ssh key to your gpg-agent. Is there a way to have that flag automatically added?

it appears that you can actually export TELEPORT_USE_LOCAL_SSH_AGENT=false in your shell's profile or rc file, where it will preserve the effect of not adding the tsh ssh keys to your gpg-agent

[webvictim]

@Chili-Man What happens if you run export TELEPORT_USE_LOCAL_SSH_AGENT=false (or whatever is appropriate for your shell) and then try? Does the embedded tsh command honour that? (it should)

[Chili-Man]

@webvictim thanks, that works! I've updated my comment above to reflect that.

[klizhentas]

@awly please look into the issue when working on U2F support for tsh. In addition to that we have requests for people who don't want the keys to be written to disk and stored only in memory in agent.

I think we have to build a matrix of all possible combinations we are going to support and provide some UX that integrates well with U2f support you are adding.

[klizhentas]

#4863

[natikgadzhi]

+1. Just shot myself in the foot with this, too.

[thardie]

The only reason I was using gpg-agent was to use a yubikey GPG based auth for SSH. You can instead use PIV with Yubikey to achieve similar functionality, then use the regular SSH Agent instead of gpg-agent. See here for details on how to set this up: https://developers.yubico.com/PIV/Guides/SSH_with_PIV_and_PKCS11.html

Using this method then lets teleport and yubikey coexist properly with a regular SSH Agent.

[Chili-Man]

@thardie unfortunately, it appears that PIV only works if your gpg key was an RSA 2048-bit key 😞

Yubico/yubico-piv-tool#58

Seems to be an underlying issue with the specification for PIV

[webvictim]

Related to #5487

[santiago-mooser]

Sorry to comment on a closed issue, but this is happening again with Teleport Connect. However, this only happens when carrying out SSH operations outside of Teleport Connect. Teleport Conect itself works fine, as does tsh.

e.g. We'r using yubikeys that use gpg-agent to authenticate to git, so when running a git fetch usually we'd be able to authenticate with a yubikey and pulling the updates from the repo. However, if we launch Teleport Connect, it'll add new keys to the ~/.gnupg/private-keys-v1.d/ folder and cause issues with the git operation:

debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: get_agent_identities: ssh_fetch_identitylist: Invalid key length
debug1: Will attempt key: /Users/username/.ssh/id_rsa
debug1: Will attempt key: /Users/username/.ssh/id_ecdsa
debug1: Will attempt key: /Users/username/.ssh/id_ecdsa_sk
debug1: Will attempt key: /Users/username/.ssh/id_ed25519
debug1: Will attempt key: /Users/username/.ssh/id_ed25519_sk
debug1: Will attempt key: /Users/username/.ssh/id_xmss
debug1: Will attempt key: /Users/username/.ssh/id_dsa
debug1: Trying private key: /Users/username/.ssh/id_rsa
debug1: Trying private key: /Users/username/.ssh/id_ecdsa
debug1: Trying private key: /Users/username/.ssh/id_ecdsa_sk
debug1: Trying private key: /Users/username/.ssh/id_ed25519
debug1: Trying private key: /Users/username/.ssh/id_ed25519_sk
debug1: Trying private key: /Users/username/.ssh/id_xmss
debug1: Trying private key: /Users/username/.ssh/id_dsa
debug1: No more authentication methods to try.
git@domain.tld: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I don't know whether this has to do with the key length of the server, but openssh refuses to work with keys of length <1024: https://www.openssh.com/txt/release-7.6

The keys added by teleport connect to ~/.gnupg/private-keys-v1.d/ seem to have a length of only 1024 bits, which might be related to the issue:

wc ~/.gnupg/private-keys-v1.d/*
      32      40    2262 /Users/username/.gnupg/private-keys-v1.d/02C7149F9848E8E8B31FF45BF606B09FE055D72C.key
      32      40    2266 /Users/username/.gnupg/private-keys-v1.d/14DC5FA73FE0A29CF0254CB0D07874E732DE9979.key
      17      24    1174 /Users/username/.gnupg/private-keys-v1.d/239393F0CFF1FF4693C18238330EDF046AC1D1FC.key
      17      24    1174 /Users/username/.gnupg/private-keys-v1.d/2BBD2A9775E18D657DF3CBD31641EF2D2D60115D.key
      17      24    1174 /Users/username/.gnupg/private-keys-v1.d/4E6C9CE723CD0DD11E16077EA6C35E990CAFD76A.key
      17      24    1174 /Users/username/.gnupg/private-keys-v1.d/90589F05018180CA70A80C8E3AEC0ABB513B0CDC.key
      17      24    1174 /Users/username/.gnupg/private-keys-v1.d/A6D1F7B1B5A1AE187C0BDA12D04E8C14E1E91F47.key
      32      40    2266 /Users/username/.gnupg/private-keys-v1.d/C42AC1C46F1D4E211C735CC7DFAD4FF8391110E9.key
      17      24    1174 /Users/username/.gnupg/private-keys-v1.d/F28808D6BC7B8C8E484EB4F4F41DAB0D3886BECA.key

tsh works just fine

Version numbers

OS: MacOS Sonoma 14.7
Teleport Connect v15.2.5
SSH OpenSSH_9.7p1, LibreSSL 3.3.6
gpg: (GnuPG/MacGPG2) 2.2.32 libgcrypt 1.8.8

[ravicious]

@santiago-mooser Thanks for a comprehensive report. Could you check if connecting to a node through tsh ssh with the --forward-agent flag also breaks your setup?

We think it's because Teleport Connect adds that flag to all SSH connections made from Connect. We're thinking of making it a configurable option in v16.4.x and v15.4.x and then turning it into an opt-in rather than an opt-out setting in v17.

[santiago-mooser]

We don't currently have any SSH servers set up (only web applications, EKS and DBs), but we're adding the servers this month so I'll let you know as soon as onboard them to teleport

It would be great if we could toggle it from the UI! This would fix the issue for all of our devs.

[ravicious]

Thanks for clarifying, it seems that I've made a mistake. Both you and another user on Zendesk who made a similar report did not really need to SSH into a node to trigger this problem. So I'm not sure how much this is related to SSH agent forwarding.

Though there is yet another issue, #11662, which I think describes the same problem that you're having where accessing a node with agent forwarding is one of the repro steps.

Alas, I suspect it might be due to tsh (and thus Connect) incorrectly guessing that your agent supports SSH certs, basically replicating the original problem described in this issue.

When you run into the problem with git fetch, what is your $SSH_AUTH_SOCK at that point? tsh checks if it contains gpg-agent when deciding whether to add keys to the agent or not.

[santiago-mooser]

Yes, it seems like tsh is guessing that the agent supports SSH certs. I think there's also a restriction on MacOS specifically about the key length (it only supports >=2048 bit keys nad I think the SSH certs added by teleport are 1028 bits?).

I also asked the developer what his SSH_AUTH_SOCK was and he replied that this was what whs being used:
/private/tmp/com.apple.launchd.2ErIMl6caK/Listeners

I've since asked the developer to add export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket) to their rc file so let's see if this fixes the issue since that should change the variable to be a more standard /Users/username/.gnupg/S.gpg-agent.ssh

Any chance there's any updates on adding the --no-use-local-ssh-agent to teleport connect?

[ravicious]

I also asked the developer what his SSH_AUTH_SOCK was and he replied that this was what whs being used: /private/tmp/com.apple.launchd.2ErIMl6caK/Listeners

I've since asked the developer to add export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket) to their rc file so let's see if this fixes the issue since that should change the variable to be a more standard /Users/username/.gnupg/S.gpg-agent.ssh

Thanks, please do let us know if this had any effect, it should help with fixing the issue here.

Any chance there's any updates on adding the --no-use-local-ssh-agent to teleport connect?

I just pushed a PR that adds sshAgent.addKeysToAgent to Connect (#47270) which is the equivalent of the --add-keys-to-agent flag from tsh which superseded --use-local-ssh-agent and --no-use-local-ssh-agent. Once it's released, setting this option to "no" should serve as a workaround for this issue.

---

That being said, I still don't understand how exactly tsh manages to break gpg-agent. If SSH_AUTH_SOCK doesn't point to gpg-agent, then I assume this means that gpg-agent is used only by git. If gpg-agent is used only by git, then I don't know how tsh still manages to write stuff to ~/.gnupg/private-keys-v1.d/.

Our logic of looking at SSH_AUTH_SOCK to check if it contains gpg-agent seems mostly correct. This assumes that the user set up gpg-agent globally by doing some equivalent of gpg-agent --daemon --enable-ssh-support --sh followed by setting the env variables returned by this command. On macOS, there are several caveats regarding desktop apps since those do not read from shell files.

[santiago-mooser]

Thank you so much! This fixed the issue for our devs