SSH certificate authority support

[Okeanos]

Feature Request

Environment

What hardware/cloud provider/hypervisor is being used to run Ignition?

Any

Desired Feature

I would very much like it if ignition (and subsequently fcct/Fedora CoreOS) would natively support SSH certificate authorities. Currently the ignition spec supports a security declaration with custom certificate authorities for tls as well as a user specific declaration for individual sshAuthorizedKeys. Sadly, there is no built in way to roll out a SSH CA onto a machine with ignition.

Having the ability to roll out an SSH certificate authority with ignition would greatly simplify SSH key management and deployment of machines (especially with the same canonical name). The dreaded host key verification dialog would thus be a thing of the past because a client could verify authenticity of the remote machine by virtue of the given certificate signed by a mutually trusted CA.

Other Information

This might be slightly icky because the host keys would have to be supplied alongside the host certificates via ignition because I don't see that injecting the signing authority certificate into the boot process to sign randomly generated certificates is a good idea.

For more information SSH CAs see e.g.:

• https://docs.fedoraproject.org/en-US/fedora/rawhide/system-administrators-guide/infrastructure-services/OpenSSH/#sec-Using_OpenSSH_Certificate_Authentication
• https://www.lorier.net/docs/ssh-ca.html
• https://jameshfisher.com/2018/03/16/how-to-create-an-ssh-certificate-authority/
• https://smallstep.com/blog/use-ssh-certificates/
• https://ibug.io/blog/2019/12/manage-servers-with-ssh-ca/

[bgilbert]

You can use Ignition's storage.files section to write SSH host keys and sshd configuration. For the latter, you can append to /etc/ssh/sshd_config (not recommended) or, with newer versions of sshd and appropriate distro-side configuration (e.g. on Fedora CoreOS), you can write a config fragment into /etc/ssh/sshd_config.d/*.conf. It might be useful to add FCCT sugar which does this configuration for you; what config directives would you like to see?

(We usually support specific host features via FCCT sugar, rather than directly in Ignition. The two cases you mentioned are special: ignition.security.tls.certificateAuthorities section affects the CAs Ignition itself uses for HTTPS connections, and sshAuthorizedKeys is a convenience for writing keys to individual user directories with appropriate permissions.)

[Okeanos]

You note concerning FCCT sugars makes sense and didn't even occur to me when I wrote the ticket, thanks.

Concerning the missing (so to speak) directives to conveniently support SSH CAs I suppose something like the following might make sense (using fcct wording):

ignition:
  config:
    security:
      ssh:
        certificate_authority:
          # This entry should create a matching "TrustedUserCAKeys /etc/ssh/<NAME>.pub" in /etc/ssh/sshd_config and also create the file at "/etc/ssh/<NAME>.pub"
          authority:
            name: # string
            certificate: # one of compression, source, inline, local
          host_keys: # Each entry here should ceate a matching "HostCertificate /etc/ssh/ssh_host_<TYPE>_key-cert.pub" entry in /etc/ssh/sshd_config I believe
            - type: rsa # (ssh key types)
              key: 
                contents: # one of compression, source, inline, local
              certificate:
                contents: # one of compression, source, inline, local

There's a couple of other attributes etc. (such as AuthorizedPrincipalsFile and RevokedKeys) but I don't have the time right now to give a "full" spec for that … I can try to improve the example above later on if it is of interest to you.

[bgilbert]

Yes, it'd be helpful if you can provide as complete an example as possible. Thank you!

We'd normally put sugar for specific services into its own top-level struct, e.g. ssh.

[Okeanos]

I'd personally put this into the security map next to TLS because this touches base line security for the machine, i.e. who can access it and which (SSH) CAs to trust but in the end it's up to you I guess ;-) .

Okay, I am 95% confident that the following should work as expected:

ignition:
  config:
    security:
      ssh:
        certificate_authority:
          user_authorities: # Create "TrustedUserCAKeys /etc/ssh/ssh_user_ca" in /etc/ssh/sshd_config and also create the file at "/etc/ssh/ssh_user_ca"
                            # create one entry per certificate listed below
            - certificate:
                contents: # like file.contents: one of compression, source, inline, local (the public certificate of the CA to be trusted)
          host_keys: # Create "HostCertificate /etc/ssh/ssh_host_<TYPE>_key-cert.pub" entry in /etc/ssh/sshd_config for each provided host_key
                     # this also needs to create a "HostKey /etc/ssh/ssh_host_<TYPE>_key" enty /etc/ssh/sshd_config for each provided host_key overriding the defaults
            - type: rsa # string (ssh key types such as rsa, ed25519, ecdsa)
              key:
                contents: # like file.contents: one of compression, source, inline, local
              pub: # only relevant assuming ign/fcc shouldn't re-calculate the public keey during transformation
                   # e.g. using "ssh-keygen -y -f ~/.ssh/id_rsa > ~/.ssh/id_rsa.pub"
                contents: # like file.contents: one of compression, source, inline, local
              cert:
                contents: # like file.contents: one of compression, source, inline, local
          revoked_keys: # Create "RevokedKeys /etc/ssh/ssh_user_ca_revoked_keys" in /etc/ssh/sshd_config and also create the file "/etc/ssh/ssh_user_ca_revoked_keys"
                        # containing the list of public keys below in the format of "@revoked <public_key>"
            - certficate:
                contents: # like file.contents: one of compression, source, inline, local
    passwd:
      users:
        - name: core
          ssh_authorized_principales: # Generates a file at %h/.ssh/authorized_principals containing the list of principals below
                                      # also has to add AuthorizedPrincipalsFile %h/.ssh/authorized_principals in /etc/ssh/sshd_config
            - # string

Instead of modifying the /etc/ssh/sshd_config directly you probably want to specify two files in /etc/ssh/sshd_config.d/ based on the above config example (ALL values specified) and CoreOS defaults:

40-host-certificates

HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub

40-user-certificates

TrustedUserCAKeys /etc/ssh/ssh_user_ca
AuthorizedPrincipalsFile %h/.ssh/authorized_principals

RevokedKeys /etc/ssh/ssh_user_ca_revoked_keys

A local test run using these to files using latest CoreOS worked perfectly fine for me.

[bgilbert]

Awesome, thanks for the detailed proposal!

I'd personally put this into the security map next to TLS because this touches base line security for the machine

The ignition section doesn't actually affect the machine at all; it only affects the behavior of Ignition itself.

[Okeanos]

That totally makes sense … may bad. I used my initial example to continue working on and just quickly glanced at the fcct spec and must have mistook the indention level for ignition.config.*.

variant: fcos
version: x.y.z
ssh: # new stuff here
        # and here
passwd:
  users:
    - name: core
      ssh_authorized_principales:

Would be the correct indention and keying of properties, right?

[bgilbert]

Yup, that's right.

[Okeanos]

So, after sleeping and looking at this again noticed a small design issue concerning the ssh_authorized_principales and where they live. My original proposal would put them into ~/.ssh/authorized_principals – which means any client being able log into that user with a matching principle is in a position to change that setting and which principles are allowed to log in.

Because using authorized principles is no longer key based (as in a particular, personal key has to be used) but "role" based (the key has to be signed to be used in this capacity) I think it might be worthwhile to do the following instead:

40-user-certificates

#...
AuthorizedPrincipalsFile /etc/ssh/authorized_principals/%u

Which would mean that any passwd.user entry with ssh_authorized_principales specified would create a new file at /etc/ssh/authorized_principals/<NAME> filled line by line with the authorized principles instead and prevent regular users without sudo from messing with it.

By the way technically the same issue can happen for ssh_authorized_keys but in that case I am not sure that should be changed even though that is possible via sshd and the AuthorizedKeysFile directive. At the very least it would also have to take into account the settings in /etc/ssh/sshd_config.d/40-ssh-key-dir.conf 🤷‍♂️.

[Okeanos]

I had some time to poke around the Go code and also had another look at the original draft for the yaml spec and simplified it somewhat. I may give it a try to implement but the whole translate logic seemed somewhat arcane to me at first glance and my Go knowledge isn't that good.
I assume by "FCCT sugar" you meant this whole block would probably (?) behave like a wrapper around the file translation layer and implicitly create storage.files objects behind the scenes.

I am still unsure about the principals file location and I am wondering whether moving the attribute from passwd.users to ssh.authorized_principals.users would be better, i.e.:

ssh:
  authorized_principals:
    - user: <name>
      principals:
      - # string

This would encapsulate all new SSH settings in one block and also let AuthorizedPrincipalsFile /etc/ssh/authorized_principals/%u make more sense.

ssh:
  certificate_authorities: # Create "TrustedUserCAKeys /etc/ssh/ssh_user_ca" in /etc/ssh/sshd_config and also create the file at "/etc/ssh/ssh_user_cas"
                           # create one entry per SSH key listed below
    - contents: # like file.contents: one of compression, source, inline, local (the public key of the CA to be trusted)
  host_keys: # Create "HostCertificate /etc/ssh/ssh_host_<TYPE>_key-cert.pub" entry in /etc/ssh/sshd_config for each provided host_key
             # this also needs to create a "HostKey /etc/ssh/ssh_host_<TYPE>_key" enty /etc/ssh/sshd_config for each provided host_key overriding the defaults
    - type: <type> # string (ssh key types such as rsa, ed25519, ecdsa)
      key:
        contents: # like file.contents: one of compression, source, inline, local
      pub: # only relevant assuming ign/fcc shouldn't re-calculate the public keey during transformation
           # e.g. using "ssh-keygen -y -f ~/.ssh/id_rsa > ~/.ssh/id_rsa.pub"
        contents: # like file.contents: one of compression, source, inline, local
      cert:
        contents: # like file.contents: one of compression, source, inline, local
  revoked_keys: # Create "RevokedKeys /etc/ssh/ssh_user_revoked_keys" in /etc/ssh/sshd_config and also create the file "/etc/ssh/ssh_user_revoked_keys"
                # containing the list of public keys below in the format of "@revoked <public_key>"
    - contents: # like file.contents: one of compression, source, inline, local
passwd:
  users:
    - name: <name>
      ssh_authorized_principals: # Generates a file at %h/.ssh/authorized_principals containing the list of principals below
                                  # also has to add AuthorizedPrincipalsFile %h/.ssh/authorized_principals in /etc/ssh/sshd_config
      - # string

[Okeanos]

In before comment spam 🤷‍♂️ – sorry.

Had some time to verify the concept sans butane sugar actually works as desired:

name='fcos-ssh-certs'
buInc=$(realpath ./ssh)
ca=$(realpath ./ca)

ssh-keygen -t ecdsa -N "" -f "${buInc}/ssh_host_ecdsa_key" -C "${name}"
ssh-keygen -t ed25519 -N "" -f "${buInc}/ssh_host_ed25519_key" -C "${name}"
ssh-keygen -t rsa -N "" -f "${buInc}/ssh_host_rsa_key" -C "${name}"

ssh-keygen -s "${ca}" \
     -I "${name} host key" \
     -n "${name},${name}.local" \
     -V -5m:+3650d \
     -h \
     "${buInc}/ssh_host_ecdsa_key" \
     "${buInc}/ssh_host_ed25519_key" \
     "${buInc}/ssh_host_rsa_key"

ign_config=$(butane --strict --files-dir="${buInc}"  ssh-certs.bu | base64 | tr -d '\n')

HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub

variant: fcos
version: 1.3.0
storage:
  files:
    - path: /etc/hostname
      mode: 420
      overwrite: true
      contents:
        inline: |
          fcos-ssh-certs
    - path: /etc/ssh/sshd_config.d/40-host-certificates.conf
      mode: 0600
      append:
        - local: /sshd_config.d/40-host-certificates.conf
    - path: /etc/ssh/ssh_host_ecdsa_key
      mode: 0600
      append:
        - local: ssh_host_ecdsa_key
    - path: /etc/ssh/ssh_host_ecdsa_key-cert.pub
      mode: 0644
      append:
        - local: ssh_host_ecdsa_key-cert.pub
    - path: /etc/ssh/ssh_host_ecdsa_key.pub
      mode: 0644
      append:
        - local: ssh_host_ecdsa_key.pub
    - path: /etc/ssh/ssh_host_ed25519_key
      mode: 0600
      append:
        - local: ssh_host_ed25519_key
    - path: /etc/ssh/ssh_host_ed25519_key-cert.pub
      mode: 0644
      append:
        - local: ssh_host_ed25519_key-cert.pub
    - path: /etc/ssh/ssh_host_ed25519_key.pub
      mode: 0644
      append:
        - local: ssh_host_ed25519_key.pub
    - path: /etc/ssh/ssh_host_rsa_key
      mode: 0600
      append:
        - local: ssh_host_rsa_key
    - path: /etc/ssh/ssh_host_rsa_key-cert.pub
      mode: 0644
      append:
        - local: ssh_host_rsa_key-cert.pub
    - path: /etc/ssh/ssh_host_rsa_key.pub
      mode: 0644
      append:
        - local: ssh_host_rsa_key.pub
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-ed25519 <snip>

$ ssh -v core@fcos-ssh-certs
#...
debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com SHA256:..., serial 0 ID "fcos-ssh-certs host key" CA ssh-rsa ... valid from 2021-04-21T18:38:36 to 2031-04-19T18:43:36
#...

🥳

[dcode]

Not to throw a wrench in the proposal, as I think ssh certificates would be a great option to support, this approach kinda flies in the face of best practices. In production environments at least, private keys should be generated on the box and never leave. So, instead, and I understand this isn't as ideal, the host should generate the keys, use information in the certificate_authorities and a host_certificate stanza to submit a certificate signing request to the CA, possibly using the ACME protocol (same protocol used by let's encrypt). This way, host secrets are never leaked are are ephemeral to the lifetime of the host.

I found this issue because I was looking to solve the same problem, but with generic TLS certificates. For my use case, I will likely implement something like step-ca and a minimal letsencrypt client in a container after network-online.target is reached.

[Okeanos]

That's a very fair point and Step-Ca or HashiCorp Vault (with its client + SSH Engine) are also on my own radar (and have been for a while). This is one of the reasons I have not invested any time beyond the things written here. Other than that I tried to present a collection of "working" things to prevent a Wisdom of the Ancients situation.

At the time I wrote this proposal + workarounds above I was not very familiar with step-ca/vault as options nor was I interested in the perfect solution … more of any (better) solution than what exists out of the box.