SSH provisioner incompatible with Fedora 33, 34, 35 and CentOS Stream 9; also RHEL 8 with FIPS

[Ramblurr]

Fedora 33 updated the system-wide crypto policy to disallow SHA-1 hashes in signatures. This is a good thing, especially since OpenSSH follows suit as of version 8.8

The upstream golang issue is golang/go#39885

Terraform Version

Terraform v1.1.0
on linux_amd64

Terraform Configuration Files

resource "null_resource" "my_config" {

  provisioner "file" {
    content = file("${path.module}/test.config")
    destination = "/home/fedora/test.config"
  }

  connection {
    type     = "ssh"
    host     = "myhost.com"
    user     = "fedora"
    agent    = "true"
    timeout = "15s"
  }
}

Debug Output

...
2021-12-10T15:00:37.014+0100 [INFO]  Starting apply for null_resource.my_config
2021-12-10T15:00:37.014+0100 [DEBUG] null_resource.my_config: applying the planned Create change
null_resource.my_config: Provisioning with 'file'...
2021-12-10T15:00:37.047+0100 [DEBUG] Connecting to myhost.com:22 for SSH
2021-12-10T15:00:37.050+0100 [DEBUG] Connection established. Handshaking for user fedora
2021-12-10T15:00:37.094+0100 [WARN]  SSH authentication failed (fedora@myhost.com:22): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2021-12-10T15:00:37.094+0100 [WARN]  retryable error: SSH authentication failed (fedora@myhost.com:22): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
...

Expected Behavior

The file should be copied via ssh.

Actual Behavior

The file was not copied due to an ssh connection error

│ Error: file provisioner error
│ 
│   with null_resource.my_config,
│   on apps.tf line 106, in resource "null_resource" "my_config":
│  106:   provisioner "file" {
│ 
│ timeout - last error: SSH authentication failed (fedora@myhost.com:22): ssh: handshake failed:
│ ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
╵
Releasing state lock. This may take a few moments...

Steps to Reproduce

1. Deploy a vanilla Fedora 33, 34, 35 or CentOS 9 Stream box OR a RHEL8 box with update-crypto-policies --set FIPS
2. use the null_resource + file provisioner with ssh settings to copy a file to the host

Additional Context

The workaround is to enable the insecure ssh-rsa key type:

# place this file on the server at /etc/ssh/sshd_config.d/10-insecure-rsa-keysig.conf
PubkeyAcceptedKeyTypes=+ssh-rsa

then systemctl reload sshd

References

There are a host of related issues about this across the go ecosystem:

Related issues:

• hashicorp/packer: ssh handshake failure after changing system-wide crypto policy to FIPS packer#8609
• golang/go: x/crypto/ssh: publicKeyCallback cannot handshake using ssh-rsa keys signed using the ssh-rsa-sha2-256 algorithm golang/go#39885
• golang/go:x/crypto/ssh: cannot sign certificate with different algorithm golang/go#36261
• golang/go:x/crypto/ssh: support RSA SHA-2 host key signatures golang/go#37278

Unless hashicorp wants to maintain a forked version of the golang lib, then there's not much to do but wait. I've opened this ticket to track the issue.

[alisdair]

Thanks @Ramblurr for this detailed issue report! We're tracking this at #27768 also, and once this is resolved upstream we can fix the issue in Terraform too.

[Ramblurr]

Ah, sorry for the duplicate. I did search beforehand, but not well enough it seems :(

I was surprised that it hadn't been reported given F33's age.

Feel free to close this one if you want.

[rogierm]

This is also the case with Amazon Linux 2022 as that is derived from Fedora.

2022/01/14 22:58:26 packer-builder-amazon-ebs plugin: Using host value: 10.X.X.X
2022/01/14 22:58:26 packer-builder-amazon-ebs plugin: [INFO] Attempting SSH connection to 10.X.X.X:22...
2022/01/14 22:58:26 packer-builder-amazon-ebs plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/01/14 22:58:26 packer-builder-amazon-ebs plugin: [DEBUG] handshaking with SSH
2022/01/14 22:58:27 packer-builder-amazon-ebs plugin: [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2022/01/14 22:58:27 packer-builder-amazon-ebs plugin: [DEBUG] Detected authentication error. Increasing handshake attempts.

[Shilpi-Das1]

Is there any update on this issue

[crw]

Per #30134 (comment), it looks like the upstream issue is not yet resolved.

[Miosame]

I believe golang/go#39885 was mentioned somewhere along the lines as a blocker for this issue, which now has been closed - however the above comments mention a "closing as duplicate" loop linking to each other, which issue is the one to follow now that the golang upstream closed it out? Thanks!

[crw]

We are still tracking this dependency: golang/go#49952

I believe that is the last puzzle piece before this issue can be resolved.

[adarobin]

@crw the only outstanding issue on golang/go#49952 appears to relate to the Go SSH server implementation. The Go SSH client implementation is fixed? That's how it reads to me at least.

[crw]

Just re-read the internal conversation. We are waiting for the entire project as tracked in that ticket to be completed before updating. You may be correct that the relevant client bits are completed. In this case, we are erring on the side of caution.

[lorengordon]

Considering how slowly the golang project is moving to fix this issue fully, and the ssh provisioner is only using the client-side code, can we please reconsider the decision to wait? At least pull it in and test it. If it works, great! If it doesn't, now you have a branch with the test code.

[soakes]

Just for completeness, this issue is also present when using Terraform to deploy the recently released Ubuntu 22.04 templates when using the SSH provisioner.

[crw]

These changes should be visible in the next 1.2 build. Thanks for the on-going feedback, we do appreciate it!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.