Packer v1.8.0 can't SSH to AWS Ubuntu 22.04

[marcosdiez]

packer version:
Packer v1.8.0 (running on ubuntu-20.04 AMD64)

command: PACKER_LOG=1 packer build -on-error=ask ubuntu-22.04.json
(ubuntu-22.04.json is in the bottom of this github issue)

trying to use packer with Ubuntu 22.04 AWS AMI, which was released today, does not work. It can't ssh:

==> edr-ubuntu-22.04-: Waiting for SSH to become available...
2022/04/21 15:37:22 packer-builder-amazon-ebs plugin: [INFO] Waiting for SSH, up to timeout: 5m0s
2022/04/21 15:37:22 packer-builder-amazon-ebs plugin: Using host value: 172.19.17.149
2022/04/21 15:37:37 packer-builder-amazon-ebs plugin: [DEBUG] TCP connection to SSH ip/port failed: dial tcp 172.19.17.149:22: i/o timeout
2022/04/21 15:37:42 packer-builder-amazon-ebs plugin: Using host value: 172.19.17.149
2022/04/21 15:37:42 packer-builder-amazon-ebs plugin: [INFO] Attempting SSH connection to 172.19.17.149:22...
2022/04/21 15:37:42 packer-builder-amazon-ebs plugin: [DEBUG] reconnecting to TCP connection for SSH
2022/04/21 15:37:43 packer-builder-amazon-ebs plugin: [DEBUG] handshaking with SSH
2022/04/21 15:37:44 packer-builder-amazon-ebs plugin: [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2022/04/21 15:37:44 packer-builder-amazon-ebs plugin: [DEBUG] Detected authentication error. Increasing handshake attempts.

from my terminal it works:

mdiez@batman:~$ ssh -v ubuntu@172.19.17.149 -i ~/.ssh/id_rsa_terraform
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/mdiez/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: auto-mux: Trying existing master
debug1: Control socket "/tmp/xmdiez-ssh-ubuntu@172.19.17.149:22.sock" does not exist
debug1: Connecting to 172.19.17.149 [172.19.17.149] port 22.
debug1: Connection established.
debug1: identity file /home/mdiez/.ssh/id_rsa_terraform type 0
debug1: identity file /home/mdiez/.ssh/id_rsa_terraform-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3
debug1: match: OpenSSH_8.9p1 Ubuntu-3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.19.17.149:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:RuS0DVs/kq2OiVP/4bCe6YDdzd7Zr16Zyh/GlaQbr44
debug1: Host '172.19.17.149' is known and matches the ECDSA host key.
debug1: Found key in /home/mdiez/.ssh/known_hosts:1824
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/mdiez/.ssh/id_rsa RSA SHA256:bPGRz/lG4uVKTPZxZStRlksQqjhtzg205sax/VoQaNM agent
debug1: Will attempt key: /home/mdiez/.ssh/id_rsa_terraform RSA SHA256:1b9TeR6gKb6YXQSyMUhftAhle6u4M1cEWtb6Mg4JwRU explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug1: kex_input_ext_info: publickey-hostbound@openssh.com (unrecognised)
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/mdiez/.ssh/id_rsa RSA SHA256:bPGRz/lG4uVKTPZxZStRlksQqjhtzg205sax/VoQaNM agent
debug1: Authentications that can continue: publickey
debug1: Offering public key: /home/mdiez/.ssh/id_rsa_terraform RSA SHA256:1b9TeR6gKb6YXQSyMUhftAhle6u4M1cEWtb6Mg4JwRU explicit
debug1: Server accepts key: /home/mdiez/.ssh/id_rsa_terraform RSA SHA256:1b9TeR6gKb6YXQSyMUhftAhle6u4M1cEWtb6Mg4JwRU explicit
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).
Authenticated to 172.19.17.149 ([172.19.17.149]:22).
debug1: setting up multiplex master socket
debug1: channel 0: new [/tmp/xmdiez-ssh-ubuntu@172.19.17.149:22.sock]
debug1: channel 1: new [client-session]
debug1: Entering interactive session.
debug1: pledge: id
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Remote: /home/ubuntu/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /home/ubuntu/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending environment.
Welcome to Ubuntu 22.04 LTS (GNU/Linux 5.15.0-1004-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Apr 21 18:39:56 UTC 2022

  System load:  0.03271484375     Processes:             111
  Usage of /:   18.9% of 7.58GB   Users logged in:       0
  Memory usage: 2%                IPv4 address for ens5: 172.19.17.149
  Swap usage:   0%


0 updates can be applied immediately.


Last login: Thu Apr 21 18:39:58 2022 from 10.26.41.70
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@ip-172-19-17-149:~$

This is the packer settings file ubuntu-22.04.json

{
    "variables": {
      "HOME": "{{env `HOME`}}",
      "TSTAMP": "{{env `TSTAMP`}}"
    },
    "builders": [
      {
        "name": "ubuntu-22.04-{{user `TSTAMP`}}",
        "type": "amazon-ebs",
        "region": "us-east-1",
        "source_ami_filter": {
            "filters": {
                "virtualization-type": "hvm",
                "name": "ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*",
                "root-device-type": "ebs"
            },
            "owners": [
                "099720109477"
            ],
            "most_recent": true
        },
        "instance_type": "t3.large",
        "subnet_id": "subnet-XXXXXXXXXX",
        "vpc_id": "vpc-XXXXXXXXXX",
        "ssh_username": "ubuntu",
        "ami_name": "ubuntu-22.04-{{user `TSTAMP`}}",
        "ami_description": "Ubuntu 22.04 GoldenImage {{user `TSTAMP`}}",
        "ssh_keypair_name": "id_rsa_terraform",
        "ssh_private_key_file": "{{user `HOME`}}/.ssh/id_rsa_terraform.pem"
      }
    ],
    "provisioners": [
      {
        "type": "shell",
        "inline": [
          "echo hello world"
        ]
      }
    ]
  }

[mhahl]

Could you try one of the workarounds from here? #11656

[marcosdiez]

This workaround worked: #11656 (comment)

I'll keep the issue open because I still can ssh with my normal keys and packer can't.
Thank you!

[lorengordon]

Probably same root cause as #8609

[FelicianoTech]

For Ubuntu 22.04, OpenSSH was updated to v8.x and rsa host keys are disabled by default. Either a client key using ecc needs to be used, or my temp workaround, was to renable rsa on the host side.

[asottile]

at least for amazon-ebs I was able to get this working by using "temporary_key_pair_type": "ed25519", in my builder configuration

[sc250024]

Not working though for local ISO builds. See my comment here: #11656 (comment)

[sc250024]

I also tried this with the latest Nightly build (https://github.com/hashicorp/packer/releases/tag/nightly) as of today, and it's still the same problem with v1.8.1-dev.

[yukoba]

This problem comes from here. golang/go#49952

[sc250024]

@yukoba Thanks for posting that. I saw that issue in another thread, and it seems pretty core to Packer itself. I've tried to workaround this issue locally with virtualbox-iso builds, but unlike with the amazon-ebs builder, it seems that there's no workaround at the moment.

[sc250024]

I found a temporary workaround which should work for *-iso builds. I tested it on virtualbox-iso. I remembered that the https://github.com/chef/bento project already had a working Ubuntu 20.04 LTS build of the live version of their ISO. I was wondering how they were able to get theirs working.

This is the magic sauce: https://github.com/chef/bento/blob/118ad132f6bd7c09cbf40b4933281d32dfe139fe/packer_templates/ubuntu/http/user-data#L9-L11

Basically, you need to stop SSH during the Subiquity install process so that Packer doesn't freak out, and continues when SSH connectivity is restored.

[MichaelKorn]

Same issue in GCP with Ubuntu 22.04, but temporary_key_pair_type = "ed25519" works as workaround. Thanks for mention.

[voltagex]

Is it possible to get a new nightly release? I'm seeing this error with password auth as well.

[nywilken]

Thanks for all the help in communicating various workarounds. The latest Packer SDK has been patched with the Golang crypto/ssh fix . We will be rolling out the fixes to each individual plugin as the changes get merged.

Once all plugins have been updated we I’ll work to get a release of Packer core out with the update crypto fix. For those using HCL pinning each plugin to the latest available release should be enough to get the fix in place.

[sc250024]

Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.