SSH Authentication issue with Linux AMIs

[peishuli]

Overview of the Issue

When building AWS Linux AMIs using packer, we encountered the following error:
==> amazon-ebs.NewRez-us-east-1: Error waiting for SSH: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Reproduction Steps

Run the following command against packer config (e.g., the folder containing main.pkr.hcl file):

packer build .
v1.7.10

Plugin and Packer version

v1.0.9

Simplified Packer Buildfile

If the file is longer than a few dozen lines, please include the URL to the
source "amazon-ebs" "NewRez-us-east-1" {
ami_name = "${var.ami_name}${var.build_date}"
associate_public_ip_address = "false"
instance_type = var.instance_type
security_group_ids = var.sg_ids
temporary_iam_instance_profile_policy_document {
Statement {
Action = [
"secretsmanager:GetSecretValue",
"kms:Decrypt",
"s3:ListBucket",
"s3:GetObject"
]
Effect = "Allow"
Resource = ["*"]
}
Version = "2012-10-17"
}
ami_regions = ["us-east-1", "us-west-2"]
region = "us-east-1"
region_kms_key_ids = {
us-east-1 = var.kms_key
us-west-2 = var.west_kms_key
}
source_ami_filter {
filters = {
name = "AWS_Linux2_Golden_AMI*"
}
most_recent = true
owners = ["self"]
}

ssh_timeout = "7m"
ssh_username = "ec2-user"
ssh_interface = "private_ip"
subnet_id = var.subnet_id
vpc_id = var.vpc_id
}

Operating system and Environment details

Amazon Linux

Log Fragments and crash.log files

INFO 2022-04-15 12:09:10 ==> amazon-ebs.NewRez-us-east-1: Waiting for instance (i-03c45eb95b4aa8cb5) to become ready...
ERROR 2022-04-15 12:09:40 2022/04/15 17:09:39 packer-plugin-amazon_v1.0.9_x5.0_linux_amd64 plugin: 2022/04/15 17:09:39 [INFO] Not using winrm communicator, skipping get password...
INFO 2022-04-15 12:09:40 ==> amazon-ebs.NewRez-us-east-1: Using SSH communicator to connect: 10.63.32.167
ERROR 2022-04-15 12:09:41 2022/04/15 17:09:39 packer-plugin-amazon_v1.0.9_x5.0_linux_amd64 plugin: 2022/04/15 17:09:39 [INFO] Waiting for SSH, up to timeout: 7m0s
ERROR 2022-04-15 12:09:41 2022/04/15 17:09:39 packer-plugin-amazon_v1.0.9_x5.0_linux_amd64 plugin: 2022/04/15 17:09:39 Using host value: 10.63.32.167
INFO 2022-04-15 12:09:41 ==> amazon-ebs.NewRez-us-east-1: Waiting for SSH to become available...
ERROR 2022-04-15 12:09:55 2022/04/15 17:09:54 packer-plugin-amazon_v1.0.9_x5.0_linux_amd64 plugin: 2022/04/15 17:09:54 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 10.63.32.167:22: i/o timeout
ERROR 2022-04-15 12:10:00 2022/04/15 17:09:59 packer-plugin-amazon_v1.0.9_x5.0_linux_amd64 plugin: 2022/04/15 17:09:59 Using host value: 10.63.32.167
ERROR 2022-04-15 12:10:00 2022/04/15 17:09:59 packer-plugin-amazon_v1.0.9_x5.0_linux_amd64 plugin: 2022/04/15 17:09:59 [INFO] Attempting SSH connection to 10.63.32.167:22...
ERROR 2022-04-15 12:10:00 2022/04/15 17:09:59 packer-plugin-amazon_v1.0.9_x5.0_linux_amd64 plugin: 2022/04/15 17:09:59 [DEBUG] reconnecting to TCP connection for SSH
ERROR 2022-04-15 12:10:00 2022/04/15 17:09:59 packer-plugin-amazon_v1.0.9_x5.0_linux_amd64 plugin: 2022/04/15 17:09:59 [DEBUG] handshaking with SSH
ERROR 2022-04-15 12:10:00
......
ERROR 2022-04-15 12:11:05 2022/04/15 17:11:03 packer-plugin-amazon_v1.0.9_x5.0_linux_amd64 plugin: 2022/04/15 17:11:03 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
ERROR 2022-04-15 12:11:05 2022/04/15 17:11:03 packer-plugin-amazon_v1.0.9_x5.0_linux_amd64 plugin: 2022/04/15 17:11:03 [DEBUG] Detected authentication error. Increasing handshake attempts.
INFO 2022-04-15 12:11:05 ==> amazon-ebs.NewRez-us-east-1: Error waiting for SSH: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Additional Contexts

We have been using essentially the same packer build configuration for all of our Linux AMIs without issues until recently. Now all of them have the same ssh authentication problem. Our Windows AMIs (still) work just fine.

[nywilken]

Thanks for all the help in communicating various workarounds. The latest Packer SDK has been patched with the Golang crypto/ssh fix . We will be rolling out the fixes to each individual plugin as the changes get merged.

Once all plugins have been updated we I’ll work to get a release of Packer core out with the update crypto fix. For those using HCL pinning each plugin to the latest available release should be enough to get the fix in place.