Failed to destroy AWS node with volume: * aws_volume_attachment.jenkins_disk_attachment: Error waiting for Volume (vol-XXXX) to detach from Instance: i-XXXXX

[evgeniagusakova]

Hi Team!

I faced with issue when I'm not able to destroy instance with volume created by terraform

Terraform Version

bash-4.3$ uname  -s
Darwin

bash-4.3$ terraform version
Terraform v0.10.6

Issue Explanation

I created instance and attached pre-created volume with the following code:
(all code not related to issue like sec. groups, EIP etc are skipped)

data "aws_ebs_volume" "jenkins_disk" {
  most_recent = true

  filter {
    name   = "volume-id"
    values = ["vol-XXXXX"]
  }
}
resource "aws_volume_attachment" "jenkins_disk_attachment" {
  device_name = "/dev/sdx"
  volume_id   = "${data.aws_ebs_volume.jenkins_disk.id}"
  instance_id = "${aws_instance.jenkins_server.id}"
}

resource "aws_instance" "jenkins_server" {
  connection {
    type        = "ssh"
    user        = "${var.instance_user}"
    host        = "${aws_instance.jenkins_server.public_ip}"
    private_key = "${file("${var.jenkins_server_ssh_key_path}")}"
  }
...
}

And got an error when I tried to do terraform destroy

terraform  destroy   -var secret_key=<KEY> -var access_key=<KEY>
aws_security_group.jenkins_server_security_group: Refreshing state... (ID: sg-XXXXX)
<other resources here>
data.aws_ebs_volume.jenkins_disk: Refreshing state...
<other resources here>
aws_instance.jenkins_server: Refreshing state... (ID: i-XXXXX)
aws_volume_attachment.jenkins_disk_attachment: Refreshing state... (ID: vai-XXXXX)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:
<full and correct list of resources to delete here>
Plan: 0 to add, 0 to change, 12 to destroy.
Do you really want to destroy?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.
  Enter a value: yes

<deleting resource>
aws_volume_attachment.jenkins_disk_attachment: Destroying... (ID: vai-XXXXX)
<deleting resources>
aws_volume_attachment.jenkins_disk_attachment: Still destroying... (ID: vai-XXXXX, 10s elapsed)
aws_volume_attachment.jenkins_disk_attachment: Still destroying... (ID: vai-XXXXX, 20s elapsed)
aws_volume_attachment.jenkins_disk_attachment: Still destroying... (ID: vai-XXXXX, 30s elapsed)
aws_volume_attachment.jenkins_disk_attachment: Still destroying... (ID: vai-XXXXX, 40s elapsed)
Error applying plan:
1 error(s) occurred:
* aws_volume_attachment.jenkins_disk_attachment (destroy): 1 error(s) occurred:
* aws_volume_attachment.jenkins_disk_attachment: Error waiting for Volume (vol-XXXXX) to detach from Instance: i-XXXXX

Debug Output

Expected Behavior

Instance is deleted, volume is deattached but not deleted.

Actual Behavior

EIP is deattached
Sec. Groups are not deleted
Instance is not deleted
Volume is not deattaced

Workaround

on node:

1. Stop all processes using disk mount point (lsof helps to find all processes)
2. Sync disk and unmount it
3. Deattach volume using amazon console
4. Remove attachment from state terraform state rm aws_volume_attachment.jenkins_disk_attachment
5. Destroy instance: terraform destroy -var ....

Additional details

I'm really sure that disk was not de-attached because it was mounted and in-use. From Amazon Console I was able to deattach it only after I did disk unmount.

But I expected Forced deattach from terraform after some (configurable?) timeout.

So if this is expected behaviour I guess we need to convert this bug report into feature request :)

References

Not sure but this issue looks close to my.
It was with old terraform version so I decided to create new issue.

[hashibot]

This issue has been automatically migrated to hashicorp/terraform-provider-aws#1991 because it looks like an issue with that provider. If you believe this is not an issue with the provider, please reply to this issue and let us know.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.