Destroying some Cloudfront certificates fail on first attempt

[ophintor]

When I run a terraform destroy I get the error below:

Error: Error applying plan:

2 error(s) occurred:

* module.event_query_service.aws_acm_certificate.reporting_domain (destroy): 1 error(s) occurred:

* aws_acm_certificate.reporting_domain: Error deleting certificate: ResourceInUseException: Certificate arn:aws:acm:us-east-1:353795026198:certificate/e5ade17b-fcf4-4da7-941c-d8c89b0dcbb3 in account 353795026198 is in use.
        status code: 400, request id: d67726cc-c7b9-11e8-9ddd-fb35ba3de3d5
* module.event_receipt_service.aws_acm_certificate.reporting_domain (destroy): 1 error(s) occurred:

* aws_acm_certificate.reporting_domain: Error deleting certificate: ResourceInUseException: Certificate arn:aws:acm:us-east-1:353795026198:certificate/242ff6fc-d918-41ad-a8cf-760e40fbfb26 in account 353795026198 is in use.
        status code: 400, request id: d5f37794-c7b9-11e8-9ddd-fb35ba3de3d5

If I run it again it deletes them properly:

...
module.event_query_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/e5ade17b-fcf4-4da7-941c-d8c89b0dcbb3, 50s elapsed)
module.event_receipt_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/242ff6fc-d918-41ad-a8cf-760e40fbfb26, 50s elapsed)
module.event_receipt_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/242ff6fc-d918-41ad-a8cf-760e40fbfb26, 1m0s elapsed)
module.event_query_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/e5ade17b-fcf4-4da7-941c-d8c89b0dcbb3, 1m0s elapsed)
module.event_receipt_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/242ff6fc-d918-41ad-a8cf-760e40fbfb26, 1m10s elapsed)
module.event_query_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/e5ade17b-fcf4-4da7-941c-d8c89b0dcbb3, 1m10s elapsed)
module.event_receipt_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/242ff6fc-d918-41ad-a8cf-760e40fbfb26, 1m20s elapsed)
module.event_query_service.aws_acm_certificate.reporting_domain: Still destroying... (ID: arn:aws:acm:us-east-1:353795026198:cert...e/e5ade17b-fcf4-4da7-941c-d8c89b0dcbb3, 1m20s elapsed)
module.event_receipt_service.aws_acm_certificate.reporting_domain: Destruction complete after 1m21s
module.event_query_service.aws_acm_certificate.reporting_domain: Destruction complete after 1m21s

Destroy complete! Resources: 2 destroyed.

There are other modules with acm certificates that get deleted on the first go but for some reason the two above need terraform destroy to be run twice. The code for the acm certs is pretty much the same, but these are the only two certs used for API GWs:

resource "aws_acm_certificate" "reporting_domain" {
  provider          = "aws.us-east-1"
  domain_name       = "${format("%s.%s",local.reporting_component, local.reporting_domain)}"
  validation_method = "DNS"
  tags              = "${merge(var.tags, map("Environment", "${terraform.workspace}"))}"

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_api_gateway_domain_name" "reporting_service" {
  domain_name     = "${format("%s.%s",local.reporting_component, local.reporting_domain)}"
  certificate_arn = "${aws_acm_certificate.reporting_domain.arn}"
  depends_on      = ["aws_acm_certificate_validation.reporting_domain"]
}

Terraform and AWS providers are at the latest versions:

$ terraform -v
Terraform v0.11.8
+ provider.archive v1.1.0
+ provider.aws v1.39.0
+ provider.null v1.0.0
+ provider.random v2.0.0
+ provider.template v1.0.0
+ provider.tls v1.2.0

How can I troubleshoot this further? Many thanks.

PS. I am aware of this one (which sounds exactly the same as my problem) --> hashicorp/terraform-provider-aws#3866
But as mentioned above I am on newer versions and instead of ELBs the certs are used with API GWs.

Cheers.

[ghost]

This issue has been automatically migrated to hashicorp/terraform-provider-aws#6082 because it looks like an issue with that provider. If you believe this is not an issue with the provider, please reply to hashicorp/terraform-provider-aws#6082.

[Andrew-Tan]

+1 on this

I think the cause of this issue is, cloudfront (or api gateway) resource in Terraform in general depends on acm certificates to be destroyed first. However, on AWS, you cannot delete a certificate when it is in use (by cloudfront, for example). So this potentially this creates a circular dependency that Terraform is not aware of.

In other words:
The correct order of procedure should be:

• Change/Delete custom domain certificate settings in Cloudfront (API Gateway) to release lock on the certificate
• Delete certificate
• Change/Delete Cloudfront (API Gateway) resources

While in reality, Terraform might be doing only the last 2 steps:

• Delete certificate (locked because Cloudfront is using it)
• Change/Delete Cloudfront (API Gateway) resources

I'm not sure if this is a problem that needs to be solved on the Terraform side or AWS provider side..

[Andrew-Tan]

By the way, the reason why I made this guess is, through my experiment, after Terraform returns error of certificate in use, I went to checked the custom domain settings in my API Gateway resource and noticed that it is still set to the certificate to be destroyed, which means terraform didn't set it before destroying the certificate to release the lock.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.