Bug in 1.9.0 when creating route53 entries for ACM approvals

[monkey-jeff]

This occurs every time on the first run of a terraform apply...on a subsequent run it works fine (So our current work around is run it again)

NOTE: I Have censored our actual DNS names for this as it might contain information we don't want posted to a public location

• module.service_definition.aws_route53_record.cn_cert_validation: aws_route53_record.cn_cert_validation: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

Terraform Version: 0.11.1
Resource ID: aws_route53_record.cn_cert_validation
Mismatch reason: attribute mismatch: records.942692827
Diff One (usually from plan): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{"ttl":*terraform.ResourceAttrDiff{Old:"", New:"60", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "type":*terraform.ResourceAttrDiff{Old:"", New:"CNAME", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "records.#":*terraform.ResourceAttrDiff{Old:"", New:"1", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "records.942692827":*terraform.ResourceAttrDiff{Old:"", New:"_1480da86a40a658a93bd31e006759e1a.acm-validations.aws.", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "name":*terraform.ResourceAttrDiff{Old:"", New:"_8154b685e678779c8f64f2bd964294b0.foo.bar.com", NewComputed:false, NewRemoved:false, NewExtra:"_8154b685e678779c8f64f2bd964294b0.foo.bar.com.", RequiresNew:true, Sensitive:false, Type:0x0}, "fqdn":*terraform.ResourceAttrDiff{Old:"", New:"", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "zone_id":*terraform.ResourceAttrDiff{Old:"", New:"Z3PNQYIZM8851A", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, Meta:map[string]interface {}(nil)}
Diff Two (usually from apply): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{"name":*terraform.ResourceAttrDiff{Old:"", New:"_1b72220dfd38e5357f1e4da5bcf452f4.zsryq6c8.foo.bar.com", NewComputed:false, NewRemoved:false, NewExtra:"_1b72220dfd38e5357f1e4da5bcf452f4.zsryq6c8.foo.bar.com.", RequiresNew:true, Sensitive:false, Type:0x0}, "records.#":*terraform.ResourceAttrDiff{Old:"", New:"1", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "records.3982558149":*terraform.ResourceAttrDiff{Old:"", New:"_7d301af98c6434cfd04ed0550f2a7164.acm-validations.aws.", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "fqdn":*terraform.ResourceAttrDiff{Old:"", New:"", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "ttl":*terraform.ResourceAttrDiff{Old:"", New:"60", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "type":*terraform.ResourceAttrDiff{Old:"", New:"CNAME", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "zone_id":*terraform.ResourceAttrDiff{Old:"", New:"Z3PNQYIZM8851A", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, Meta:map[string]interface {}(nil)}

Also include as much context as you can about your config, state, and the steps you performed to trigger this error.

This is what the code looks like

resource "aws_acm_certificate" "certificate" {
  # Using app name for common name to avoid CN character limitations
  domain_name       = "${aws_route53_record.random_subdomain.fqdn}"
  validation_method = "DNS"

  # Your DNS will be set as the first SAN in the list
  subject_alternative_names = ["${aws_route53_record.alb.fqdn}"]

  tags {
    Owner = "${var.owner}"
    Name  = "The Certificate for the ${var.app_name} service in ${var.vpc} deployed to ${var.region}"
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_route53_record" "cn_cert_validation" {
  name    = "${aws_acm_certificate.certificate.domain_validation_options.0.resource_record_name}"
  type    = "${aws_acm_certificate.certificate.domain_validation_options.0.resource_record_type}"
  zone_id = "${data.aws_route53_zone.approval_zone.id}"
  records = ["${aws_acm_certificate.certificate.domain_validation_options.0.resource_record_value}"]
  ttl     = 60
}

resource "aws_route53_record" "san_cert_validation" {
  name    = "${aws_acm_certificate.certificate.domain_validation_options.1.resource_record_name}"
  type    = "${aws_acm_certificate.certificate.domain_validation_options.1.resource_record_type}"
  zone_id = "${data.aws_route53_zone.approval_zone.id}"
  records = ["${aws_acm_certificate.certificate.domain_validation_options.1.resource_record_value}"]
  ttl     = 60
}

Thanks for any assistance on this issue.

-Jeff

[bflad]

Hi @sophos-jeff, sorry you're running into trouble here. Are you able to turn on debug logging and see if the DomainValidationOptions results coming back from AWS are in a different order between the plan and apply runs?

[aduzsardi]

on the same note , this should be optional since when using SAN in certificate requests DNS records are the same

module.test.aws_acm_certificate_validation.cert: Creating...
certificate_arn: "" => "arn:aws:acm:eu-central-1:860993037417:certificate/c224e1a5-d283-4b7b-b942-45d6e24f77a3"
validation_record_fqdns.#: "" => "1"
validation_record_fqdns.2913612333: "" => "_abd330940bad21e0ab11718cb86ca34a.mydomin.com"

Error: Error applying plan:

1 error(s) occurred:

• module.test.aws_acm_certificate_validation.cert: 1 error(s) occurred:

• aws_acm_certificate_validation.cert: Certificate needs [_abd330940bad21e0ab11718cb86ca34a.mydomin.com _abd330940bad21e0ab11718cb86ca34a.mydomin.com] to be set but only [_abd330940bad21e0ab11718cb86ca34a.mydomin.com] was passed to validation_record_fqdns

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

so instead of doing this

resource "aws_route53_record" "cn_cert_validation" {
  name    = "${aws_acm_certificate.certificate.domain_validation_options.0.resource_record_name}"
  type    = "${aws_acm_certificate.certificate.domain_validation_options.0.resource_record_type}"
  zone_id = "${data.aws_route53_zone.approval_zone.id}"
  records = ["${aws_acm_certificate.certificate.domain_validation_options.0.resource_record_value}"]
  ttl     = 60
}

resource "aws_route53_record" "san_cert_validation" {
  name    = "${aws_acm_certificate.certificate.domain_validation_options.1.resource_record_name}"
  type    = "${aws_acm_certificate.certificate.domain_validation_options.1.resource_record_type}"
  zone_id = "${data.aws_route53_zone.approval_zone.id}"
  records = ["${aws_acm_certificate.certificate.domain_validation_options.1.resource_record_value}"]
  ttl     = 60
}

resource "aws_acm_certificate_validation" "cert" {
  certificate_arn = "${aws_acm_certificate.cert.arn}"
  validation_record_fqdns = ["${aws_route53_record.cn_cert_validation.fqdn}","${aws_route53_record.san_cert_validation.fqdn}"]
}

you should be able to do just

resource "aws_route53_record" "cn_cert_validation" {
  name    = "${aws_acm_certificate.certificate.domain_validation_options.0.resource_record_name}"
  type    = "${aws_acm_certificate.certificate.domain_validation_options.0.resource_record_type}"
  zone_id = "${data.aws_route53_zone.approval_zone.id}"
  records = ["${aws_acm_certificate.certificate.domain_validation_options.0.resource_record_value}"]
  ttl     = 60
}

resource "aws_acm_certificate_validation" "cert" {
  certificate_arn = "${aws_acm_certificate.cert.arn}"
  validation_record_fqdns = ["${aws_route53_record.cn_cert_validation.fqdn}"]
}

[monkey-jeff]

Working on the debug logging still..Should have that today.

@aduzsardi The CN and SAN are different from each other. This was updated like this due to hitting the 64 character limit on a CN so there is kind of an ID CN generated and using the FQDN for the SAN instead to get around this issue.

NOTE: This issue was still occurring with only a CN as well, before I implemented the SAN, but it happened only once so I had ignored it before implementing the SA. It has happened more since then, hopefully getting the debug logs reveal more.

[bflad]

@aduzsardi the mentioned Certificate needs [SAMERECORD SAMERECORD] to be set but only [SAMERECORD] was passed to validation_record_fqdns bug was issued as #3329 and has a PR submitted #3366

[monkey-jeff]

I have a fully debug log.... here is a better look at another example of the error.

2018/02/14 14:05:20 [ERROR] root.service_definition: eval: *terraform.EvalSequence, err: aws_route53_record.cert_validation: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

    Terraform Version: 0.11.1
    Resource ID: aws_route53_record.cert_validation
    Mismatch reason: attribute mismatch: records.3903573995
    Diff One (usually from plan): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{"zone_id":*terraform.ResourceAttrDiff{Old:"", New:"${data.aws_route53_zone.approval_zone
.id}", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, "type":*terraform.ResourceAttrDiff{Old:"", New:"CNAME", NewComputed:false, NewRemoved:false, NewExtra:interface {
}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "name":*terraform.ResourceAttrDiff{Old:"", New:"_2c83b422bd94319bf50a15e9e66084ca.xlbwodhi.cloudhub.eu-west-1.foo.com", NewComputed:false, NewRemoved:false, NewExt
ra:"_2c83b422bd94319bf50a15e9e66084ca.xlbwodhi.cloudhub.eu-west-1.foo.com.", RequiresNew:true, Sensitive:false, Type:0x0}, "ttl":*terraform.ResourceAttrDiff{Old:"", New:"60", NewComputed:false, NewRemoved:false, NewExt
ra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "records.3903573995":*terraform.ResourceAttrDiff{Old:"", New:"_876adf0067c1aad0001657ed41e43cdb.acm-validations.aws.", NewComputed:false, NewRemoved:false, NewEx
tra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "records.#":*terraform.ResourceAttrDiff{Old:"", New:"1", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Ty
pe:0x0}, "fqdn":*terraform.ResourceAttrDiff{Old:"", New:"", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false,
 Meta:map[string]interface {}(nil)}
    Diff Two (usually from apply): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{"fqdn":*terraform.ResourceAttrDiff{Old:"", New:"", NewComputed:true, NewRemoved:false,
NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "type":*terraform.ResourceAttrDiff{Old:"", New:"CNAME", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false
, Type:0x0}, "name":*terraform.ResourceAttrDiff{Old:"", New:"_e82d6f4e2d3a3db062dbb71979c819b1.demo-ms-cloudhub-eu-west-1.foo.com", NewComputed:false, NewRemoved:false, NewExtra:"_e82d6f4e2d3a3db062dbb71979c819b1.demo-
ms-cloudhub-eu-west-1.foo.com.", RequiresNew:true, Sensitive:false, Type:0x0}, "ttl":*terraform.ResourceAttrDiff{Old:"", New:"60", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sen
sitive:false, Type:0x0}, "records.#":*terraform.ResourceAttrDiff{Old:"", New:"1", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "records.3968952290":*terraform.Reso
urceAttrDiff{Old:"", New:"_9189fec2498aa8334c83df1e8b884815.acm-validations.aws.", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "zone_id":*terraform.ResourceAttrDi
ff{Old:"", New:"Z3PNQYIZM8851A", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, Meta:map[string]interface
{}(nil)}

Also include as much context as you can about your config, state, and the steps you performed to trigger this error.

[bflad]

What happens if you run this with Terraform 0.11.3? Does "${aws_route53_record.alb.fqdn}" have uppercase letters or a period at the end? I have not been able to reproduce this over a few dozen iterations of acceptance testing on 0.11.3, but we're also using hardcoded lowercase values with no ending period for everything.

If your debug log contains sensitive information you can encrypt it using the Hashicorp GPG key and I can take a look.

[bflad]

Closing due to lack of response. If you're still having this problem please open a new issue.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!