Terraform does not apply planned changes when a plan file is used,

[stefano-cherchi]

After some EC2 instance has been stopped and started again, I run terraform plan -out foo.plan and Terraform outputs this as expected, since instance's IP and hostnames have changed:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ aws_route53_record.se[0]
      records.1190243470: "ec2-xxx-xxx-xxx-xxx.us-west-2.compute.amazonaws.com" => ""
      records.3360685831: "" => "ec2-yyy-yyy-yyy-yyy.us-west-2.compute.amazonaws.com"

  ~ aws_route53_record.se[1]
      records.1911928299: "ec2-xxx-xxx-xxx-xxx.us-west-2.compute.amazonaws.com" => ""
      records.2488098414: "" => "ec2-yyy-yyy-yyy-yyy.us-west-2.compute.amazonaws.com"

  ~ aws_route53_record.se[2]
      records.2116317077: "ec2-xxx-xxx-xxx-xxx.us-west-2.compute.amazonaws.com" => ""
      records.3438509894: "" => "ec2-yyy-yyy-yyy-yyy.us-west-2.compute.amazonaws.com"


Plan: 0 to add, 3 to change, 0 to destroy.

------------------------------------------------------------------------

This plan was saved to: foo.plan

To perform exactly these actions, run the following command to apply:
    terraform apply "foo.plan"

When I run terraform apply "foo.plan" this is the result:

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

If I run terraform apply with no plan file, instead, changes are applied as expected:

[ ... ]

Plan: 0 to add, 3 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_route53_record.se[1]: Modifying... (...

[ ... ]

Apply complete! Resources: 0 added, 3 changed, 0 destroyed.

Terraform Version

$ terraform -v
Terraform v0.11.11
+ provider.aws v1.60.0
+ provider.random v2.0.0

Terraform Configuration Files

resource "aws_route53_record" "se" {
  zone_id = "${data.aws_route53_zone.ec2farm.id}"
  name    = "${aws_instance.se.*.tags.Name[count.index]}.${data.aws_route53_zone.ec2farm.name}"
  type    = "CNAME"
  ttl     = "300"
  records = ["${aws_instance.se.*.public_dns[count.index]}"]
  allow_overwrite = true
  count   = "${aws_instance.se.count}"
}

Expected Behavior

Planned changes should be applied when a plan file is passed to the terraform apply command

Actual Behavior

Planned changes are applied only if no plan file is used. If a plan file is passed as argument to terraform apply, the command returns no error but planned changes are completely ignored.

Steps to Reproduce

1. terraform plan -out foo.plan
2. terraform apply "foo.plan"

Additional Context

none

References

none

[mildwonkey]

Hi @stefano-cherchi !
I'm sorry you're experiencing this issue; it's an odd one.

It will help us to see the debug output of the misbehaving terraform apply command. You can run TF_LOG=DEBUG terraform apply foo.plan and copy the output into a gist (removing any sensitive information first).

[sryabkov]

@mildwonkey I ran into the same problem with terraform 0.11.13: terraform apply works fine (a resource is added), but terraform plan -out default.tfplan followed by terraform apply default.tfplan results in zero changes.
For me, only adding new resources is affected, delete works fine, and I didn't test resource change. The issue seems to be code specific: some TF modules works fine, but one of my TF modules exhibits this behavior.
Should I create a new issue for this or post a link to the gist with debugging info to this issue?

[mildwonkey]

@sryabkov - great question! Go ahead and share what you have here, please. We'd love to get a reproducible test case. Thanks!

[sryabkov]

@mildwonkey The detailed debugging info is available here: https://gist.github.com/sryabkov/1aa9548bbd909a76b43fd9b67150b075

[mildwonkey]

Thanks @sryabkov, this is a fantastic write up and very helpful information.

Would you mind sending along one more piece of information? It would be very helpful to see the full trace log from the apply command (adding whatever flags you need):

TF_LOG=trace terraform apply tf.plan

[sryabkov]

@mildwonkey thank you, will do shortly

[sryabkov]

@mildwonkey I updated the gist with the trace for the apply operation from a pre-generated plan. Here is the direct link: https://gist.github.com/sryabkov/1aa9548bbd909a76b43fd9b67150b075#file-tf_apply_from_saved_plan_trace-log

[apparentlymart]

After reading through the trace log, I see the following curious entry:

2019/04/11 20:48:35 [WARN] Interpolation "count" failed: Error reading data.template_file.projects_permissions count: cannot parse "${length(var.big_query_project_names) * length(var.gcp_bigquery_project_iam_permissions)}" as an integer

I'm not sure why that wasn't being returned as a proper error or why it wasn't visible at TF_LOG=debug, but let's put that aside for the moment and see if we can explain the error message itself.

Looking at your configuration, I notice that the count expression here is actually coming from the count of another resource:

resource "google_project_iam_member" "project" {
  count = "${data.template_file.projects_permissions.count}"

  project = "${element(split(",", data.template_file.projects_permissions.*.rendered[count.index]),0)}"
  role    = "${element(split(",", data.template_file.projects_permissions.*.rendered[count.index]),1)}"
  member  = "${element(split(",", data.template_file.projects_permissions.*.rendered[count.index]),2)}"
}

I think the cause of the problem here, then, is that the data resource got processed early on during the refresh phase and so Terraform didn't include it in the plan at all -- as far as it was concerned, all of the work for it was done already. Because Terraform didn't visit the data resource during the apply phase, it didn't get the opportunity to run the interpolator on the count value, and so (due to the design of the interpolator in Terraform v0.11) it ended up using the source string instead. This inadvertently works on a one-shot terraform apply because the already-processed count expression is still in memory from the refresh step where the data resource was processed.

In the forthcoming Terraform v0.12 (what is currently represented by the master branch) this special .count attribute is no longer available, since it was implemented as a rather-troublesome special case; this bug is a new example of how troublesome it can be, since it depends on reaching across into the configuration of another resource to do its work. For Terraform v0.12 the way to write this would be length(data.template_file.projects_permissions), which can work in v0.12 because we've refactored how count is handled.

Unfortunately I expect a fix for this issue in Terraform v0.11 is not practical because it would require replicating similar refactoring we did for v0.12 into the very-differently-designed v0.11 interpolator. Therefore I think the best we can do about it for now is a workaround: instead of using the count of one resource to populate another, instead make the count expression a named local value and interpolate it into both places:

locals {
  project_permissions_count = "${length(var.big_query_project_names) * length(var.gcp_bigquery_project_iam_permissions)}"
}

data "template_file" "projects_permissions" {
  count = "${local.project_permissions_count}"

  # ...
}

resource "google_project_iam_member" "project" {
  count = "${local.project_permissions_count}"

  # ...
}

What we may be able to do for v0.11 is to figure out why this error is only appearing as a warning in the logs, and not as a real error message. But the error message itself is not particularly helpful, so that would only go so far here. Since the Terraform team at HashiCorp is currently 100% focused on completing the v0.12.0 release, unfortunately I expect that upgrading to v0.12.0 once it's released will end up being the best remedy we can manage here.

@stefano-cherchi, it looks like the same is true in your situation: you're also using the .count attribute of one resource in the count argument for another, and the other one is not included in the plan. I think a similar approach to my example above will work for you too, assuming the count expression for aws_instance.se can be factored out into a local value.

[sryabkov]

@apparentlymart Martin, thank you very much for the detailed explanation of what's going on. I tried your suggested workaround, and it worked, so thank you for unblocking us.

I agree it doesn't make much sense to try to retrofit the fix for this issue into v0.11, but I also think that v0.11 should throw an error and return a non-zero completion code so that downstream (or upstream?) tools (in our case, terragrunt and atlantis) can correctly report the result of the operation. If figuring out how to make v0.11 return proper error could take a while, which would further delay the v0.12 release, may I suggest still doing it after v0.12.0 is released? I think it is going to be a while before everyone upgrades (we, for example, would need to wait for terragrunt and atlantis versions that support v0.12 to be released). We are aware of the problem now, and can compensate for it using the workaround you provided, but other other unsuspecting TF developers might still run into this issue, so it would be useful for them if TF were properly returning an error.

[hashibot]

Hello! 🤖

This issue relates to an older version of Terraform that is no longer in active development, and because the area of Terraform it relates to has changed significantly since the issue was opened we suspect that the issue is either fixed or that the circumstances around it have changed enough that we'd need an updated issue report in order to reproduce and address it.

If you're still seeing this or a similar issue in the latest version of Terraform, please do feel free to open a new bug report! Please be sure to include all of the information requested in the template, even if it might seem redundant with the information already shared in this issue, because the internal details relating to this problem are likely to be different in the current version of Terraform.

Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.