Defer data source reads with output depends_on

[bharathkkb]

Terraform Version

1.1.7

Terraform Configuration Files

# main.tf
locals {
  bucket = "YOUR_BUCKET"
}

module "inner" {
  source = "./foo"
  bucket = local.bucket
}

data "google_storage_bucket_object" "output" {
  name   = module.inner.op
  bucket = local.bucket
}

output "name" {
  value = data.google_storage_bucket_object.output.media_link
}

# foo/main.tf
locals {
  static = "test"
}

variable "bucket" {
}

resource "google_storage_bucket_object" "create" {
  name   = local.static
  content = local.static
  bucket = var.bucket
}

output "op" {
  value = google_storage_bucket_object.create.name
  depends_on = [
    google_storage_bucket_object.create
  ]
}

Debug Output

Expected Behavior

data "google_storage_bucket_object" "output" read should have been deferred to apply time as it needs module.inner.op which has a dependency on google_storage_bucket_object.create.

Adding an explicit module dependency as mentioned below should not be necessary if the output has explicit dependency.

It also looks like this worked as expected in 0.12.31 but is not working for subsequent versions.

Actual Behavior

Datasource read is not deferred to apply time resulting in a 404

terraform apply
╷
│ Error: Error retrieving storage bucket object: googleapi: Error 404: No such object: YOUR_BUCKET/test, notFound
│ 
│   with data.google_storage_bucket_object.output,
│   on main.tf line 10, in data "google_storage_bucket_object" "output":
│   10: data "google_storage_bucket_object" "output" {

Adding an explicit dependency like

data "google_storage_bucket_object" "output" {
  name   = module.inner.op
  bucket = local.bucket
  depends_on = [
    module.inner
  ]
}

does defer the read to apply time

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.google_storage_bucket_object.output will be read during apply
  # (config refers to values not yet known)
 <= data "google_storage_bucket_object" "output"  {

Steps to Reproduce

Additional Context

The example is contrived and created for simplicity. terraform-google-modules/terraform-google-kubernetes-engine#1181 and trying to fix it via terraform-google-modules/terraform-google-kubernetes-engine#1189 (comment) was the reason behind this issue.

References

[jbardin]

Hi @bharathkkb,

The data source will be deferred until apply under 2 circumstances: if the configuration for the data source contains an unknown value, or the data source uses depends_on which includes resource which has a planned change. The depends_on within the output has no effect, because the output already references the resource, and since the result of the output is known during plan, it will be the same during apply.

It's generally recommended to not have a managed resource and a data source represent the same logical resource within the same configuration to avoid these types of issues. When that situation is unavoidable, the correct solution is to use depends_on with the data source to force the evaluation to happen during apply.

The behavior in 0.12 was incorrect in many cases, but prior to v0.14 data source evaluation was basically a "best effort" attempt, because it happened during a separate refresh phase where there was incomplete information to determine what could and couldn't be read. Now that data source evaluation happens during plan, the results have been made consistent.

We use GitHub issues for tracking bugs and enhancements, rather than for questions. While we can sometimes help with certain simple problems here, it's better to use the community forum where there are more people ready to help.

Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.