google_cloud_run_service using environment variables from secret manager is ordering sensitive making it difficult to use

[txomon]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Problem overview

The main objective is to be able to deploy cloud_run instances with secret_manager resources referenced. There are two main problems that get on the way of supporting deployments external to terraform (the ability to also deploy new images from outside terraform):

Documentation

The current documentation is not correct, https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_run_service#example-usage---cloud-run-service-secret-environment-variables says the following syntax is correct:

resource "google_cloud_run_service" "default" {
  provider = google-beta

  name     = "cloudrun-srv"
  location = "us-central1"

  template {
    spec {
      containers {
        image = "gcr.io/cloudrun/hello"
        env {
          name = "SECRET_ENV_VAR"
          value_from {
            secret_key_ref {
              name = google_secret_manager_secret.secret.secret_id
              key = "1"
            }
          }
        }
      }
    }
  }

  autogenerate_revision_name = true

}

However this would never work, as the validation requires it to only contain valid kubernetes secret references, and a secret_id is of the form projects/myproject/secrets/mysecret. Instead, one needs to populate the annotations with an attribute run.googleapis.com/secrets of key:value comma separated list, where key is a string starting with secret- and a random UUID, and value is the secret id of the form projects/myproject/secrets/mysecret like this instead:

resource "google_cloud_run_service" "default" {
  provider = google-beta

  name     = "cloudrun-srv"
  location = "us-central1"

  template {
    metadata {
      annotations = {
        "run.googleapis.com/secrets" = "secret-cdb5c845-50c1-482a-9e5c-9ac1597e9d98:${google_secret_manager_secret.secret.secret_id}"
      }
    }
    spec {
      containers {
        image = "gcr.io/cloudrun/hello"
        env {
          name = "SECRET_ENV_VAR"
          value_from {
            secret_key_ref {
              name = "secret-cdb5c845-50c1-482a-9e5c-9ac1597e9d98"
              key = "1"
            }
          }
        }
      }
    }
  }

  autogenerate_revision_name = true

}

Ordering of env blocks

The current implementation of the google_cloud_run_service's template uses blocks for configuration, and when using dynamic as in the terraform file pasted below, after a gcloud run deploy, the blocks need to be ordered in the same way (which is not possible to my knowledge) to avoid having a change like the following:

  # module.api.module.cloud_run_api.google_cloud_run_service.instance will be updated in-place
  ~ resource "google_cloud_run_service" "instance" {
        id                         = "locations/europe-west1/namespaces/first-cedar-269508/services/api"
        name                       = "api"
        # (4 unchanged attributes hidden)


      ~ template {

          ~ spec {
                # (4 unchanged attributes hidden)

              ~ containers {
                    # (3 unchanged attributes hidden)

                  ~ env {
                      ~ name = "B" -> "A"

                      ~ value_from {
                          ~ secret_key_ref {
                              ~ name = "secret-1111111-1111-1111-1111-111111111111" -> "secret-00000000-0000-0000-0000-000000000000"
                                # (1 unchanged attribute hidden)
                            }
                        }
                    }
                  ~ env {
                      ~ name = "A" -> "B"

                      ~ value_from {
                          ~ secret_key_ref {
                              ~ name = "secret-00000000-0000-0000-0000-000000000000" -> "secret-1111111-1111-1111-1111-111111111111"
                                # (1 unchanged attribute hidden)
                            }
                        }
                    }

                    # (24 unchanged blocks hidden)
                }
            }
        }

        # (3 unchanged blocks hidden)
    }

Terraform Version

Terraform v0.15.0
on linux_amd64

• provider registry.terraform.io/hashicorp/google v3.67.0
• provider registry.terraform.io/hashicorp/google-beta v3.67.0
• provider registry.terraform.io/hashicorp/local v2.1.0
• provider registry.terraform.io/hashicorp/random v3.1.0

Your version of Terraform is out of date! The latest version
is 0.15.3. You can update by downloading from https://www.terraform.io/downloads.html

Affected Resource(s)

• google_cloud_run_service

Terraform Configuration Files

data "google_cloud_run_service" "instance" {
  project  = var.project.id
  name     = var.name
  location = var.project.region
}

resource "random_uuid" "secrets" {
  for_each = var.env_secrets
}

locals {
  # esu == existing_secret_uuids
  esu_string = lookup(data.google_cloud_run_service.instance.template[0].metadata[0].annotations, "run.googleapis.com/secrets", "")
  esu_parts = local.esu_string == ""? []: split(",", local.esu_string)
  # Key should be the second part, value should be the first part
  esu_values = local.esu_string == ""? [] : [for part in local.esu_parts : split(":", part)[0]]
  esu_keys = local.esu_string == ""? [] : [for part in local.esu_parts : split(":", part)[1]]
  esu = local.esu_string == ""? {} : zipmap(local.esu_keys, local.esu_values)
  # Choose the existing uuid or a new one if it doesn't exist
  secret_uuids                = { for env_name, secret_id in var.env_secrets : env_name => (lookup(local.esu, secret_id, "") == ""? "secret-${random_uuid.secrets[env_name].result}" : local.esu[secret_id])}
  secret_mappings            = { for env_name, secret_id in var.env_secrets : env_name => "${local.secret_uuids[env_name]}:${secret_id}" }
  run_googleapis_com_secrets = join(",", values(local.secret_mappings))
}

resource "google_cloud_run_service" "instance" {
  provider = google-beta

  project                    = var.project.id
  name                       = var.name
  location                   = var.project.region
  autogenerate_revision_name = true

  template {
    metadata {
      annotations = {
        "autoscaling.knative.dev/maxScale"      = var.max_scale
        "run.googleapis.com/cloudsql-instances" = nonsensitive(join(",", var.connection_names))
        "run.googleapis.com/client-name"        = "terraform"
        "autoscaling.knative.dev/minScale"      = var.min_scale
        "run.googleapis.com/secrets"            = local.run_googleapis_com_secrets
      }
    }
    spec {
      container_concurrency = 80
      service_account_name  = var.service_account.email
      timeout_seconds       = 299

      containers {
        image = data.google_cloud_run_service.instance.template == null ? var.image : data.google_cloud_run_service.instance.template[0].spec[0].containers[0].image

        dynamic "env" {
          for_each = var.env_vars
          content {
            name  = env.key
            value = env.value
          }
        }

        dynamic "env" {
          for_each = local.secret_uuids
          content {
            name = env.key
            value_from {
              secret_key_ref {
                name = env.value
                key  = "latest"
              }
            }
          }
        }

        ports {
          container_port = var.container_port
        }

        resources {
          limits = {
            "cpu"    = var.cpu_limit
            "memory" = var.memory_limit
          }
          requests = {}
        }
      }
    }
  }

}

resource "google_cloud_run_domain_mapping" "instance" {
  # TLS Certificate is internally managed by GCP
  project = var.project.id

  location = var.project.region
  name     = var.fqdn

  metadata {
    namespace = var.project.id
  }

  spec {
    route_name = google_cloud_run_service.instance.name
  }
}


resource "google_cloud_run_service_iam_binding" "instance" {
  count   = var.public ? 1 : 0
  members = ["allUsers"]
  role    = "roles/run.invoker"

  service  = google_cloud_run_service.instance.name
  location = google_cloud_run_service.instance.location
  project  = google_cloud_run_service.instance.project
}

Expected Behavior

When applying through terraform after creating a deployment from gcloud, there should be no changes detected if the key-values in var.env_secrets don't show diferences

Actual Behavior

Terraform detects changes in the order of the env blocks created dynamically

Steps to Reproduce

1. terraform apply
2. gcloud run deploy <cloud_run_instance_name> --project <project_id> --region <region_name> --image <image> --platform managed --quiet
3. terraform apply # Shouldn't show changes but it does

Important Factoids

The code is trying to make sure that external deployments are accepted from outside terraform

References

• Ignoring order of resource blocks upon modification fortinetdev/terraform-provider-fortios#132

[edwardmedia]

@txomon you mentioned there are two issues here. Mind file separate issues so they can be tracked properly. Also can you post the debug logs for each case?

I noticed your analysis below. I am not quite clear. Here is what the doc suggests. Following is a test that runs everyday. Can you share the debug log so I can get clearer picture of what the problem is?

However this would never work, as the validation requires it to only contain valid kubernetes secret references, and a secret_id is of the form projects/myproject/secrets/mysecret. Instead, one needs to populate the annotations with an attribute run.googleapis.com/secrets of key:value comma separated list, where key is a string starting with secret- and a random UUID, and value is the secret id of the form projects/myproject/secrets/mysecret like this instead:

https://github.com/hashicorp/terraform-provider-google-beta/blob/236e6b03711d7c830a63ad073d3443125fcf2003/google-beta/resource_cloud_run_service_test.go#L319

[txomon]

This one is related to #9159

[edwardmedia]

I can see the problem of Ordering of env blocks

[txomon]

Let's then keep this ticket for the ordering env blocks, and I will tackle the documentation in the other one

[megan07]

Hi @txomon ! Sorry you're running into this issue. Unfortunately, I wasn't able to reproduce it, but I have a fix in that if the order of env is ordered differently than the config, it won't show a diff. Let me know if this works for you. Thanks!

[txomon]

Hello @megan07 , I have tested the latest version, and although the difference when ordering is different is solved, when there is an actual removal of an env block, something like this displays:

...
                  ~ env {                                                      
                      ~ name = "DELETED_VARIABLE" -> "EXISTING_VARIABLE"                                                                                  
                                                                               
                      ~ value_from {
                          ~ secret_key_ref {                                   
                              ~ name = "secret-068031ba-0569-ca6f-be57-f3b0b27c87bc" -> "secret-17bada44-c8bc-6848-d91c-d5f5f07e12b5"                          
                                # (1 unchanged attribute hidden)               
                            }                                    
                        }                                                      
                    }               
                  - env {                   
                      - name = "EXISTING_VARIABLE" -> null                                                                                                 
                                                                               
                      - value_from {
                          - secret_key_ref {      
                              - key  = "latest" -> null
                              - name = "secret-17bada44-c8bc-6848-d91c-d5f5f07e12b5" -> null                                                                   
                            }                                    
                        }                                                      
                    }               
...

To reproduce, apply a cloud run config with two env vars, apply, and then remove one of them.

[megan07]

Hi @txomon ! This will be released in our next version. Did you build the provider locally and see this or were you using 3.72.0? Thanks!

[txomon]

Hello megan, Sorry, I thought it had been released already, I will wait until it's released and report back, Cheers, Javier

…

On Fri, Jun 18, 2021 at 9:09 PM megan07 ***@***.***> wrote: Hi @txomon <https://github.com/txomon> ! This will be released in our next version. Did you build the provider locally and see this or were you using 3.72.0? Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9175 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABXXGVUVRGWRJA2QIWINWLTTOKW5ANCNFSM45DBHE4Q> .

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.