workload identity existing sa iam.gke.io/gcp-service-account overwrite

[ronaldslc]

Following the documented approach to use existing sa would result in an error

resource "kubernetes_service_account" "iden-test" {
  metadata {
    name = "iden-test"
    annotations = {
      "iam.gke.io/gcp-service-account": "iden-test@myproject.iam.gserviceaccount.com"
    }
  }
  secret {
    name = my-custom-secret
  }
}

module "workload_identity" {
  source              = "terraform-google-modules/kubernetes-engine/google//modules/workload-identity"
  project_id          = var.project
  name                = "iden-test"
  use_existing_k8s_sa = true
}

Output: error: --overwrite is false but found the following declared annotation(s): 'iam.gke.io/gcp-service-account' already has a value

I can see two options to fix this:

1. add --overwrite parameter to module "annotate-sa", which would require users to implement lifecycle ignore annotations
2. remove module "annotate-sa" completely (and have users define their own annotations as per documentation)

I would prefer 2. approach to keep user defined terraform files clean

kubectl version: v1.18.2
cluster version: v1.16.8-gke.15

[morgante]

If you remove annotations from the Terraform config will it attempt to overwrite them?

I'm inclined to go with option #1 as one of the primary use-cases for use_existing_k8s_sa is where the k8s SA was created outside Terraform (ex. via a Helm chart) and annotating it manually would be cumbersome.

/cc @bharathkkb

[ronaldslc]

Do you mean remove annotations from the resource "kubernetes_service_account" block? If I don't provide annotations, it will run fine, there is nothing to overwrite so it will just insert, but will result in lifecycle changes in subsequent runs where it will delete the annotation added by annotate-sa.

I guess for Option #1 it could work without manually adding lifecycle ignore if the iam.gke.io/gcp-service-account annotation value is exactly the same as what's added by the workload_identity module

[bharathkkb]

I am also leaning towards option #1. The WI module is responsible for creating the GSA (iden-test) that binds to the KSA, so I think it makes sense to let the module handle annotation for most cases. We should also update docs and remove annotations in kubernetes_service_account. The only use case I can think outside of this, is if the GSA is created outside of the module.

Also @morgante we don't seem to be using the kubectl_wrapper.sh for the annotation command. If kubectl doesn't have the correct kubeconfig wouldn't this break?

[morgante]

Also @morgante we don't seem to be using the kubectl_wrapper.sh for the annotation command. If kubectl doesn't have the correct kubeconfig wouldn't this break?

Yeah we should probably use it.

[bharathkkb]

cool, will submit a PR