GKE modules squash ASM cluster labels

[bjhshadow]

GKE submodules will delete cluster labels on any subsequent run after creation.

ASM uses 'asmv" and "mesh_id" for observability and mesh management and are created dutifully by ASM module. Any run with a GKE module (beta-private-cluster-update-variant, beta-private-cluster, private-cluster) will remove ASM cluster resource labels acting as the authorative source. The module does not allow the use of the lifecycle meta argement to ignore them. The modules cannot be used in conjuction or with separate terraform plans on the same cluster.

The ASM module never recognizes that the cluster resource labels are removed regardless of the 'enable_cluster_labels' input setting.

GKE should allow lifecycle meta or ignore labels as an 'additive' option. ASM module should recognize and replace if removed.

[morgante]

Unfortunately the current ASM model does not work very well with the declarative approach of Terraform.

The best workaround is just to add the asmv and mesh_id as labels in your Terraform config. Is there a reason this wouldn't work?

[bjornsen]

+1 for this bug. In scenarios where terraform is continuously applied in an automated fashion, the ASM resource labels may be stripped before users know what they are or realize they are required. The mesh_id looks as though it can be auto-generated because it's simply proj-{PROJECT_NUMBER} but I don't see a good way to know asmv before setting up ASM. Further, I believe you'll run into the issue where ASM is updated which updates asmv but Terraform switches in back to the version fixed in its resource labels. I imagine that'll cause a bug in ASM though I'm not sure how it would manifest.

[richiefrich]

Thats fine for adding them myself but how do I fetch the asmv ? As I am using a managed control plane. Can I pull the asmv from output of the module? From what I see I cant..

[morgante]

The truth is that asmv is not actually required. It's just used for debugging. So you could exclude it entirely and set enable_cluster_labels = false when invoking the ASM install.

[richiefrich]

But asmv is needed cause we need it to fetch the correct version of istioctl to install the cluster secrets.
gke-install-multi-cluster

I do notice the warning at the top to use asmcli will the module be switching to this soon? Thanks!

But we will still need the asmv unless the secrets will be include in the Terraform module.

[morgante]

But asmv is needed cause we need it to fetch the correct version of istioctl to install the cluster secrets.

Can you clarify where you're seeing the requirement for that? My understanding is that the asmv label isn't really needed.

I do notice the warning at the top to use asmcli will the module be switching to this soon? Thanks!

We are planning to refactor to a different approach sometime in the next few months.

[richiefrich]

So again I am using managed control planes so how do I match the ASM major and minor versions without knowing what version is being installed, so I can download the correct version of istioctl. I do know you shouldn't be using just any version of istioctl to create the secrets. I get that you can set the ASM version in this module but that isn't the minor version. So how do you create your secrets? I posted the article for you. I do not want to manually add these secrets, I make to many environments to keep track of these. Thanks!

[morgante]

@richiefrich Please open a ticket with your Google support account. I don't think you actually need to use istioctl with the managed control plane at this point. Regardless, there are limits to how much we can support you. It's not a Terraform issue, it's an ASM issue.

[richiefrich]

@morgante You are correct this is an ASM issue and this is the ASM module. If I cant get the version of ASM I cant install the secrets. I did open a support ticket with them and they said I still need the secrets, with managed control planes. So how else can get the version of ASM? I really don't want to fork this module, can you export the version to a file within TF? That way I can use it from there. Thanks!

Purpose of Secrets
A: The purpose of secrets is for use in multi-cluster setups. The secret is used so that istioctl can access a remote cluster's API Server.

[morgante]

@morgante You are correct this is an ASM issue and this is the ASM module. If I cant get the version of ASM I cant install the secrets. I did open a support ticket with them and they said I still need the secrets, with managed control planes.

I'm still confused by you need the asmv label to create the secrets. You can set the ASM version with the asm_version variable. Moreover, I think you should really consider hardcoding your istioctl download.

I did open a support ticket with them and they said I still need the secrets.

Please share your case number (you can email it to morgantep@google.com).

[bjornsen]

I found an easier workaround. Terraform's ignore_changes allows you to ignore specific keys in maps. When you declare your GKE cluster, ignore resource_labels["asmv"] and resource_labels["mesh_id"] and Terraform will preserve them on subsequent runs.

[morgante]

@bjhshadow That's a great workaround. i didn't realize you can ignore a specific key in a map, this would make sense to add.

[richiefrich]

I know this is closed and I see it merged in version 17.2.0. but its still removing them for me. Which is odd.

[morgante]

@richiefrich This hasn't been released yet. It will go in the next release: #1063

[richiefrich]

@morgante thanks