ASM 1.10 script is failing to validate when istio-system namespace doesn't exist. #831

giulianobr · 2021-07-22T08:10:27Z

Hi there, it seems the #638 is back in the 1.10 script installation.
I'm doing a new setup with ASM 1.10 + terraform and failing because of the same describe in the #638
Would be nice to have the fix merged in the 1.10 script.

Thanks,

zerobfd · 2021-07-22T16:51:35Z

Hello! Thanks for reporting.

We took a preliminary look at this yesterday and the commit that fixed it in previous versions is also in 1.10, so we're still looking to see what could be causing it. If you have any output from the TF run it would be useful to help debug.

zerobfd · 2021-07-27T20:35:32Z

Oh, does this mean that using --only-validate fails?

giulianobr · 2021-07-27T21:37:35Z

Hi there, sorry my delay. Tomorrow I'll do a new installation, so I can provide more details.
Thanks!

giulianobr · 2021-07-28T09:17:11Z

Hi all, here is the error log:

module.asm-gke.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + rm -rf /tmp/kubectl_wrapper_27061_5623

Error: local-exec provisioner error

  with module.asm-gke.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0],
  on .terraform/modules/asm-gke.asm_install/main.tf line 231, in resource "null_resource" "run_command":
 231:   provisioner "local-exec" {

Error running command 'PATH=/google-cloud-sdk/bin:$PATH
.terraform/modules/asm-gke.asm_install/modules/kubectl-wrapper/scripts/kubectl_wrapper.sh MY_GKE_CLUSTER_ID europe-west1 MY_PROJECT_ID false false
.terraform/modules/asm-gke/modules/asm/scripts/install_asm.sh MY_PROJECT_ID MY_GKE_CLUSTER_ID europe-west1 1.10 install false false cloud-tracing
./ingress-backendconfig-operator.yaml false true true true true none meshca none none none none tf-local@MY_PROJECT_ID.iam.gserviceaccount.com sa.json none
': exit status 2. Output: ARNING: version difference between client (1.21) and server (1.19) exceeds the supported minor version skew of +/-1
install_asm_1.10: Checking Istio installations...
install_asm_1.10: Running: '/usr/local/bin/kubectl --kubeconfig asm_kubeconfig get deployment -A --ignore-not-found=true'
install_asm_1.10: -------------
install_asm_1.10: Running: '/usr/local/bin/kubectl --kubeconfig asm_kubeconfig get deployment -n istio-system --ignore-not-found=true'
install_asm_1.10: -------------
install_asm_1.10: Running: '/usr/local/bin/kubectl --kubeconfig asm_kubeconfig get deployment -n istio-system --ignore-not-found=true'
install_asm_1.10: -------------
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud services list --enabled --format=get(config.name) --project=MY_PROJECT_ID'
install_asm_1.10: -------------
install_asm_1.10: Checking required APIs...
install_asm_1.10: Running: '/usr/local/bin/kubectl --kubeconfig asm_kubeconfig api-resources --api-group=hub.gke.io'
install_asm_1.10: -------------
error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud container clusters describe --project=MY_PROJECT_ID --region europe-west1
MY_GKE_CLUSTER_ID --format=json'
install_asm_1.10: -------------
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud container clusters describe MY_GKE_CLUSTER_ID --zone=europe-west1
--project=MY_PROJECT_ID --format=value(selfLink, network)'
install_asm_1.10: -------------
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud container hub memberships list --format=value(name) --project MY_PROJECT_ID'
install_asm_1.10: -------------
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud container hub memberships list --format=value(name) --project MY_PROJECT_ID'
install_asm_1.10: -------------
install_asm_1.10: Registering the cluster as MY_GKE_CLUSTER_ID...
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud beta container hub memberships register MY_GKE_CLUSTER_ID --project=MY_PROJECT_ID
--gke-uri=https://container.googleapis.com/v1/projects/MY_PROJECT_ID/locations/europe-west1/clusters/MY_GKE_CLUSTER_ID --enable-workload-identity'
install_asm_1.10: -------------
kubeconfig entry generated for MY_GKE_CLUSTER_ID.
Waiting for membership to be created...
.......................done.
Created a new membership [projects/MY_PROJECT_ID/locations/global/memberships/MY_GKE_CLUSTER_ID] for the cluster [MY_GKE_CLUSTER_ID]
Generating the Connect Agent manifest...
Deploying the Connect Agent on cluster [MY_GKE_CLUSTER_ID] in namespace [gke-connect]...
Deployed the Connect Agent on cluster [MY_GKE_CLUSTER_ID] in namespace [gke-connect].
Finished registering the cluster [MY_GKE_CLUSTER_ID] with the Hub.
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud projects get-iam-policy MY_PROJECT_ID --flatten=bindings[].members
--filter=bindings.members:serviceAccount:tf-local@MY_PROJECT_ID.iam.gserviceaccount.com --format=value(bindings.role)'
install_asm_1.10: -------------
install_asm_1.10: Checking for project MY_PROJECT_ID...
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud projects describe MY_PROJECT_ID --format=value(projectNumber)'
install_asm_1.10: -------------
install_asm_1.10: Reading labels for europe-west1/MY_GKE_CLUSTER_ID...
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud container clusters describe MY_GKE_CLUSTER_ID --zone=europe-west1
--project=MY_PROJECT_ID --format=value(resourceLabels)[delimiter=","]'
install_asm_1.10: -------------
install_asm_1.10: Adding labels to europe-west1/MY_GKE_CLUSTER_ID...
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud container clusters update MY_GKE_CLUSTER_ID --project=MY_PROJECT_ID
--zone=europe-west1 --update-labels=asmv=1-10-2-asm-3,mesh_id=proj-978270309481'
install_asm_1.10: -------------
Updating MY_GKE_CLUSTER_ID...
.........................done.
Updated [https://container.googleapis.com/v1/projects/MY_PROJECT_ID/zones/europe-west1/clusters/MY_GKE_CLUSTER_ID].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/europe-west1/MY_GKE_CLUSTER_ID?project=MY_PROJECT_ID
install_asm_1.10: Initializing meshconfig API...
install_asm_1.10: Running: 'curl --request POST --fail --data  -o /dev/null https://meshconfig.googleapis.com/v1alpha1/projects/MY_PROJECT_ID:initialize -K /dev/fd/63'
install_asm_1.10: -------------
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud --project=MY_PROJECT_ID auth print-access-token'
install_asm_1.10: -------------
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     3    0     3    0     0      4      0 --:--:-- --:--:-- --:--:--     4
install_asm_1.10: Enabling Stackdriver on europe-west1/MY_GKE_CLUSTER_ID...
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud container clusters update MY_GKE_CLUSTER_ID --project=MY_PROJECT_ID
--zone=europe-west1 --enable-stackdriver-kubernetes'
install_asm_1.10: -------------
Updating MY_GKE_CLUSTER_ID...
........................done.
Updated [https://container.googleapis.com/v1/projects/MY_PROJECT_ID/zones/europe-west1/clusters/MY_GKE_CLUSTER_ID].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/europe-west1/MY_GKE_CLUSTER_ID?project=MY_PROJECT_ID
install_asm_1.10: Querying for core/account...
install_asm_1.10: Running: '/Users/giulianoribeiro/Developer/google-cloud-sdk/bin/gcloud config get-value core/account'
install_asm_1.10: -------------
Your active configuration is: [devoteam]
install_asm_1.10: Binding tf-local@MY_PROJECT_ID.iam.gserviceaccount.com to cluster admin role...
install_asm_1.10: Running: '/usr/local/bin/kubectl --kubeconfig asm_kubeconfig create clusterrolebinding tf-local-cluster-admin-binding --clusterrole=cluster-admin
--user=tf-local@MY_PROJECT_ID.iam.gserviceaccount.com --dry-run -o yaml'
install_asm_1.10: -------------
W0728 09:55:27.024740   10009 helpers.go:557] --dry-run is deprecated and can be replaced with --dry-run=client.
install_asm_1.10: Running: '/usr/local/bin/kubectl --kubeconfig asm_kubeconfig apply -f -'
install_asm_1.10: -------------
clusterrolebinding.rbac.authorization.k8s.io/tf-local-cluster-admin-binding created
install_asm_1.10: Checking for istio-system namespace...
install_asm_1.10: Running: '/usr/local/bin/kubectl --kubeconfig asm_kubeconfig get ns'
install_asm_1.10: -------------
install_asm_1.10: [ERROR]: The istio-system namespace doesn't exist.
Please create the "istio-namespace" and retry, or run the script with the
'--enable_namespace_creation' flag to allow the script to enable it on your behalf.
Alternatively, use --enable_all|-e to allow this tool to handle all dependencies.
+ cleanup
+ rm -rf /tmp/kubectl_wrapper_27061_5623

giulianobr · 2021-07-28T09:17:59Z

That service account has these roles:

ROLE
roles/container.admin
roles/editor
roles/gkehub.admin
roles/gkehub.gatewayAdmin
roles/gkehub.viewer
roles/iam.serviceAccountAdmin
roles/logging.logWriter
roles/meshconfig.admin
roles/monitoring.metricWriter
roles/serviceusage.serviceUsageConsumer

giulianobr · 2021-07-28T09:19:27Z

The ASM module:

module "asm-gke" {
  source                = "terraform-google-modules/kubernetes-engine/google//modules/asm"
  version               = "15.0.1"
  asm_version           = var.asm_version
  project_id            = var.project_id
  cluster_name          = module.gke.name
  location              = module.gke.location
  cluster_endpoint      = module.gke.endpoint
  enable_all            = false
  enable_gcp_apis       = true
  enable_gcp_components = true
  enable_cluster_labels = true
  enable_cluster_roles  = true
  enable_registration   = true
  enable_gcp_iam_roles  = false
  options               = ["cloud-tracing"]
  custom_overlays       = ["./ingress-backendconfig-operator.yaml"]
}

ZhengzheYang · 2021-08-05T20:57:19Z

add enable_namespace_creation flag and start testing with 1.10 #968
is merged. Please try with an additional flag enable_namespace_creation = true in the ASM module. Let me know if it does not work. Thanks!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ASM 1.10 script is failing to validate when istio-system namespace doesn't exist. #831

ASM 1.10 script is failing to validate when istio-system namespace doesn't exist. #831

giulianobr commented Jul 22, 2021

zerobfd commented Jul 22, 2021

zerobfd commented Jul 27, 2021

giulianobr commented Jul 27, 2021

giulianobr commented Jul 28, 2021

giulianobr commented Jul 28, 2021

giulianobr commented Jul 28, 2021

ZhengzheYang commented Aug 5, 2021

ASM 1.10 script is failing to validate when istio-system namespace doesn't exist. #831

ASM 1.10 script is failing to validate when istio-system namespace doesn't exist. #831

Comments

giulianobr commented Jul 22, 2021

zerobfd commented Jul 22, 2021

zerobfd commented Jul 27, 2021

giulianobr commented Jul 27, 2021

giulianobr commented Jul 28, 2021

giulianobr commented Jul 28, 2021

giulianobr commented Jul 28, 2021

ZhengzheYang commented Aug 5, 2021