Merge branch 'master' into mn/chore/promote-config-connector-ga

README.md

@@ -131,7 +131,7 @@ Then perform the following commands on the root folder:
 | add\_master\_webhook\_firewall\_rules | Create master\_webhook firewall rules for ports defined in `firewall_inbound_ports` | `bool` | `false` | no |
 | add\_shadow\_firewall\_rules | Create GKE shadow firewall (the same as default firewall rules with firewall logs enabled). | `bool` | `false` | no |
 | authenticator\_security\_group | The name of the RBAC security group for use with Google security groups in Kubernetes RBAC. Group name must be in format gke-security-groups@yourdomain.com | `string` | `null` | no |
-| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br>    enabled       = bool<br>    min_cpu_cores = number<br>    max_cpu_cores = number<br>    min_memory_gb = number<br>    max_memory_gb = number<br>    gpu_resources = list(object({ resource_type = string, minimum = number, maximum = number }))<br>  })</pre> | <pre>{<br>  "enabled": false,<br>  "gpu_resources": [],<br>  "max_cpu_cores": 0,<br>  "max_memory_gb": 0,<br>  "min_cpu_cores": 0,<br>  "min_memory_gb": 0<br>}</pre> | no |
+| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br>    enabled       = bool<br>    min_cpu_cores = number<br>    max_cpu_cores = number<br>    min_memory_gb = number<br>    max_memory_gb = number<br>    gpu_resources = list(object({ resource_type = string, minimum = number, maximum = number }))<br>    auto_repair   = bool<br>    auto_upgrade  = bool<br>  })</pre> | <pre>{<br>  "auto_repair": true,<br>  "auto_upgrade": true,<br>  "enabled": false,<br>  "gpu_resources": [],<br>  "max_cpu_cores": 0,<br>  "max_memory_gb": 0,<br>  "min_cpu_cores": 0,<br>  "min_memory_gb": 0<br>}</pre> | no |
 | cluster\_dns\_domain | The suffix used for all cluster service records. | `string` | `""` | no |
 | cluster\_dns\_provider | Which in-cluster DNS provider should be used. PROVIDER\_UNSPECIFIED (default) or PLATFORM\_DEFAULT or CLOUD\_DNS. | `string` | `"PROVIDER_UNSPECIFIED"` | no |
 | cluster\_dns\_scope | The scope of access to cluster DNS records. DNS\_SCOPE\_UNSPECIFIED (default) or CLUSTER\_SCOPE or VPC\_SCOPE. | `string` | `"DNS_SCOPE_UNSPECIFIED"` | no |

autogen/main/cluster.tf.tmpl

@@ -135,6 +135,12 @@ resource "google_container_cluster" "primary" {
       content {
         service_account = local.service_account
         oauth_scopes = local.node_pools_oauth_scopes["all"]
+
+        management {
+            auto_repair  = lookup(var.cluster_autoscaling, "auto_repair", true)
+            auto_upgrade = lookup(var.cluster_autoscaling, "auto_upgrade",true)
+        }
+
         {% if beta_cluster %}
         min_cpu_platform = lookup(var.node_pools[0], "min_cpu_platform", "")
         {% endif %}
@@ -156,7 +162,7 @@ resource "google_container_cluster" "primary" {
   {% if autopilot_cluster == true %}
   cluster_autoscaling {
     dynamic "auto_provisioning_defaults" {
-      for_each = var.create_service_account ? [1] : []
+      for_each = (var.create_service_account || var.service_account != "") ? [1] : []
 
       content {
         service_account = local.service_account

autogen/main/main.tf.tmpl

@@ -68,7 +68,7 @@ locals {
     resource_type = "memory"
     minimum       = var.cluster_autoscaling.min_memory_gb
     maximum       = var.cluster_autoscaling.max_memory_gb
-  }], var.cluster_autoscaling.gpu_resources) : []
+  }], var.cluster_autoscaling.gpu_resources) : []  
   {% endif %}
 
 

autogen/main/variables.tf.tmpl

@@ -241,6 +241,8 @@ variable "cluster_autoscaling" {
     min_memory_gb       = number
     max_memory_gb       = number
     gpu_resources       = list(object({ resource_type = string, minimum = number, maximum = number }))
+    auto_repair         = bool
+    auto_upgrade        = bool
   })
   default = {
     enabled             = false
@@ -252,6 +254,8 @@ variable "cluster_autoscaling" {
     max_memory_gb       = 0
     min_memory_gb       = 0
     gpu_resources       = []
+    auto_repair         = true
+    auto_upgrade        = true
   }
   description = "Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling)"
 }

autogen/safer-cluster/variables.tf.tmpl

@@ -233,6 +233,8 @@ variable "cluster_autoscaling" {
     min_memory_gb       = number
     max_memory_gb       = number
     gpu_resources       = list(object({ resource_type = string, minimum = number, maximum = number }))
+    auto_repair         = bool
+    auto_upgrade        = bool
   })
   default = {
     enabled             = false
@@ -242,6 +244,8 @@ variable "cluster_autoscaling" {
     max_memory_gb       = 0
     min_memory_gb       = 0
     gpu_resources       = []
+    auto_repair         = true
+    auto_upgrade        = true
   }
   description = "Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling)"
 }

cluster.tf

@@ -104,6 +104,12 @@ resource "google_container_cluster" "primary" {
       content {
         service_account = local.service_account
         oauth_scopes    = local.node_pools_oauth_scopes["all"]
+
+        management {
+          auto_repair  = lookup(var.cluster_autoscaling, "auto_repair", true)
+          auto_upgrade = lookup(var.cluster_autoscaling, "auto_upgrade", true)
+        }
+
       }
     }
     dynamic "resource_limits" {

examples/deploy_service/main.tf

@@ -52,7 +52,7 @@ resource "kubernetes_pod" "nginx-example" {
 
   spec {
     container {
-      image = "nginx:1.23.2"
+      image = "nginx:1.23.3"
       name  = "nginx-example"
     }
   }

examples/node_pool/README.md

@@ -7,7 +7,7 @@ This example illustrates how to create a cluster with multiple custom node-pool
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
-| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br>    enabled             = bool<br>    autoscaling_profile = string<br>    min_cpu_cores       = number<br>    max_cpu_cores       = number<br>    min_memory_gb       = number<br>    max_memory_gb       = number<br>    gpu_resources = list(object({<br>      resource_type = string<br>      minimum       = number<br>      maximum       = number<br>    }))<br>  })</pre> | <pre>{<br>  "autoscaling_profile": "BALANCED",<br>  "enabled": false,<br>  "gpu_resources": [],<br>  "max_cpu_cores": 0,<br>  "max_memory_gb": 0,<br>  "min_cpu_cores": 0,<br>  "min_memory_gb": 0<br>}</pre> | no |
+| cluster\_autoscaling | Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling) | <pre>object({<br>    enabled             = bool<br>    autoscaling_profile = string<br>    min_cpu_cores       = number<br>    max_cpu_cores       = number<br>    min_memory_gb       = number<br>    max_memory_gb       = number<br>    gpu_resources = list(object({<br>      resource_type = string<br>      minimum       = number<br>      maximum       = number<br>    }))<br>    auto_repair  = bool<br>    auto_upgrade = bool<br>  })</pre> | <pre>{<br>  "auto_repair": true,<br>  "auto_upgrade": true,<br>  "autoscaling_profile": "BALANCED",<br>  "enabled": false,<br>  "gpu_resources": [],<br>  "max_cpu_cores": 0,<br>  "max_memory_gb": 0,<br>  "min_cpu_cores": 0,<br>  "min_memory_gb": 0<br>}</pre> | no |
 | cluster\_name\_suffix | A suffix to append to the default cluster name | `string` | `""` | no |
 | compute\_engine\_service\_account | Service account to associate to the nodes in the cluster | `any` | n/a | yes |
 | ip\_range\_pods | The secondary ip range to use for pods | `any` | n/a | yes |

examples/node_pool/variables.tf

@@ -65,6 +65,8 @@ variable "cluster_autoscaling" {
       minimum       = number
       maximum       = number
     }))
+    auto_repair  = bool
+    auto_upgrade = bool
   })
   default = {
     enabled             = false
@@ -74,6 +76,8 @@ variable "cluster_autoscaling" {
     max_memory_gb       = 0
     min_memory_gb       = 0
     gpu_resources       = []
+    auto_repair         = true
+    auto_upgrade        = true
   }
   description = "Cluster autoscaling configuration. See [more details](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters#clusterautoscaling)"
 }

examples/simple_zonal_with_acm/README.md

@@ -1,6 +1,6 @@
 # Simple Zonal Cluster
 
-This example illustrates how to create a simple cluster and install [Anthos Config Management](https://cloud.google.com/anthos-config-management/docs/).
+This example illustrates how to create a simple cluster and install [Anthos Config Management](https://cloud.google.com/anthos-config-management/docs/)'s [Config Sync](https://cloud.google.com/anthos-config-management/docs/config-sync-overview) and [Policy Controller](https://cloud.google.com/anthos-config-management/docs/concepts/policy-controller) with the [Policy Essentials v2022 policy bundle](https://cloud.google.com/anthos-config-management/docs/how-to/using-policy-essentials-v2022).
 
 It incorporates the standard cluster module and the [ACM install module](../../modules/acm).
 
@@ -27,13 +27,19 @@ After applying the Terraform configuration, you can run the following commands t
     kubectl describe ns shipping-dev
     ```
 
+4. You can also use `kubectl` to view any policy violations on the cluster:
+
+    ```
+    kubectl get constraint -l policycontroller.gke.io/bundleName=policy-essentials-v2022 -o json | jq -cC '.items[]| [.metadata.name,.status.totalViolations]'
+    ```
+
 <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | cluster\_name\_suffix | A suffix to append to the default cluster name | `string` | `""` | no |
-| project\_id | The project ID to host the cluster in | `any` | n/a | yes |
+| project\_id | The project ID to host the cluster in | `string` | n/a | yes |
 | region | The region to host the cluster in | `string` | `"us-central1"` | no |
 | zone | The zone to host the cluster in | `string` | `"us-central1-a"` | no |
 

examples/simple_zonal_with_acm/acm.tf

@@ -25,4 +25,8 @@ module "acm" {
   policy_dir  = "foo-corp"
 
   secret_type = "ssh"
+
+  policy_bundles = ["https://github.com/GoogleCloudPlatform/acm-policy-controller-library/bundles/policy-essentials-v2022#e4094aacb91a35b0219f6f4cf6a31580e85b3c28"]
+
+  create_metrics_gcp_sa = true
 }

examples/simple_zonal_with_acm/variables.tf

@@ -16,15 +16,18 @@
 
 variable "project_id" {
   description = "The project ID to host the cluster in"
+  type        = string
 }
 
 variable "cluster_name_suffix" {
   description = "A suffix to append to the default cluster name"
+  type        = string
   default     = ""
 }
 
 variable "region" {
   description = "The region to host the cluster in"
+  type        = string
   default     = "us-central1"
 }
 

examples/simple_zonal_with_acm/versions.tf

@@ -21,7 +21,12 @@ terraform {
       version = "~> 4.0"
     }
     kubernetes = {
-      source = "hashicorp/kubernetes"
+      source  = "hashicorp/kubernetes"
+      version = "~> 2.10"
+    }
+    random = {
+      source  = "hashicorp/random"
+      version = ">= 2.1"
     }
   }
   required_version = ">= 0.13"

modules/acm/README.md

@@ -3,10 +3,12 @@
 This module installs [Anthos Config Management](https://cloud.google.com/anthos-config-management/docs/) (ACM) in a Kubernetes cluster.
 
 Specifically, this module automates the following steps for [installing ACM](https://cloud.google.com/anthos-config-management/docs/how-to/installing):
-1. Enabling the ACM feature on the fleet.
+1. Enabling the ACM feature on the fleet
 2. Registering the cluster to the fleet
 3. Optionally, generating an SSH key for accessing Git and providing it to the Operator
 4. Configuring the ACM feature on your cluster
+5. Optionally, installing ACM Policy Controller [Policy Bundle(s)](https://cloud.google.com/anthos-config-management/docs/concepts/policy-controller-bundles)
+6. Optionally, create and configure a Google Cloud Service Account for writing ACM metrics to Cloud Monitoring
 
 ## Fleet feature
 Only the first cluster in a fleet should activate the ACM fleet feature.
@@ -35,6 +37,11 @@ module "acm" {
   sync_repo        = "git@github.com:GoogleCloudPlatform/anthos-config-management-samples.git"
   sync_branch      = "1.0.0"
   policy_dir       = "foo-corp"
+
+  # ACM Policy Controller Policy Essentials Policy Bundle: https://cloud.google.com/anthos-config-management/docs/how-to/using-policy-essentials-v2022
+  policy_bundles = ["https://github.com/GoogleCloudPlatform/acm-policy-controller-library/bundles/policy-essentials-v2022#e4094aacb91a35b0219f6f4cf6a31580e85b3c28"]
+
+  create_metrics_gcp_sa = true
 }
 ```
 
@@ -67,7 +74,9 @@ data "google_client_config" "default" {}
 | cluster\_membership\_id | The cluster membership ID. If unset, one will be autogenerated. | `string` | `""` | no |
 | cluster\_name | GCP cluster Name used to reach cluster and which becomes the cluster name in the Config Sync kubernetes custom resource. | `string` | n/a | yes |
 | configmanagement\_version | Version of ACM. | `string` | `""` | no |
+| create\_metrics\_gcp\_sa | Create a Google service account for ACM metrics writing | `bool` | `false` | no |
 | create\_ssh\_key | Controls whether a key will be generated for Git authentication | `bool` | `true` | no |
+| enable\_config\_sync | Whether to enable the ACM Config Sync on the cluster | `bool` | `true` | no |
 | enable\_fleet\_feature | Whether to enable the ACM feature on the fleet. | `bool` | `true` | no |
 | enable\_fleet\_registration | Whether to create a new membership. | `bool` | `true` | no |
 | enable\_log\_denies | Whether to enable logging of all denies and dryrun failures for ACM Policy Controller. | `bool` | `false` | no |
@@ -77,19 +86,22 @@ data "google_client_config" "default" {}
 | https\_proxy | URL for the HTTPS proxy to be used when communicating with the Git repo. | `string` | `null` | no |
 | install\_template\_library | Whether to install the default Policy Controller template library | `bool` | `true` | no |
 | location | GCP location used to reach cluster. | `string` | n/a | yes |
+| metrics\_gcp\_sa\_name | The name of the Google service account for ACM metrics writing | `string` | `"acm-metrics-writer"` | no |
+| policy\_bundles | A list of Policy Controller policy bundles git urls (example: https://github.com/GoogleCloudPlatform/acm-policy-controller-library.git/bundles/policy-essentials-v2022) to install on the cluster. | `list(string)` | `[]` | no |
 | policy\_dir | Subfolder containing configs in ACM Git repo. If un-set, uses Config Management default. | `string` | `""` | no |
 | project\_id | GCP project\_id used to reach cluster. | `string` | n/a | yes |
 | secret\_type | git authentication secret type, is passed through to ConfigManagement spec.git.secretType. Overriden to value 'ssh' if `create_ssh_key` is true | `string` | `"ssh"` | no |
 | source\_format | Configures a non-hierarchical repo if set to 'unstructured'. Uses [ACM defaults](https://cloud.google.com/anthos-config-management/docs/how-to/installing#configuring-config-management-operator) when unset. | `string` | `""` | no |
 | ssh\_auth\_key | Key for Git authentication. Overrides 'create\_ssh\_key' variable. Can be set using 'file(path/to/file)'-function. | `string` | `null` | no |
 | sync\_branch | ACM repo Git branch. If un-set, uses Config Management default. | `string` | `""` | no |
-| sync\_repo | ACM Git repo address | `string` | n/a | yes |
+| sync\_repo | ACM Git repo address | `string` | `""` | no |
 | sync\_revision | ACM repo Git revision. If un-set, uses Config Management default. | `string` | `""` | no |
 
 ## Outputs
 
 | Name | Description |
 |------|-------------|
+| acm\_metrics\_writer\_sa | The ACM metrics writer Service Account |
 | configmanagement\_version | Version of ACM installed. |
 | git\_creds\_public | Public key of SSH keypair to allow the Anthos Config Management Operator to authenticate to your Git repository. |
 | wait | An output to use when you want to depend on cmd finishing |