Skip to content

Commit

Permalink
rebase
Browse files Browse the repository at this point in the history
  • Loading branch information
bberriot committed Dec 29, 2022
2 parents 5da662a + ac062f8 commit 8786814
Show file tree
Hide file tree
Showing 47 changed files with 571 additions and 116 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/stale.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
stale:
runs-on: ubuntu-latest
steps:
- uses: actions/stale@v6
- uses: actions/stale@v7
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
stale-issue-message: 'This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days'
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,7 @@ Then perform the following commands on the root folder:
| filestore\_csi\_driver | The status of the Filestore CSI driver addon, which allows the usage of filestore instance as volumes | `bool` | `false` | no |
| firewall\_inbound\_ports | List of TCP ports for admission/webhook controllers. Either flag `add_master_webhook_firewall_rules` or `add_cluster_firewall_rules` (also adds egress rules) must be set to `true` for inbound-ports firewall rules to be applied. | `list(string)` | <pre>[<br> "8443",<br> "9443",<br> "15017"<br>]</pre> | no |
| firewall\_priority | Priority rule for firewall rules | `number` | `1000` | no |
| gce\_pd\_csi\_driver | Whether this cluster should enable the Google Compute Engine Persistent Disk Container Storage Interface (CSI) Driver. | `bool` | `true` | no |
| grant\_registry\_access | Grants created cluster-specific service account storage.objectViewer and artifactregistry.reader roles. | `bool` | `false` | no |
| horizontal\_pod\_autoscaling | Enable horizontal pod autoscaling addon | `bool` | `true` | no |
| http\_load\_balancing | Enable httpload balancer addon | `bool` | `true` | no |
Expand Down Expand Up @@ -282,6 +283,7 @@ The node_pools variable takes the following parameters:
| tags | The list of instance tags applied to all nodes | | Required |
| value | The value for the taint | | Required |
| version | The Kubernetes version for the nodes in this pool. Should only be set if auto_upgrade is false | " " | Optional |
| location_policy | [Location policy](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/container_node_pool#location_policy) specifies the algorithm used when scaling-up the node pool. Location policy is supported only in 1.24.1+ clusters. | " " | Optional |

## windows_node_pools variable
The windows_node_pools variable takes the same parameters as [node_pools](#node\_pools-variable) but is reserved for provisioning Windows based node pools only. This variable is introduced to satisfy a [specific requirement](https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-cluster-windows#create_a_cluster_and_node_pools) for the presence of at least one linux based node pool in the cluster before a windows based node pool can be created.
Expand Down
1 change: 1 addition & 0 deletions autogen/main/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,7 @@ The node_pools variable takes the following parameters:
| tags | The list of instance tags applied to all nodes | | Required |
| value | The value for the taint | | Required |
| version | The Kubernetes version for the nodes in this pool. Should only be set if auto_upgrade is false | " " | Optional |
| location_policy | [Location policy](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/container_node_pool#location_policy) specifies the algorithm used when scaling-up the node pool. Location policy is supported only in 1.24.1+ clusters. | " " | Optional |

## windows_node_pools variable
The windows_node_pools variable takes the same parameters as [node_pools](#node\_pools-variable) but is reserved for provisioning Windows based node pools only. This variable is introduced to satisfy a [specific requirement](https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-cluster-windows#create_a_cluster_and_node_pools) for the presence of at least one linux based node pool in the cluster before a windows based node pool can be created.
Expand Down
28 changes: 19 additions & 9 deletions autogen/main/cluster.tf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,17 @@ resource "google_container_cluster" "primary" {
}
}
{% endif %}
{% if autopilot_cluster == true %}
cluster_autoscaling {
dynamic "auto_provisioning_defaults" {
for_each = var.create_service_account ? [1] : []

content {
service_account = local.service_account
}
}
}
{% endif %}
vertical_pod_autoscaling {
enabled = var.enable_vertical_pod_autoscaling
}
Expand Down Expand Up @@ -210,7 +221,6 @@ resource "google_container_cluster" "primary" {
disabled = !var.horizontal_pod_autoscaling
}


{% if autopilot_cluster != true %}
network_policy_config {
disabled = !var.network_policy
Expand All @@ -223,6 +233,14 @@ resource "google_container_cluster" "primary" {
gcp_filestore_csi_driver_config {
enabled = var.filestore_csi_driver
}

dynamic "gce_persistent_disk_csi_driver_config" {
for_each = local.cluster_gce_pd_csi_config

content {
enabled = gce_persistent_disk_csi_driver_config.value.enabled
}
}
{% endif %}
{% if beta_cluster and autopilot_cluster != true %}

Expand All @@ -239,14 +257,6 @@ resource "google_container_cluster" "primary" {
}
}

dynamic "gce_persistent_disk_csi_driver_config" {
for_each = local.cluster_gce_pd_csi_config

content {
enabled = gce_persistent_disk_csi_driver_config.value.enabled
}
}

kalm_config {
enabled = var.kalm_config
}
Expand Down
2 changes: 1 addition & 1 deletion autogen/main/main.tf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ locals {
enabled = false
provider = null
}]
cluster_gce_pd_csi_config = var.gce_pd_csi_driver ? [{ enabled = true }] : [{ enabled = false }]
{% endif %}
{% if beta_cluster and autopilot_cluster != true %}
cluster_cloudrun_config_load_balancer_config = (var.cloudrun && var.cloudrun_load_balancer_type != "") ? {
Expand All @@ -109,7 +110,6 @@ locals {
)
] : []
cluster_cloudrun_enabled = var.cloudrun
cluster_gce_pd_csi_config = var.gce_pd_csi_driver ? [{ enabled = true }] : [{ enabled = false }]
gke_backup_agent_config = var.gke_backup_agent_config ? [{ enabled = true }] : [{ enabled = false }]
{% endif %}
logmon_config_is_set = length(var.logging_enabled_components) > 0 || length(var.monitoring_enabled_components) > 0 || var.monitoring_enable_managed_prometheus
Expand Down
12 changes: 6 additions & 6 deletions autogen/main/variables.tf.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -603,6 +603,12 @@ variable "cluster_dns_domain" {
default = ""
}

variable "gce_pd_csi_driver" {
type = bool
description = "Whether this cluster should enable the Google Compute Engine Persistent Disk Container Storage Interface (CSI) Driver."
default = true
}

{% endif %}
variable "timeouts" {
type = map(string)
Expand Down Expand Up @@ -712,11 +718,5 @@ variable "enable_identity_service" {
description = "Enable the Identity Service component, which allows customers to use external identity providers with the K8S API."
default = false
}

variable "gce_pd_csi_driver" {
type = bool
description = "(Beta) Whether this cluster should enable the Google Compute Engine Persistent Disk Container Storage Interface (CSI) Driver."
default = false
}
{% endif %}
{% endif %}
23 changes: 20 additions & 3 deletions build/int.cloudbuild.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -401,9 +401,26 @@ steps:
- verify private-zonal-with-networking
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
args: ['/bin/bash', '-c', 'cft test run TestPrivateZonalWithNetworking --stage teardown --verbose --test-dir test/integration']



- id: init simple-autopilot-private-non-default-sa
waitFor:
- prepare
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
args: ['/bin/bash', '-c', 'cft test run TestSimpleAutopilotPrivateNonDefaultSA --stage init --verbose']
- id: apply simple-autopilot-private-non-default-sa
waitFor:
- init simple-autopilot-private-non-default-sa
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
args: ['/bin/bash', '-c', 'cft test run TestSimpleAutopilotPrivateNonDefaultSA --stage apply --verbose']
- id: verify simple-autopilot-private-non-default-sa
waitFor:
- apply simple-autopilot-private-non-default-sa
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
args: ['/bin/bash', '-c', 'cft test run TestSimpleAutopilotPrivateNonDefaultSA --stage verify --verbose']
- id: teardown simple-autopilot-private-non-default-sa
waitFor:
- verify simple-autopilot-private-non-default-sa
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
args: ['/bin/bash', '-c', 'cft test run TestSimpleAutopilotPrivateNonDefaultSA --stage teardown --verbose']
tags:
- 'ci'
- 'integration'
Expand Down
9 changes: 8 additions & 1 deletion cluster.tf
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,6 @@ resource "google_container_cluster" "primary" {
disabled = !var.horizontal_pod_autoscaling
}


network_policy_config {
disabled = !var.network_policy
}
Expand All @@ -166,6 +165,14 @@ resource "google_container_cluster" "primary" {
gcp_filestore_csi_driver_config {
enabled = var.filestore_csi_driver
}

dynamic "gce_persistent_disk_csi_driver_config" {
for_each = local.cluster_gce_pd_csi_config

content {
enabled = gce_persistent_disk_csi_driver_config.value.enabled
}
}
}

datapath_provider = var.datapath_provider
Expand Down
42 changes: 42 additions & 0 deletions docs/upgrading_to_v25.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Upgrading to v25.0
The v25.0 release of *kubernetes-engine* is a backwards incompatible
release.

### gce_pd_csi_driver is GA and enabled by default

`gce_pd_csi_driver` is now supported in GA modules and defaults to true. To opt out, set `gce_pd_csi_driver` to `false`.

```diff
module "gke" {
- source = "terraform-google-modules/kubernetes-engine"
- version = "~> 24.0"
+ source = "terraform-google-modules/kubernetes-engine"
+ version = "~> 25.0"
...
+ gce_pd_csi_driver = false
}
```

### Use the created service account when creating autopilot clusters

When `create_service_account` is `true` pass the created service account to the `cluster_autoscaling` -> `auto_provisioning_defaults` block
for the `beta-autopilot-private-cluster` / `beta-autopilot-public-cluster` modules.

This will mean that the `Nodes` will use the created service account, where previously the default service account was erronously used instead.

To opt out, set `create_service_account` to `false`

```diff
module "gke" {
- source = "terraform-google-modules/kubernetes-engine"
- version = "~> 24.0"
+ source = "terraform-google-modules/kubernetes-engine"
+ version = "~> 25.0"
...
+ create_service_account = false
}
```

### Minimum Google Provider versions

Minimum Google Provider versions have been updated to `4.44.0`.
33 changes: 33 additions & 0 deletions examples/simple_autopilot_private_non_default_sa/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Simple Regional Autopilot Cluster

This example illustrates how to create a simple autopilot cluster with beta features and
not using the default service account.

<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| project\_id | The project ID to host the cluster in | `any` | n/a | yes |

## Outputs

| Name | Description |
|------|-------------|
| cluster\_name | Cluster name |
| kubernetes\_endpoint | The cluster endpoint |
| location | n/a |
| master\_kubernetes\_version | Kubernetes version of the master |
| network\_name | The name of the VPC being created |
| region | The region in which the cluster resides |
| service\_account | The service account to default running nodes as if not overridden in `node_pools`. |
| subnet\_names | The names of the subnet being created |
| zones | List of zones in which the cluster resides |

<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

To provision this example, run the following from within this directory:
- `terraform init` to get the plugins
- `terraform plan` to see the infrastructure plan
- `terraform apply` to apply the infrastructure build
- `terraform destroy` to destroy the built infrastructure
59 changes: 59 additions & 0 deletions examples/simple_autopilot_private_non_default_sa/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

locals {
cluster_type = "simple-ap-private-non-default-sa"
network_name = "${local.cluster_type}-network"
subnet_name = "${local.cluster_type}-subnet"
master_auth_subnetwork = "${local.cluster_type}-master-subnet"
pods_range_name = "ip-range-pods-${local.cluster_type}"
svc_range_name = "ip-range-svc-${local.cluster_type}"
subnet_names = [for subnet_self_link in module.gcp-network.subnets_self_links : split("/", subnet_self_link)[length(split("/", subnet_self_link)) - 1]]
}


data "google_client_config" "default" {}

provider "kubernetes" {
host = "https://${module.gke.endpoint}"
token = data.google_client_config.default.access_token
cluster_ca_certificate = base64decode(module.gke.ca_certificate)
}

module "gke" {
source = "../../modules/beta-autopilot-private-cluster/"
project_id = var.project_id
name = "${local.cluster_type}-cluster"
regional = true
region = "us-central1"
network = module.gcp-network.network_name
subnetwork = local.subnet_names[index(module.gcp-network.subnets_names, local.subnet_name)]
ip_range_pods = local.pods_range_name
ip_range_services = local.svc_range_name
release_channel = "REGULAR"
enable_vertical_pod_autoscaling = true
enable_private_endpoint = true
enable_private_nodes = true
master_ipv4_cidr_block = "172.16.0.0/28"
datapath_provider = "ADVANCED_DATAPATH"

master_authorized_networks = [
{
cidr_block = "10.60.0.0/17"
display_name = "VPC"
},
]
}
50 changes: 50 additions & 0 deletions examples/simple_autopilot_private_non_default_sa/network.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
/**
* Copyright 2022 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

module "gcp-network" {
source = "terraform-google-modules/network/google"
version = ">= 4.0.1"

project_id = var.project_id
network_name = local.network_name

subnets = [
{
subnet_name = local.subnet_name
subnet_ip = "10.0.0.0/17"
subnet_region = "us-central1"
subnet_private_access = true
},
{
subnet_name = local.master_auth_subnetwork
subnet_ip = "10.60.0.0/17"
subnet_region = "us-central1"
},
]

secondary_ranges = {
(local.subnet_name) = [
{
range_name = local.pods_range_name
ip_cidr_range = "192.168.0.0/18"
},
{
range_name = local.svc_range_name
ip_cidr_range = "192.168.64.0/18"
},
]
}
}
Loading

0 comments on commit 8786814

Please sign in to comment.