From 570a15197156c43d91928da2950019a253cfc06e Mon Sep 17 00:00:00 2001 From: Farhad Sharabiani Date: Thu, 20 Jun 2024 03:21:21 +0000 Subject: [PATCH 1/4] Create 'pre-existing-gke-cluster' module --- .../pre-existing-gke-cluster/README.md | 88 +++++++++++++++++++ .../pre-existing-gke-cluster/main.tf | 21 +++++ .../pre-existing-gke-cluster/metadata.yaml | 19 ++++ .../pre-existing-gke-cluster/outputs.tf | 28 ++++++ .../pre-existing-gke-cluster/variables.tf | 30 +++++++ .../pre-existing-gke-cluster/versions.tf | 30 +++++++ modules/README.md | 3 + 7 files changed, 219 insertions(+) create mode 100644 community/modules/scheduler/pre-existing-gke-cluster/README.md create mode 100644 community/modules/scheduler/pre-existing-gke-cluster/main.tf create mode 100644 community/modules/scheduler/pre-existing-gke-cluster/metadata.yaml create mode 100644 community/modules/scheduler/pre-existing-gke-cluster/outputs.tf create mode 100644 community/modules/scheduler/pre-existing-gke-cluster/variables.tf create mode 100644 community/modules/scheduler/pre-existing-gke-cluster/versions.tf diff --git a/community/modules/scheduler/pre-existing-gke-cluster/README.md b/community/modules/scheduler/pre-existing-gke-cluster/README.md new file mode 100644 index 0000000000..50676ba8d6 --- /dev/null +++ b/community/modules/scheduler/pre-existing-gke-cluster/README.md @@ -0,0 +1,88 @@ +## Description + +This module discovers a Google Kubernetes Engine ([GKE](https://cloud.google.com/kubernetes-engine)) cluster that already exists in Google Cloud and +outputs cluster attributes that uniquely identify it for use by other modules. +The module outputs are aligned with the [gke-cluster module][gke-cluster] so that it can be used +as a drop-in substitute when a GKE cluster already exists. + +The below sample blueprint discovers the existing GKE cluster named "my-gke-cluster" in "us-central1" region. With the `use` keyword, the +[gke-node-pool] module accepts the `cluser_id` +input variable that uniquely identifies the existing GKE cluster in which the +GKE node pool will be created. + +[gke-cluster]: ../gke-cluster/README.md +[gke-node-pool]: ../../compute/gke-node-pool/README.md + +### Example + +```yaml +- id: existing-gke-cluster + source: community/modules/scheduler/pre-existing-gke-cluster + settings: + project_id: $(vars.project_id) + cluster_name: my-gke-cluster + region: us-central1 + +- id: compute_pool + source: community/modules/compute/gke-node-pool + use: [existing-gke-cluster] +``` + +> **_NOTE:_** The `project_id` and `region` settings would be inferred from the +> deployment variables of the same name, but they are included here for clarity. + +## License + + +Copyright 2024 Google LLC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 0.14.0 | +| [google](#requirement\_google) | > 5.0 | + +## Providers + +| Name | Version | +|------|---------| +| [google](#provider\_google) | > 5.0 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [google_container_cluster.existing_gke_cluster](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/container_cluster) | data source | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [cluster\_name](#input\_cluster\_name) | Name of the existing cluster | `string` | n/a | yes | +| [project\_id](#input\_project\_id) | Project that hosts the existing cluster | `string` | n/a | yes | +| [region](#input\_region) | Region in which to search for the cluster | `string` | n/a | yes | + +## Outputs + +| Name | Description | +|------|-------------| +| [cluster\_id](#output\_cluster\_id) | An identifier for the gke cluster with format projects//locations//clusters/. | +| [gke\_cluster\_exists](#output\_gke\_cluster\_exists) | A static flag that signals to downstream modules that a cluster exists. | + diff --git a/community/modules/scheduler/pre-existing-gke-cluster/main.tf b/community/modules/scheduler/pre-existing-gke-cluster/main.tf new file mode 100644 index 0000000000..c59e35e8da --- /dev/null +++ b/community/modules/scheduler/pre-existing-gke-cluster/main.tf @@ -0,0 +1,21 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +data "google_container_cluster" "existing_gke_cluster" { + name = var.cluster_name + project = var.project_id + location = var.region +} diff --git a/community/modules/scheduler/pre-existing-gke-cluster/metadata.yaml b/community/modules/scheduler/pre-existing-gke-cluster/metadata.yaml new file mode 100644 index 0000000000..17bedb471b --- /dev/null +++ b/community/modules/scheduler/pre-existing-gke-cluster/metadata.yaml @@ -0,0 +1,19 @@ +# Copyright 2024 "Google LLC" +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--- + +spec: + requirements: + services: + - container.googleapis.com diff --git a/community/modules/scheduler/pre-existing-gke-cluster/outputs.tf b/community/modules/scheduler/pre-existing-gke-cluster/outputs.tf new file mode 100644 index 0000000000..6166aa6bad --- /dev/null +++ b/community/modules/scheduler/pre-existing-gke-cluster/outputs.tf @@ -0,0 +1,28 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +output "cluster_id" { + description = "An identifier for the gke cluster with format projects//locations//clusters/." + value = data.google_container_cluster.existing_gke_cluster.id +} + +output "gke_cluster_exists" { + description = "A static flag that signals to downstream modules that a cluster exists." + value = true + depends_on = [ + data.google_container_cluster.existing_gke_cluster + ] +} diff --git a/community/modules/scheduler/pre-existing-gke-cluster/variables.tf b/community/modules/scheduler/pre-existing-gke-cluster/variables.tf new file mode 100644 index 0000000000..5d2121ba69 --- /dev/null +++ b/community/modules/scheduler/pre-existing-gke-cluster/variables.tf @@ -0,0 +1,30 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "project_id" { + description = "Project that hosts the existing cluster" + type = string +} + +variable "cluster_name" { + description = "Name of the existing cluster" + type = string +} + +variable "region" { + description = "Region in which to search for the cluster" + type = string +} diff --git a/community/modules/scheduler/pre-existing-gke-cluster/versions.tf b/community/modules/scheduler/pre-existing-gke-cluster/versions.tf new file mode 100644 index 0000000000..9083cd5299 --- /dev/null +++ b/community/modules/scheduler/pre-existing-gke-cluster/versions.tf @@ -0,0 +1,30 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +terraform { + required_providers { + google = { + source = "hashicorp/google" + version = "> 5.0" + } + } + + provider_meta "google" { + module_name = "blueprints/terraform/hpc-toolkit:pre-existing-gke-cluster/v1.35.0" + } + + required_version = ">= 0.14.0" +} diff --git a/modules/README.md b/modules/README.md index b9475195e5..bae9d14143 100644 --- a/modules/README.md +++ b/modules/README.md @@ -184,6 +184,8 @@ Pub/Sub subscription. Primarily used for [FSI - MonteCarlo Tutorial][fsi-monteca submission of Google Cloud Batch jobs. * **[gke-cluster]** ![community-badge] ![experimental-badge] : Creates a Kubernetes cluster using GKE. +* **[pre-existing-gke-cluster]** ![community-badge] ![experimental-badge] : Retrieves an existing + GKE cluster's attributes to be used in other modules as a substitute for creating a new cluster ([gke-cluster]). * **[schedmd-slurm-gcp-v5-controller]** ![community-badge] : Creates a Slurm controller node using [slurm-gcp-version-5]. * **[schedmd-slurm-gcp-v5-login]** ![community-badge] : @@ -210,6 +212,7 @@ Pub/Sub subscription. Primarily used for [FSI - MonteCarlo Tutorial][fsi-monteca [batch-job-template]: ../modules/scheduler/batch-job-template/README.md [batch-login-node]: ../modules/scheduler/batch-login-node/README.md [gke-cluster]: ../community/modules/scheduler/gke-cluster/README.md +[pre-existing-gke-cluster]: ../community/modules/scheduler/pre-existing-gke-cluster/README.md [htcondor-setup]: ../community/modules/scheduler/htcondor-setup/README.md [htcondor-pool-secrets]: ../community/modules/scheduler/htcondor-pool-secrets/README.md [htcondor-access-point]: ../community/modules/scheduler/htcondor-access-point/README.md From a9abdadcb5a6dd8ca819851caa6b80e9108622b8 Mon Sep 17 00:00:00 2001 From: Farhad Sharabiani Date: Tue, 25 Jun 2024 07:37:54 +0000 Subject: [PATCH 2/4] fixed dilimiters issue in some Readme files --- community/modules/scheduler/gke-cluster/README.md | 2 +- community/modules/scheduler/pre-existing-gke-cluster/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/community/modules/scheduler/gke-cluster/README.md b/community/modules/scheduler/gke-cluster/README.md index f98ce0fcc5..94e2dd48cb 100644 --- a/community/modules/scheduler/gke-cluster/README.md +++ b/community/modules/scheduler/gke-cluster/README.md @@ -155,7 +155,7 @@ limitations under the License. | Name | Description | |------|-------------| -| [cluster\_id](#output\_cluster\_id) | An identifier for the resource with format projects//locations//clusters/. | +| [cluster\_id](#output\_cluster\_id) | An identifier for the resource with format projects/{{project\_id}}/locations/{{region}}/clusters/{{name}}. | | [gke\_cluster\_exists](#output\_gke\_cluster\_exists) | A static flag that signals to downstream modules that a cluster has been created. Needed by community/modules/scripts/kubernetes-operations. | | [instructions](#output\_instructions) | Instructions on how to connect to the created cluster. | | [k8s\_service\_account\_name](#output\_k8s\_service\_account\_name) | Name of k8s service account. | diff --git a/community/modules/scheduler/pre-existing-gke-cluster/README.md b/community/modules/scheduler/pre-existing-gke-cluster/README.md index 50676ba8d6..4bc8933908 100644 --- a/community/modules/scheduler/pre-existing-gke-cluster/README.md +++ b/community/modules/scheduler/pre-existing-gke-cluster/README.md @@ -83,6 +83,6 @@ No modules. | Name | Description | |------|-------------| -| [cluster\_id](#output\_cluster\_id) | An identifier for the gke cluster with format projects//locations//clusters/. | +| [cluster\_id](#output\_cluster\_id) | An identifier for the gke cluster with format projects/{{project\_id}}/locations/{{region}}/clusters/{{name}}. | | [gke\_cluster\_exists](#output\_gke\_cluster\_exists) | A static flag that signals to downstream modules that a cluster exists. | From 5b6ca6059486e1641c0ece26a0d1681de61779a9 Mon Sep 17 00:00:00 2001 From: Farhad Sharabiani Date: Tue, 25 Jun 2024 16:21:12 +0000 Subject: [PATCH 3/4] miinor changes on outputs description --- community/modules/scheduler/gke-cluster/outputs.tf | 2 +- .../modules/scheduler/pre-existing-gke-cluster/outputs.tf | 2 +- modules/README.md | 3 +-- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/community/modules/scheduler/gke-cluster/outputs.tf b/community/modules/scheduler/gke-cluster/outputs.tf index 408b9aec47..53ee068ca2 100644 --- a/community/modules/scheduler/gke-cluster/outputs.tf +++ b/community/modules/scheduler/gke-cluster/outputs.tf @@ -15,7 +15,7 @@ */ output "cluster_id" { - description = "An identifier for the resource with format projects//locations//clusters/." + description = "An identifier for the resource with format projects/{{project_id}}/locations/{{region}}/clusters/{{name}}." value = google_container_cluster.gke_cluster.id } diff --git a/community/modules/scheduler/pre-existing-gke-cluster/outputs.tf b/community/modules/scheduler/pre-existing-gke-cluster/outputs.tf index 6166aa6bad..9bfd571b61 100644 --- a/community/modules/scheduler/pre-existing-gke-cluster/outputs.tf +++ b/community/modules/scheduler/pre-existing-gke-cluster/outputs.tf @@ -15,7 +15,7 @@ */ output "cluster_id" { - description = "An identifier for the gke cluster with format projects//locations//clusters/." + description = "An identifier for the gke cluster with format projects/{{project_id}}/locations/{{region}}/clusters/{{name}}." value = data.google_container_cluster.existing_gke_cluster.id } diff --git a/modules/README.md b/modules/README.md index bae9d14143..56d7862947 100644 --- a/modules/README.md +++ b/modules/README.md @@ -184,8 +184,7 @@ Pub/Sub subscription. Primarily used for [FSI - MonteCarlo Tutorial][fsi-monteca submission of Google Cloud Batch jobs. * **[gke-cluster]** ![community-badge] ![experimental-badge] : Creates a Kubernetes cluster using GKE. -* **[pre-existing-gke-cluster]** ![community-badge] ![experimental-badge] : Retrieves an existing - GKE cluster's attributes to be used in other modules as a substitute for creating a new cluster ([gke-cluster]). +* **[pre-existing-gke-cluster]** ![community-badge] ![experimental-badge] : Retrieves an existing GKE cluster. Substitute for ([gke-cluster]) module. * **[schedmd-slurm-gcp-v5-controller]** ![community-badge] : Creates a Slurm controller node using [slurm-gcp-version-5]. * **[schedmd-slurm-gcp-v5-login]** ![community-badge] : From f6bfeb40dec12c1c1b1c8a7e0c7f77a1edebe823 Mon Sep 17 00:00:00 2001 From: Farhad Sharabiani Date: Tue, 25 Jun 2024 16:39:31 +0000 Subject: [PATCH 4/4] terraform required version updated --- community/modules/scheduler/pre-existing-gke-cluster/README.md | 2 +- .../modules/scheduler/pre-existing-gke-cluster/versions.tf | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/community/modules/scheduler/pre-existing-gke-cluster/README.md b/community/modules/scheduler/pre-existing-gke-cluster/README.md index 4bc8933908..ebd4950e1b 100644 --- a/community/modules/scheduler/pre-existing-gke-cluster/README.md +++ b/community/modules/scheduler/pre-existing-gke-cluster/README.md @@ -52,7 +52,7 @@ limitations under the License. | Name | Version | |------|---------| -| [terraform](#requirement\_terraform) | >= 0.14.0 | +| [terraform](#requirement\_terraform) | >= 1.0.0 | | [google](#requirement\_google) | > 5.0 | ## Providers diff --git a/community/modules/scheduler/pre-existing-gke-cluster/versions.tf b/community/modules/scheduler/pre-existing-gke-cluster/versions.tf index 9083cd5299..30d00afe9c 100644 --- a/community/modules/scheduler/pre-existing-gke-cluster/versions.tf +++ b/community/modules/scheduler/pre-existing-gke-cluster/versions.tf @@ -26,5 +26,5 @@ terraform { module_name = "blueprints/terraform/hpc-toolkit:pre-existing-gke-cluster/v1.35.0" } - required_version = ">= 0.14.0" + required_version = ">= 1.0.0" }