Composer environment fields are missing, one is required to create private environments

[bw-josepavonalvarez]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

Some Composer fields are missing from the google_composer_environment resource.

There are some changes in Composer behaviour since 10th April and creating a private Composer environment from terraform doesn't work anymore because one of these fields is new required.

The following example code:

resource "google_composer_environment" "test" {
  provider = google-beta
  name    = "private-composer"
  region  = "europe-west2"
  project = "service-project"
  config {
    node_count = 3

    node_config {
      zone         = "europe-west2-a"
      machine_type = "n1-standard-1"

      network    = "projects/host-project/global/networks/shared-vpc"
      subnetwork = "projects/host-project/regions/europe-west2/subnetworks/composer-subnet"

      service_account = google_service_account.test.name

      ip_allocation_policy {
        use_ip_aliases                = true
        cluster_secondary_range_name  = "composer-pods"
        services_secondary_range_name = "composer-services"
      }
    }

    private_environment_config {
      enable_private_endpoint = true
      master_ipv4_cidr_block  = "172.16.5.0/28"
    }
  }

  depends_on = [google_project_iam_member.composer-worker]
}

Fails with the following message:

google_composer_environment.test: Creating...

Error: googleapi: Error 400: Found 1 problem:
        1) Web server network access control must be provided in Private IP mode. If you are using gcloud, please upgrade to the newest version., badRequest

From gcloud update (288.0.0), we can see this new requirement:

Cloud Composer
  * Added the following flags to gcloud beta composer environments commands to specify IP ranges for Web Server and Cloud SQL network subranges:
    * --web-server-ipv4-cidr
    * --cloud-sql-ipv4-cidr
Require explicitly specifying web server access control when using --enable-private-environment in gcloud beta composer environments create.

Missing fields from the API are:

• WebServerNetworkAccessControl which is the one required when creating a private environment.
• PrivateEnvironmentConfig / webServerIpv4CidrBlock
• PrivateEnvironmentConfig / cloudSqlIpv4CidrBlock

I think the most important addition neede is the web server network access control because, otherwise, it's not posible to create a private Composer environment using terraform.

New or Affected Resource(s)

• google_composer_environment

Potential Terraform Configuration

resource "google_composer_environment" "test" {
  provider = google-beta
  config {
    private_environment_config {
      enable_private_endpoint = true
      master_ipv4_cidr_block = "172.16.5.0/28"
      web_server_ipv4_cidr_block = "172.16.6.0/28"
      cloud_sql_ipv4_cidr_block = "172.16.7.0/28"
    }
    web_server_network_access_control {
      allowed_ip_ranges = [
        {
           description = "network1"
           value = "10.0.0.0/8"
        },
        {
           description = "network2"
           value = "192.168.0.0/16"
         }
     ]
  }
}

References

• Web server network access control: https://cloud.google.com/composer/docs/reference/rest/v1beta1/projects.locations.environments#webservernetworkaccesscontrol
• Private environment config: https://cloud.google.com/composer/docs/reference/rest/v1beta1/projects.locations.environments#privateenvironmentconfig

[PeWu]

Composer no longer requires web server network access control in Private IP mode.

[alfredw33]

Hello,

I've been playing with this resource for the last couple of weeks. Ideally if we're building a private cluster could we atleast expose the master_authorized_networks_config for it so we can set a bastion host ip or a set of trusted networks we would allow connection to?

RIght now, there will be a manual process to setup a bastion host and then manually go into the console to attach the bation's host private ip to the master authorized network.

If there is away around this already then some clarity or direction on how to do this would be great.

[slevenick]

Hello,

I've been playing with this resource for the last couple of weeks. Ideally if we're building a private cluster could we atleast expose the master_authorized_networks_config for it so we can set a bastion host ip or a set of trusted networks we would allow connection to?

RIght now, there will be a manual process to setup a bastion host and then manually go into the console to attach the bation's host private ip to the master authorized network.

If there is away around this already then some clarity or direction on how to do this would be great.

Feel free to submit another feature request for this and we will process it as usual. I only added support for sql/web server ranges and the WebServerNetworkAccessControl field

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!