google_composer_environment says it failed when it didn't

[kardiff18]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

0.11

Affected Resource(s)

• google_composer_environment

Terraform Configuration Files

# Create Composer Environment
resource "google_composer_environment" "env" {
  provider = "google-beta"
  name = "cud-correction-env"
  region = "${var.region}"
  depends_on = ["google_project_service.composerapi"]

  config {
    node_config {
      zone = "${var.zone}"
      service_account = "${google_service_account.cud_service_account.email}"
    }
    software_config {
      python_version = 3
      pypi_packages {
        pytest = ""
        google-cloud-bigquery = ""
        google-cloud-storage = ""
        apache-airflow = "[gcp_api]"
        werkzeug = "==0.15.4"
      }

      env_variables {
        project_id = "${var.project_id}"
        billing_export_table_name = "${var.billing_export_table_name}"
        corrected_dataset_id = "${var.corrected_dataset_id}"
        corrected_table_name = "${var.commitment_table_name}"
        commitment_table_name = "${var.commitment_table_name}"
        enable_cud_cost_attribution = "${var.enable_cud_cost_attribution}"
      }
    }

  }
}

Debug Output

• google_composer_environment.env: googleapi: Error 400: Found 1 problem:
  1. Error validating key apache-airflow. Updating the 'airflow' package is not permitted., badRequest

Panic Output

None

Expected Behavior

Should have given success message.

Actual Behavior

It thinks that the environment upload failed, but when I look in the Airflow UI it says that it succeeded. However, I am trying to access other resources (for instance, uploading DAG files) that depend on the composer resource. I cannot do this because tfstate never realizes that the environment was successful.

Steps to Reproduce

1. terraform apply

No

References

No

[Chupaka]

What if you remove "apache-airflow" from pypi_packages? Composer manages AirFlow automatically.

[kardiff18]

Yes, that works. But the problem is that terraform should still reflect the state in GCP. Because terraform says that it failed, but in GCP the environment was still created successfully.

[paddycarver]

It sounds like the API is telling us there was an error, but it's a recoverable error. I think we could solve this a few ways:

• Throw an error ourselves (before attempting to create the environment) if we detect the apache-airflow package. (this is pretty straightforward to do)
• Silently ignore the apache-airflow package, making it unmanageable with Terraform and removing it from the request even if you put it in your Terraform config. (This is semi-straightforward to do.)
• Learn to detect which errors aren't terminal and the environment is actually created, and quash them. (this is semi-complicated and finicky.)

[chrisst]

@paddycarver how about trying to fetch the composer environment before removing the id from state. I believe it's a fair assumption that if the resource is still get-able then it's in a state that exists but is in error and likely tainted.
I'm also not keen on playing whack-a-mole with unmanageable packages.

[edwardmedia]

I can't repro this issue. From terminal, tf apply stopped with below exception, while the Composer itself seems to be fine. The link to Airflow UI is good. DAGs Folder has been created.

Below is the HCL I was using, followed by the request & response in the log.

resource "google_composer_environment" "env" {
  provider = google-beta
  name = "issue4146-3"
  region = "us-central1"
#  depends_on = ["google_project_service.composerapi"]
  config {
    node_config {
      zone = "us-central1-a"
      service_account = "fake-svcaccount@myproject.iam.gserviceaccount.com"
    }
    software_config {
      python_version = 3
      pypi_packages = {
        pytest = ""
        google-cloud-bigquery = ""
        google-cloud-storage = ""
        apache-airflow = "[gcp_api]"
        werkzeug = "==0.15.4"
      }
      env_variables = {
        project_id = "myproject"
      }
    }
  }
}

 ---[ REQUEST ]---------------------------------------
 PATCH /v1beta1/projects/myproject/locations/us-central1/environments/issue4146-3?alt=json&
 {
  "config": {
   "softwareConfig": {
    "pypiPackages": {
     "apache-airflow": "[gcp_api]",
     "google-cloud-bigquery": "",
     "google-cloud-storage": "",
     "pytest": "",
     "werkzeug": "==0.15.4"
    }
   }
  }
 }

 ---[ RESPONSE ]--------------------------------------
 {
   "error": {
] plugin.terraform-provider-google-beta_v3.16.0_x5:     "code": 400,
     "message": "Found 1 problem:\n\t1) Error validating key apache-airflow. Updating the 'airflow' pa
ckage is not permitted.",
     "errors": [
       {
         "message": "Found 1 problem:\n\t1) Error validating key apache-airflow. Updating the 'airflow
' package is not permitted.",
         "domain": "global",
         "reason": "badRequest"
       }
     ],
     "status": "INVALID_ARGUMENT"
   }
 }

[megan07]

Hi @Kardiff!

I’ve been looking into this some, and after some trial, error and discussion, I think it might be working as intended in 0.12. This is significantly different than in 0.11, as terraform 0.12 now marks the resource as tainted after the error, but still stores the rest of the values in state. Being tainted, you can check the resource in the console, and if it appears to be valid, you can run terraform untaint <resource> and it will be as if no error had happened.

After updating to 0.12 could you retry and let me know if it’s working more aligned to your expectations?

Thanks!

[ScottSuarez]

Based on @megan07's comment and no response from the user I will be closing this ticket.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.