google_composer_environment: bucket resource not found?

[kdzhao]

While using google_composer_environment to create a new composer environment, following this doc: https://www.terraform.io/docs/providers/google/r/composer_environment.html, after waiting for 58m, it gives me error like:

1 error occurred:
* module.my-composer.google_composer_environment.my-composer-env: 1 error occurred:
* google_composer_environment.my-composer-env: Error waiting to create Environment: Error waiting for Creating Environment: Error code 9, message: Resource not found (resource=us-central1-my-composer--a5de68f6-composer-backend-sub-a5de68f6-d705-47c9-b293-fd913685090b).,
Http error status code: 400
Http error message: BAD REQUEST
Additional errors:
{"ResourceType":"gcp-types/storage-v1:storage.buckets.insert","ResourceErrorCode":"400","ResourceErrorMessage":{"code":400,"errors":[{"domain":"global","message":"Invalid argument","reason":"invalid"}],"message":"Invalid argument","statusMessage":"Bad Request","requestPath":"https://www.googleapis.com/storage/v1/b","httpMethod":"POST"}}

From the error, looks like it can't find the bucket (since we don't specify any bucket in the code, I assume it is a bucket created by the script in the background).

Anyone see similar issue before?

Thanks

[emilymye]

This looks like an error returned from the actual API - the composer environment creates a bucket for storage, and it looks like at some point they fail to write to the bucket and that causes the entire creation operation to fail, which seems wrong! I can file a bug against the upstream GCP team but unfortunately I don't know that we can do much else on the Terraform provider side.

[emilymye]

I'm also curious, did the composer operation create a cluster? If so, did it get deleted?

[kdzhao]

Yes, please do. Thanks! Also, it will be great if in the log it can have more details.
The kubernete cluster was created and running, though the composer creation failed.

[emilymye]

yeah, that's also something we can ask for from the API team, but I'm not sure if it'll be visible from Terraform. Right now the only calls we make are to create the environment and we then just wait on the operation from composer, so if we ask them to surface more logs, it depends on how they want to output logs.

[emilymye]

assigning to myself and leaving this open for now to follow up once the upstream team gets back to me

[emilymye]

Hi @kdzhao - would you happen to have domain restricted sharing for your organization?

The composer team suspect that the error is occuring because you have it on and they sent me this from docs:

If you have enabled domain restricted sharing for your organization, you must disable the policy constraint before creating an environment. This action permits Cloud Composer to apply the required ALCs to the Cloud Storage bucket for your environment. You can re-enable the policy constraint after environment creation.

Let me know if you do have this enabled - if so, the workaround is to disable it while creating the environment.

[kdzhao]

I can't confirm it yet, still finding ways to confirm it within our organization/settings.

[m1racoli]

We had the same issue (not using terraform). It was indeed domain restricted sharing, which prevented the composer instance to launch.

[kardiff18]

Found the same error when using "No organization" under GCP. I wasn't able to edit the policy constraint because it's under "No organization". Can you edit resource manager policies in terraform somehow? I don't see a resource listed in our configs.

[emilymye]

@kardiff18 I'm not sure you'd be able to edit this setting for a org-less project - it could be that this is just enabled by default for no-org projects, but I'd have to make sure. Do you have config or debug logs that I could look at to verify? Thank you!

[kardiff18]

Hey -- it was enabled and we disabled it, so nevermind. Thanks!

[emilymye]

Closing this issue for now - it seems that the consistent fix has been to disable the org-level domain restricted sharing policy. Feel free to reopen if you're running into this issue but fix didn't work.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!