Terraform doesn't support GCP "external_account_authorized_user" #34234

FabioAntunes · 2023-11-13T17:59:19Z

Terraform Version

v1.6.2aTerraform v1.6.3
on darwin_amd64

Terraform Configuration Files

provider "google" {
  region = "us-east1"
  project = "project-name"
}
terraform {
  backend "gcs" {
    bucket = "bucket-name"
    prefix = "state-name.tfstate"
  }
}

Debug Output

2023-11-13T17:54:31.126Z [INFO]  Terraform version: 1.6.3
2023-11-13T17:54:31.127Z [DEBUG] using github.com/hashicorp/go-tfe v1.36.0
2023-11-13T17:54:31.127Z [DEBUG] using github.com/hashicorp/hcl/v2 v2.19.1
2023-11-13T17:54:31.127Z [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1
2023-11-13T17:54:31.127Z [DEBUG] using github.com/zclconf/go-cty v1.14.1
2023-11-13T17:54:31.127Z [INFO]  Go runtime version: go1.21.3
2023-11-13T17:54:31.127Z [INFO]  CLI args: []string{"terraform", "init"}
2023-11-13T17:54:31.127Z [DEBUG] Attempting to open CLI config file: /Users/fabioantunes/.terraformrc
2023-11-13T17:54:31.127Z [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2023-11-13T17:54:31.128Z [INFO]  Loading CLI configuration from /Users/fabioantunes/.terraform.d/credentials.tfrc.json
2023-11-13T17:54:31.128Z [DEBUG] checking for credentials in "/Users/fabioantunes/.terraform.d/plugins"
2023-11-13T17:54:31.128Z [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2023-11-13T17:54:31.128Z [DEBUG] will search for provider plugins in /Users/fabioantunes/.terraform.d/plugins
2023-11-13T17:54:31.130Z [DEBUG] ignoring non-existing provider search directory /Users/fabioantunes/Library/Application Support/io.terraform/plugins
2023-11-13T17:54:31.130Z [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2023-11-13T17:54:31.130Z [INFO]  CLI command args: []string{"init"}

Initializing the backend...
2023-11-13T17:54:31.156Z [DEBUG] New state was assigned lineage "c7a673bd-fbc4-c147-1114-964bfdba5430"
2023-11-13T17:54:31.156Z [DEBUG] checking for provisioner in "."
2023-11-13T17:54:31.170Z [DEBUG] checking for provisioner in "/usr/local/bin"
2023-11-13T17:54:31.170Z [DEBUG] checking for provisioner in "/Users/fabioantunes/.terraform.d/plugins"
╷
│ Error: storage.NewClient() failed: dialing: unknown credential type: "external_account_authorized_user"

Expected Behavior

Terraform init should work

Actual Behavior

Terraform init fails even though the credentials are valid.

Steps to Reproduce

gcloud auth login
gcloud auth application-default login

cat /Users/fabioantunes/.config/gcloud/application_default_credentials.json
{
  "audience": "//iam.googleapis.com/locations/global/workforcePools/random/stuff/in/here/name-of-the-provider",
  "client_id": "randomnumber.apps.googleusercontent.com",
  "client_secret": "super-client-secret",
  "quota_project_id": "my-super-project",
  "refresh_token": "super-refresh-token",
  "token_info_url": "https://sts.googleapis.com/v1/introspect",
  "token_url": "https://sts.googleapis.com/v1/oauthtoken",
  "type": "external_account_authorized_user"
}

terraform init

Additional Context

I have raised this with terragrunt as well and it seems they managed to fix this by upgrading the gcp SDK
gruntwork-io/terragrunt#2775

References

No response

The text was updated successfully, but these errors were encountered:

crw · 2023-11-13T23:17:32Z

Thanks for this report! I'll raise it with the appropriate team. Please be aware that backend development happens fairly infrequently and is per the priorities of the team in question. Thanks again!

FabioAntunes · 2023-11-14T11:41:32Z

How can I follow the progress? Would this issue be closed when this is fixed? or do I need to keep an eye on releases?

crw · 2023-11-14T20:12:41Z

@FabioAntunes Indeed, this issue would be closed. Before that, a PR would be opened that referenced this issue, which would also show up in the comment thread of this issue. Usually the PRs see a lot of activity, so would be worth tracking itself. If you do not see a PR reference, it is likely not yet being worked on. Thanks!

bschaatsbergen · 2024-01-02T20:09:12Z

Hey @FabioAntunes, thanks for taking the time to raise this issue. I would happily take a stab at this and see if I can get it fixed! (I'm expecting this to be a simple package bump, if not done already). cc @crw.

bschaatsbergen · 2024-01-02T22:31:39Z

@FabioAntunes could you please try a more recent release of Terraform Core, for example 1.6.4 or anything more recent?

The type external_account_authorized_user was added to the cloud.google.com/go/auth package on the 18th of October: https://github.com/googleapis/google-cloud-go/blame/d1019803ecd506e93848d6d761f41b955c00d31a/auth/internal/internaldetect/internaldetect.go#L68 by Google.

And on the 25th of October we updated the dependencies in Terraform Core, to v1.30.1 of the cloud.google.com/go/storage package. This change was released under Terraform Core 1.6.4 on the 15th of November.

thomas-dussouillez · 2024-01-10T10:20:01Z

I still have the issue. I am using Terraform v1.6.6 on darwin_arm64 and provider hashicorp/google in version 5.11.0.
Is it supposed to be fixed ?
Thank you

bschaatsbergen · 2024-01-10T18:08:51Z

@thomas-dussouillez that's odd. Allow me as soon as possible to take a look at it and see whether I can reproduce it.

thomas-dussouillez · 2024-01-17T10:21:00Z

Still not working for me. I notice that the Go runtime version is 1.21.5.
Here is my gcloud version:

Google Cloud SDK 459.0.0
bq 2.0.101
core 2024.01.06
gcloud-crc32c 1.0.0
gke-gcloud-auth-plugin 0.5.8
gsutil 5.27
kubectl 1.27.9

Besides that, I tested on a Linux server (otherwise I work/tested on an Apple MacBook with M2 chip) and I have the same error.
I tested with terraform installed with tfenv and direct installation from brew hashicorp tap and from binary, same error.

Here is my error DEBUG log:

2024-01-17T11:00:22.038+0100 [INFO]  Terraform version: 1.6.6
2024-01-17T11:00:22.039+0100 [DEBUG] using github.com/hashicorp/go-tfe v1.36.0
2024-01-17T11:00:22.039+0100 [DEBUG] using github.com/hashicorp/hcl/v2 v2.19.1
2024-01-17T11:00:22.039+0100 [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1
2024-01-17T11:00:22.039+0100 [DEBUG] using github.com/zclconf/go-cty v1.14.1
2024-01-17T11:00:22.039+0100 [INFO]  Go runtime version: go1.21.5
2024-01-17T11:00:22.039+0100 [INFO]  CLI args: []string{"/opt/homebrew/Cellar/tfenv/3.0.0/versions/1.6.6/terraform", "init", "-backend-config=/Users/tdussouillez/Documents/gitlab/vpg-gke/environments/test/_backend.tfconfig"}
2024-01-17T11:00:22.039+0100 [DEBUG] Attempting to open CLI config file: /Users/tdussouillez/.terraformrc
2024-01-17T11:00:22.039+0100 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2024-01-17T11:00:22.040+0100 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2024-01-17T11:00:22.040+0100 [DEBUG] ignoring non-existing provider search directory /Users/tdussouillez/.terraform.d/plugins
2024-01-17T11:00:22.040+0100 [DEBUG] ignoring non-existing provider search directory /Users/tdussouillez/Library/Application Support/io.terraform/plugins
2024-01-17T11:00:22.040+0100 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2024-01-17T11:00:22.040+0100 [INFO]  CLI command args: []string{"init", "-backend-config=/Users/tdussouillez/Documents/gitlab/vpg-gke/environments/test/_backend.tfconfig"}

Initializing the backend...
Initializing modules...
2024-01-17T11:00:22.061+0100 [DEBUG] Module installer: begin arcleaner
2024-01-17T11:00:22.063+0100 [DEBUG] Module installer: begin argocd
2024-01-17T11:00:22.065+0100 [DEBUG] Module installer: begin certs
2024-01-17T11:00:22.066+0100 [DEBUG] Module installer: begin datadog
2024-01-17T11:00:22.069+0100 [DEBUG] Module installer: begin gke
2024-01-17T11:00:22.082+0100 [DEBUG] Module installer: begin kubernetes
2024-01-17T11:00:22.083+0100 [DEBUG] Module installer: begin logging
2024-01-17T11:00:22.084+0100 [DEBUG] Module installer: begin networking
2024-01-17T11:00:22.086+0100 [DEBUG] Module installer: begin permissions
2024-01-17T11:00:22.087+0100 [DEBUG] Module installer: begin private-gateway
2024-01-17T11:00:22.088+0100 [DEBUG] Module installer: begin project
2024-01-17T11:00:22.091+0100 [DEBUG] Module installer: begin public-gateway
2024-01-17T11:00:22.092+0100 [DEBUG] Module installer: begin pubsub
2024-01-17T11:00:22.093+0100 [DEBUG] Module installer: begin pubsub-notification
2024-01-17T11:00:22.103+0100 [DEBUG] Module installer: begin sealed-secrets
2024-01-17T11:00:22.104+0100 [DEBUG] Module installer: begin static-external-ip
2024-01-17T11:00:22.105+0100 [DEBUG] Module installer: begin static-internal-ip
╷
│ Error: storage.NewClient() failed: dialing: unknown credential type: "external_account_authorized_user"
│ 
│ 
╵

thomas-dussouillez · 2024-02-12T08:50:19Z

Hello, any update on this issue ? Still not working for us and it is quite impacting :(
Thank you

bschaatsbergen · 2024-02-12T09:51:55Z

Hey, I've been stuck trying to reproduce the issue using different Google Cloud identities, both my Gmail and a federated cloud identity user - these are shown as authorized_user instead of external_account_authorized_user. Could you could share some details on how you authenticated? I saw something about a "workforcePool" in your application_default_credentials.json file. Knowing more about that could really help me get this figured out! Thanks in advance!

thomas-dussouillez · 2024-02-12T10:33:20Z

I'm authenticating using Workforce Pool Identity (SSO configuration providing identity federation with an OIDC provider with Azure AD as identity base). I don't have a google identity but an external identity, this is why it is called external_account_authorized_user.

I authenticate using this command :
gcloud auth login --login-config=CONFIG_FILE
gcloud auth application-default login --login-config=CONFIG_FILE

The CONFIG_FILE looks like this:

{
  "type": "external_account_authorized_user_login_config",
  "audience": "//iam.googleapis.com/locations/global/workforcePools/POOLNAME/providers/PROVIDERNAME",
  "auth_url": "https://auth.cloud.google/authorize",
  "token_url": "https://sts.googleapis.com/v1/oauthtoken",
  "token_info_url": "https://sts.googleapis.com/v1/introspect"
}

Then my application_default_credentials.json looks like this:

{
  "audience": "//iam.googleapis.com/locations/global/workforcePools/POOLNAME/providers/PROVIDERNAME",
  "client_id": "XXX.apps.googleusercontent.com",
  "client_secret": "SUPERSECRETSTRING",
  "refresh_token": "token",
  "token_info_url": "https://sts.googleapis.com/v1/introspect",
  "token_url": "https://sts.googleapis.com/v1/oauthtoken",
  "type": "external_account_authorized_user",
  "universe_domain": "googleapis.com"
}

bschaatsbergen · 2024-02-13T20:35:50Z

I believe I've identified the issue. Upon investigating, I found that our Google Cloud Storage SDK (cloud.google.com/go/storage) implementation relies on a package called golang.org/x/oauth2/google. This package introduced the external_account_authorized_user user type in version v0.12.0, which was released in September 2023. You can refer to the commit here: https://cs.opensource.google/go/x/oauth2/+/43b6a7ba1972152df70cd1e5ac7705e981df5f66.

Currently, we are using version v0.11.0 of the golang.org/x/oauth2 package.

bschaatsbergen · 2024-02-13T20:49:28Z

I've submitted a pull request to update the golang.org/x/oauth2 package to its latest version, specifically v0.17.0.

FabioAntunes · 2024-02-13T20:53:26Z

yeah I mentioned that when I raised the issue:

Additional Context
I have raised this with terragrunt as well and it seems they managed to fix this by upgrading the gcp SDK
gruntwork-io/terragrunt#2775

bschaatsbergen · 2024-02-13T20:58:03Z

yeah I mentioned that when I raised the issue:

Additional Context
I have raised this with terragrunt as well and it seems they managed to fix this by upgrading the gcp SDK
gruntwork-io/terragrunt#2775

@FabioAntunes after seeing your reply on gruntwork-io/terragrunt#2773 (comment), I wasn't sure whether the package upgrades actually solved your issue there?

The need for an upgrade wasn't specifically related to the Google Cloud SDK; instead, it involved updating the golang.org/x/auth2 package, which is used to initialize the required credentials passed into the storage client. See https://github.com/hashicorp/terraform/blob/main/internal/backend/remote-state/gcs/backend.go#L148-L163 for the code that's responsible for initializing and passing the credentials into the storage client.

Your observation was quite helpful in confirming my investigation. Thanks! 👍🏼

github-actions · 2024-04-05T02:00:03Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

FabioAntunes added bug new new issue not yet triaged labels Nov 13, 2023

apparentlymart added the backend/gcs label Nov 13, 2023

crw added the waiting-response An issue/pull request is waiting for a response from the community label Jan 4, 2024

apparentlymart added the v1.6 Issues (primarily bugs) reported against v1.6 releases label Jan 19, 2024

bschaatsbergen mentioned this issue Feb 13, 2024

Update golang.org/x/oauth2 to v0.17.0 for Google Cloud Storage SDK compatibility #34670

Merged

alisdair closed this as completed in #34670 Mar 5, 2024

github-actions bot locked as resolved and limited conversation to collaborators Apr 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Terraform doesn't support GCP "external_account_authorized_user" #34234

Terraform doesn't support GCP "external_account_authorized_user" #34234

FabioAntunes commented Nov 13, 2023 •

edited by crw

Loading

crw commented Nov 13, 2023

FabioAntunes commented Nov 14, 2023

crw commented Nov 14, 2023

bschaatsbergen commented Jan 2, 2024 •

edited

Loading

bschaatsbergen commented Jan 2, 2024 •

edited

Loading

thomas-dussouillez commented Jan 10, 2024

bschaatsbergen commented Jan 10, 2024 •

edited

Loading

thomas-dussouillez commented Jan 17, 2024 •

edited

Loading

thomas-dussouillez commented Feb 12, 2024

bschaatsbergen commented Feb 12, 2024 •

edited

Loading

thomas-dussouillez commented Feb 12, 2024 •

edited

Loading

bschaatsbergen commented Feb 13, 2024 •

edited

Loading

bschaatsbergen commented Feb 13, 2024 •

edited

Loading

FabioAntunes commented Feb 13, 2024

bschaatsbergen commented Feb 13, 2024 •

edited

Loading

github-actions bot commented Apr 5, 2024

Terraform doesn't support GCP "external_account_authorized_user" #34234

Terraform doesn't support GCP "external_account_authorized_user" #34234

Comments

FabioAntunes commented Nov 13, 2023 • edited by crw Loading

Terraform Version

Terraform Configuration Files

Debug Output

Expected Behavior

Actual Behavior

Steps to Reproduce

Additional Context

References

crw commented Nov 13, 2023

FabioAntunes commented Nov 14, 2023

crw commented Nov 14, 2023

bschaatsbergen commented Jan 2, 2024 • edited Loading

bschaatsbergen commented Jan 2, 2024 • edited Loading

thomas-dussouillez commented Jan 10, 2024

bschaatsbergen commented Jan 10, 2024 • edited Loading

thomas-dussouillez commented Jan 17, 2024 • edited Loading

thomas-dussouillez commented Feb 12, 2024

bschaatsbergen commented Feb 12, 2024 • edited Loading

thomas-dussouillez commented Feb 12, 2024 • edited Loading

bschaatsbergen commented Feb 13, 2024 • edited Loading

bschaatsbergen commented Feb 13, 2024 • edited Loading

FabioAntunes commented Feb 13, 2024

bschaatsbergen commented Feb 13, 2024 • edited Loading

github-actions bot commented Apr 5, 2024

FabioAntunes commented Nov 13, 2023 •

edited by crw

Loading

bschaatsbergen commented Jan 2, 2024 •

edited

Loading

bschaatsbergen commented Jan 2, 2024 •

edited

Loading

bschaatsbergen commented Jan 10, 2024 •

edited

Loading

thomas-dussouillez commented Jan 17, 2024 •

edited

Loading

bschaatsbergen commented Feb 12, 2024 •

edited

Loading

thomas-dussouillez commented Feb 12, 2024 •

edited

Loading

bschaatsbergen commented Feb 13, 2024 •

edited

Loading

bschaatsbergen commented Feb 13, 2024 •

edited

Loading

bschaatsbergen commented Feb 13, 2024 •

edited

Loading