Bazel Remote Cache: Cannot use WIF-generated credentials

[kylekurz]

Description of the problem / feature request:

Bazel cannot use credentials from Workload Identity Federation, exits with error:

ERROR: Failed to init auth credentials: Error reading credentials from stream, 
'type' value 'external_account' not recognized. 
Expecting 'authorized_user' or 'service_account'.

Feature requests: what underlying problem are you trying to solve with this feature?

Using remote cache without providing a permanent Service Account JSON credential in a secret or directly in my repository.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Set up a Google Cloud Storage bucket and configure WIF via these instructions: https://github.com/google-github-actions/auth. Once you've set it up, use the auth module to create a job that utilizes a credential file for remote cache access:

name: GitHub Actions CI - Build Artifacts
on: [pull_request]
jobs:
  pr-build:
    permissions:
      contents: 'read'
      id-token: 'write'
    runs-on: ubuntu-20.04
    steps:
      - name: Check out repository
        uses: actions/checkout@v2
      - id: 'auth'
        name: 'Authenticate to Google Cloud'
        uses: 'google-github-actions/auth@v0.4.0'
        with:
          token_format: 'access_token'
          workload_identity_provider:<identity_provider>
          service_account: <service_account_email>
          create_credentials_file: true
      - name: Verify that all dependencies are set up correctly
        run: |
          bazelisk run \
            --google_credentials=${{ steps.auth.outputs.credentials_file_path }} \
            --remote_cache=<cache_url> \
            //:gazelle

What operating system are you running Bazel on?

Ubuntu 20.04 on a GitHub Hosted Runner

What's the output of bazel info release?

release 4.2.1

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

N/A

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

N/A

Have you found anything relevant by searching the web?

No

Any other information, logs, or outputs that you want to share?

N/A

[rockwotj]

I believe this is easy as updating the third_party libraries here: https://github.com/bazelbuild/bazel/tree/master/third_party/auth

[bazaglia]

Facing the same issue. @kylekurz, could you find a workaround while the third-party libraries aren't updated?

[kylekurz]

@bazaglia I've resigned myself to using a SA credential in a GitHub secret for now, with notes to remove it once this is fixed.

[coeuvre]

Fixed by #15176. Closing.

[brentleyjones]

@coeuvre Is that change low enough risk to get into 5.2?

[coeuvre]

I think it's safe to get into 5.2.

[omerlh]

Will it be part of the release? I just looked at the commits on the release's branch and I don't see it there...

[rockwotj]

Looks like it needs to be cherry picked onto the release branch #15176 (comment)

[omerlh]

Yep, I saw the comment on #15210 - should I comment it as a blocker for the release?

[brentleyjones]

Getting cherry-picked here: #15383

[bazaglia]

Unfortunately, even though Bazel 5.2 updates the Google Auth library, something seems broken and the following Bazel command:

bazel test --remote_cache <cache-url> --google_credentials=${{ steps.auth.outputs.credentials_file_path }} //...

Fails with:

Caused by: java.lang.IllegalArgumentException: Can not set java.util.List field com.google.api.client.http.HttpHeaders.authorization to java.lang.String
	at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(Unknown Source)
	at java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(Unknown Source)
	at java.base/jdk.internal.reflect.UnsafeObjectFieldAccessorImpl.set(Unknown Source)
	at java.base/java.lang.reflect.Field.set(Unknown Source)
	at com.google.api.client.util.FieldInfo.setFieldValue(FieldInfo.java:245)
	at com.google.api.client.util.FieldInfo.setValue(FieldInfo.java:206)
	at com.google.api.client.util.GenericData.set(GenericData.java:125)
	at com.google.api.client.http.HttpHeaders.set(HttpHeaders.java:175)
	at com.google.api.client.http.HttpHeaders.set(HttpHeaders.java:58)
	at com.google.api.client.util.GenericData.putAll(GenericData.java:138)
	at com.google.auth.oauth2.IdentityPoolCredentials.getSubjectTokenFromMetadataServer(IdentityPoolCredentials.java:233)
	at com.google.auth.oauth2.IdentityPoolCredentials.retrieveSubjectToken(IdentityPoolCredentials.java:188)
	at com.google.auth.oauth2.IdentityPoolCredentials.refreshAccessToken(IdentityPoolCredentials.java:169)
	at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:257)
	at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:254)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.auth.oauth2.OAuth2Credentials$AsyncRefreshResult.executeIfNew(OAuth2Credentials.java:580)
	at com.google.auth.oauth2.OAuth2Credentials.asyncFetch(OAuth2Credentials.java:220)
	at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:170)
	at com.google.auth.oauth2.ExternalAccountCredentials.getRequestMetadata(ExternalAccountCredentials.java:292)
	at com.google.devtools.build.lib.remote.http.AbstractHttpHandler.addCredentialHeaders(AbstractHttpHandler.java:73)
	at com.google.devtools.build.lib.remote.http.HttpDownloadHandler.write(HttpDownloadHandler.java:141)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)
	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:808)
	at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1025)
	at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:306)
	at com.google.devtools.build.lib.remote.http.HttpCacheClient.lambda$get$6(HttpCacheClient.java:496)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
...

This looks related to this issue: googleapis/google-http-java-client#340