userinfo.email scope is needed to work with rbac

[jlewi]

KubeConfigLoader._refresh_credentials sets the scope to cloud-platform.

I think userinfo.email scope is needed as well.

See kubernetes/kubernetes#58141 for a similar issue with kubectl and a good explanation.

I think we need the userinfo.email scope because RBAC rules can be expressed in terms of the email of service accounts. But if the userinfo.email scope isn't included that APIServer ends up using the numeric id of service accounts which won't work if RBAC rules are written in terms of
the emails.

I haven't confirmed for myself this is an issue (I'm working through a variety of issues with kubectl/kubeconfig/client libs) so I could be wrong.

[axelsteingrimsson]

I believe we are experiencing the exact issue that you are describing when using the Kubernetes Python client. We are finding that when the Python client refreshes the access token in ~/.kube/config we can't make requests to the Kubernetes API server with a service account that is normally authenticated with RBAC. Though when we use kubectl the token refresh is fine.

Using a ~/.kube/config with the users section of the format:

users:
- name: application-default-credentials
  user:
    auth-provider:
      config:
        access-token: <REDACTED_ACCESS_TOKEN>
        expiry: '2018-11-11T11:11:11Z'
      name: gcp

And a GCP service account (not a user service account, but a server service account) with the format:

{
  "type": "service_account",
  "project_id": "example-project-id",
  "private_key_id": "<REDACTED_PRIVATE_KEY_ID>",
  "private_key": "<REDACTED_PRIVATE_KEY>",
  "client_email": "<REDACTED_CLIENT_EMAIL>",
  "client_id": "REDACTED_CLIENT_ID",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/<REDACTED_CLIENT_EMAIL>"
}

I found that if I used kubectl with this configuration and service account to refresh the access token then it would be valid and I'm able to make requests to the Kubernetes API server. However, if
the token is refreshed by the Python client, then the token would be technically valid, but I wouldn't be authenticated to make requests. Instead I would get back an error like:

Forbidden: pods is forbidden: User "<REDACTED_CLIENT_ID>" cannot watch pods in the namespace "<REDACTED_NAMESPACE>": Required "container.pods.list" permission.

Where the REDACTED_CLIENT_ID matches the ID from our service account, which corresponds with what you were saying:

But if the userinfo.email scope isn't included that APIServer ends up using the numeric id of service accounts which won't work if RBAC rules are written in terms of
the emails.

Using the command curl https://www.googleapis.com/oauth2/v2/tokeninfo?access_token=<access_token> to get information about the tokens that are generated:

`kubectl`
{
  "issued_to": "<REDACTED_ID>",
  "audience": "<REDACTED_ID>",
  "expires_in": 2534,
  "access_type": "offline"
  "scope": "https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email",
  "email": "<REDACTED_EMAIL>",
  "verified_email": true,
}

Python client:
{
  "issued_to": "<REDACTED_ID>",
  "audience": "<REDACTED_ID>",
  "expires_in": 3230,
  "access_type": "offline",
  "scope": "https://www.googleapis.com/auth/cloud-platform"
}

So the token generated by the Python client is technically valid, however it isn't signed by an email and therefore can't be used by the Kubernetes API server to authenticate using RBAC.

I've written a workaround where I wrap the call to google.auth.default so that it appends "https://www.googleapis.com/auth/userinfo.email" to whatever scope get passed in. That seems to have solved the issue for us, however I feel like it would make sense to change the call to google.auth.default so that it passes in this scope like:

def _refresh_credentials():
    credentials, project_id = google.auth.default(scopes=[
        'https://www.googleapis.com/auth/cloud-platform',
        'https://www.googleapis.com/auth/userinfo.email'
    ])
    request = google.auth.transport.requests.Request()
    credentials.refresh(request)
    return credentials

I can put up a PR if there's interest, from my experience identifying this as the issue when you're seeing authentication issue can be really obscure. I've also seen a couple other issues of people complaining that the refresh token isn't refreshing so this could potentially help others as well.

[axelsteingrimsson]

I figure I'll just open a PR to get things going if that's alright.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[axelsteingrimsson]

/close

[k8s-ci-robot]

@axelsteingrimsson: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.