GCP Token is not refreshed

[mikedanese]

The GCP token expires after 1 hour and is not refreshed by the transport.

[mikedanese]

cc @kashomon

[amj]

/subscribe

i'm also having this problem :(

[dekkagaijin]

I can take this on

[amj]

woohoo!

[richardsliu]

I am having the same problem. Is there a fix planned for this?

[richardsliu]

Also what is the recommended workaround? The only way I can think of is catching 401/403s on the client side, then calling gcloud get-credentials and retry. Is there a better way?

[roycaihw]

@dekkagaijin has a fix that mitigates the problem during config loading #72

The main problem is that the client only check/refresh token during config loading-- it doesn't refresh token during api invocation. The problem applies to all authentication methods. Because the client is auto-generated by swagger-codegen, we cannot directly modify it. Suggested fixes are:

1. wrap the client to refresh token during api invocation (WIP in oidc auto refresh token during api invocation python#492)
2. have another thread to refresh the token periodically in the background
3. maybe port a fix to upstream code generator somehow

[richardsliu]

I'm doing this as a workaround:

from kubernetes import client as k8s_client
from kubernetes import config as k8s_config
from kubernetes.client import rest

...
while retry_condition:
  try:
    retry_condition = False
    crd_api = k8s_client.CustomObjectsApi(client)
    # This will throw an ApiException with status 401 if the k8s client was 
    # instantiated over an hour ago
    return crd_api.get_namespaced_custom_object(...)
  except rest.ApiException as e:
    logging.exception("ApiException: %s", e)
    if e.status == 401 or e.status == 403:
      # 1. Reload the kube config, which will refresh GCP token;
      k8s_config.load_kube_config()
      # 2. Re-create the k8s client using the refreshed config.
      client = k8s_client.ApiClient()
      # Trigger retry of this block
      retry_condition = True

But it would be great to have this fixed in the client somehow.

[TrevorEdwards]

It looks like the token is set on the client_configuration in one place:
https://github.com/kubernetes-client/python-base/blob/master/config/kube_config.py#L395-L402

I think if we set client_configuration.api_key as a custom dict for gcp auth providers, we can call _load_gcp_token every time the authorization field is fetched. That way we can fix it here. If that makes sense, I can work on a PR.

[tania-python-dev]

I still have a problem with refreshing token. Python client reads token from ~./kube/config, if it is already expired, creates new token and sends next request with a new one. It looks like a usual flow but the issue is that kubectl config remains the same (with old expired token) and API responds me with 403 Forbidden error since I come with unknown for him token.
The only way I can refresh token in ~./kube/config is to execute any kubectl command from console (e.g. kubectl get namespaces). After this token refreshes and API responds 200 OK. Does anybody know how to fix it?