Recreate k8s client on auth failure

[richardsliu]

This is a work around for kubernetes-client/python-base#59

There is a bug in the python K8s client preventing GCP auth tokens from being refreshed when they expire.

• As a result our E2E tests fail waiting for workflows that take more than an hour. In this case, the OAuth token expires so the requests to get workspace status fail with unauthorized

see
#207 - Argo client should refresh credentials if OAuth token expires
#204 - argo_client doesn't handle retries correctly.

This change is 

[richardsliu]

/retest

[richardsliu]

/test all

[richardsliu]

/assign @jlewi
The workaround is to (1) refresh kube config, and (2) make sure the k8s api client is recreated. For the short term fix, I'm simply recreating the client on each k8s API call.

[jlewi]

Thanks so much.

/lgtm
/approve

Just a reminder please provide informative PR descriptions with links to appropriate issues. I went ahead and updated the PR description.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlewi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jlewi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment