Authentication and service account plan for Pipeline + Kubeflow

[IronPan]

Currently in Pipeline instruction, we deploy the cluster with cloud-platform scope. Without any configuration, the pipeline and any derived argo jobs will run under the default service account (default Compute Engine service account). This unrestricted setup was OK in the past for development and testing propose. But as project goes public, I think it might want to change the service account setup into a managed way.

Kubeflow uses different service accounts for different roles.

• kf admin: managing the kubeflow cluster. E.g. networking, kf deployment config
• kf user: have access to various GCP APIs such as GCS and BQ, for various ML jobs.

For ML pipeline, the argo job, as well as any derived workload such as tf-job, should ideally run under kf-user service account in order to access to GCP API. To achieve this here are some todo items needed from various pipeline points.

1. Pipeline System need to mount the kf-user service account key to each pod in the argo job and set up GOOGLE_APPLICATION_CREDENTIALS environment variable. Kubeflow stores the service account key as K8s secrete in the cluster.

apiVersion: v1
kind: Pod
metadata:
  generateName: gcloud
  namespace: kubeflow
spec:
  containers:
  - image: google/cloud-sdk
    name: gcloud
    command: [sleep]
    args: ["10000000"]
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: "/etc/secrets/user-gcp-sa.json"      
    volumeMounts:
    - name: sa
      mountPath: "/etc/secrets"
      readOnly: true
  volumes:
  - name: sa
    secret:
      secretName: user-gcp-sa

2. The service account need to be activated inside the pod using gcloud auth activate-service-account, before GCP API call. Please see here as an example.
3. For tf-job samples, the scheduler need to do the same and mount the service account to the tf-job. Otherwise tf-job can't write output to GCS. E.g. here
4. For other places that we create pod that reads data from GCS, we need to do the same. E.g tensorboard
5. Kaniko build which is used by notebook to build images also need to be part of this effort, since it touches cloud registry.

[IronPan]

Update-
Tensorboard fixed #273
Kaniko fixed #343
DSL supports volume ane env APIs now #300

TODO:
Update current released sample to use the strongly typed GCP op #314

[jlewi]

@IronPan what is the remaining work here?

[IronPan]

@Ark-kun How is the work to migrate the samples to use the gcp credential?

[jlewi]

@Ark-kun @IronPan Any update on this issue?

[IronPan]

The samples are updated to have the right permission now.