[Epic] Kubeflow 1.3 upgrade

[wg102]

Epic to describe and track what needs to be done to upgrade Kubeflow to 1.3

Kubeflow 1.3 upgrade

Kubeflow upgrade might be doable in two time, update to 1.3, and then deal with jupyter-api related things. This way one person could work on upstreaming central-dashboard without big rebasing, while another could get the CWA up to our needs.

The concerns:

• Losing features
• 'Merging' jupyter-apis back in
• Unknown changes in other places
• New security isssues

The items from kubeflow/kubeflow

• jupyter-api -> crud: Big item to get back under the 1 repo biggest trouble
  • Python -> Go: Go back to the same CWA structure. with a new folder. Create a folder backend-go for every backend folder?
  • Missing things in CWA when they converted it
    • gpu selecting validation
    • max cpu
    • loading messages when error
    • notebook without a value "Erreur in backend"
    • A few other validation messages
  • Features/fix we did to jupyter-apis that is ours
    • Kubecost
    • image help text
    • Advance settings
    • KF_LANG
  • Deployment needs to be adjusted as well

Things not upstreamed/yet

 - The KF_LANG Variable + settings
 - Validation for mounted PVC

Anything upstream that is not merged yet...

 - drop down
 - font service
 - CWA OL

• Volume Manager:
  • Remove from jupyter-apis (or don't touch if we just change to CWA)
  • Possibly need to disable browse option, or create.
  • Friendlier message for delete?
• Updated Manifest (more integrated with actual central dashboard structure)*
• KFP client supports authorization checks by using service accounts
• Manage Contributors (Does not seem upstream Managing contributors manually kubeflow/kubeflow#5847)
• Kubeflow Pipelines (KFP):
  • UI reorganization for better User Experience
  • Manage recurring Runs via new “Jobs” page (exact name on UI is TBD)
  • Simplified view of dependency graphs
• New UI for Tensorboard
• Need to reconfigure so we use CWA instead of jupyter-apis
  • Move code
  • Redo all the subitems not in CWA...

Could it be possible to do it in two times.
Upgrade to 1.3 Then upgrade to using CWA?

Note: upstreaming: no central dashboard i18n or manage contributors yet

The other items

• kubeflow-manifest needs to be updated. Structure of upstream kubeflow/manifests has changed and may require modifications to ours See [EPIC] Kubeflow Upgrade Planning v1.3.1 aaw-kubeflow-manifests#110
• Pipelines (we have a plan to use our i18n pipelines)
• Possible need to update kubeflow-controller since it relates to the jupyter-apis
• {...}

[davidspek]

@wg102 I think updating to Kubeflow 1.3 will probably be much easier if you move to an Argo CD based installation. This also removes the reliance on the kubeflow manifests repo as you can use all the upstream component manifests directly. This also allows for easier maintenance in the future where you can simply upgrade each component as new versions are released, and PRs for this can be automated with Renovate (here is a good example of such a PR to update KFP).

Given that work needs to be put into reformatting your manifests now anyways, it seems like a good opportunity to move to a deployment that is easier to maintain and version track changes using Git. As I've already told @ca-scribner I'm happy to help with migrating your manifests so they can be deployed by Argo CD.

[davidspek]

Regarding the Volumes Web App, there is no browse option by default, so this is not a problem. What could be a problem is that the Volumes Web App allows users to choose the Storage Class when creating a new PVC, something the Jupyter Web App doesn't allow users to do which was a conscious decision at the time.

[davidspek]

Also, the CRUD Jupyter Web App does have a few Advanced Options sections during notebook creation that might be useful if you need to transition some options.

[blairdrummond]

@wg102 I think updating to Kubeflow 1.3 will probably be much easier if you move to an Argo CD based installation. This also removes the reliance on the kubeflow manifests repo as you can use all the upstream component manifests directly. This also allows for easier maintenance in the future where you can simply upgrade each component as new versions are released, and PRs for this can be automated with Renovate (here is a good example of such a PR to update KFP).

Given that work needs to be put into reformatting your manifests now anyways, it seems like a good opportunity to move to a deployment that is easier to maintain and version track changes using Git. As I've already told @ca-scribner I'm happy to help with migrating your manifests so they can be deployed by Argo CD.

@brendangadd maybe an interesting opportunity here. Could try to tackle the ArgoCD install of Kubeflow into the kind experiment?

[davidspek]

@blairdrummond I’ve run the entire deployment on a 3 “node” kind cluster on my desktop. You’ll need a good amount of RAM, but it works as you would expect. I can also help with this if needed.

I’ve also started playing with KubeCost and extending the Prometheus and Grafana I’ve added to support namespace isolation. The next step in that journey is integrating KubeCost in some way into the central dashboard and possibly the crud Jupyter Web App down the road for per notebook metrics (Volumes Web App would need something similar for the PVC costs).

[brendangadd]

@sylus is optimistically looking into the Argo CD approach for KF 1.3. The approach we choose for deployment is his call.

@davidspek: Thank you for offering your support on this transition. If @sylus gives the 👍 on this deployment method, beginning with KF 1.3, then I'm sure we'll take you up on your offer! ❤️

[davidspek]

Just wanted to let you guys know I've reached out to @berndverst from Azure / Microsoft and he's interested in possible contributing to the ArgoFlow-Azure setup and potentially making that the default methods for Azure customers to install Kubeflow going forwards as it is hopefully easier to maintain (by not needing to deal with the Kubeflow manifests repo). If this ends up happening that could hopefully take some load off of StatCan as the deployment method could be maintained more by the community.

[rohank07]

Upstream pushed crud-web-app i18n support using localize, reverting ngx-translate. However, it uses AOT complication. To start the server in French you'd run ng serve --configuration=fr. With the ngx-translate approach we would change the browser language to French but it seems localize does not support this. Possible approach would be toggling the language via the url '/fr' which could mean changing how it will be deployed (and a lot more).
https://angular.io/guide/i18n

[wg102]

the i18n with angular of crud-web apps and it's impact is out of scope for the 1.3 upgrade since it was merged upstream in 1.4 see 1.4 release notes