Deploy Model DB with durable storage

[jlewi]

In 0.4.0 we should be able to deploy ModelDB with durable storage so that it can be used for primitive model tracking.

Context:
kubeflow/katib#137
kubeflow/community#195
kubeflow/examples#134

I know there's discussion about redesigning model tracking from scratch (kubeflow/community#195 ) but I don't see that landing in 0.4.

ModelDB on the other hand covers a lot of the use cases we care about. I wouldn't expect it to be difficult to deploy ModelDB with durable storage (e.g. PVC). That would unblock a lot of useful scenarios as people could begin using it to track models.

Related:
ModelDB v2: goo.gl/NaDXpQ

/cc @mpvartak
/cc @garganubhav

[jlewi]

I think there are two databases used by ModelDB

1. There is an instance of MongoDB
   This is the dbDeployment
   https://github.com/kubeflow/kubeflow/blob/master/kubeflow/katib/modeldb.libsonnet#L108

2. There is ModelDB itself which I think uses an SQLLite database
   https://github.com/kubeflow/kubeflow/blob/master/kubeflow/katib/modeldb.libsonnet#L44

So both of these need to be stored on PVC.

For MongoDB we are just using the standard MongoDB image; so I'm guessing there are command line arguments or environment variables that we can use to store the database on a PVC.

For ModelDB I think we might need: VertaAI/modeldb#310

It looks like that PR makes some changes to store the model in the directory "/db" which we could then mount on PV.

/cc @mpvartak @garganubhav

[jlewi]

@mpvartak To integrate this into Kubeflow you'll want to port over the changes in VertaAI/modeldb#310 into the modeldb.libsonnet file at the locations mentioned in the previous comment.

Once you have the changes integrated you can follow the instructions for using kfctl
https://www.kubeflow.org/docs/started/getting-started-gke/

To deploy Kubeflow on GCP.

If you run it from your local repo; it will use whatever changes you've made to the Katib/Modeldb components.

[jlewi]

@mpvartak any update on this?

[jlewi]

@mpvartak @YujiOshima @gaocegege; anyone working on this?

[mpvartak]

Yep! We got the kssonnet figured out. Please watch for a PR to ModelDB in the next day or two.

…

On Mon, Dec 3, 2018 at 10:41 AM Jeremy Lewi ***@***.***> wrote: @mpvartak <https://github.com/mpvartak> @YujiOshima <https://github.com/YujiOshima> @gaocegege <https://github.com/gaocegege>; anyone working on this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1860 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFeWwz-EBmOcGlStRkJSUZ_cI3Tttjm0ks5u1XBOgaJpZM4X6opV> .

[jlewi]

@mpvartak Thanks!

Looks like #2050 is pending. Would be great to get that merged today so we can include it in the 0.4 release.

[jlewi]

#2050 was merged so closing this out.