[WIP] Add PersistentVolume support to Habitat type

[asymmetric]

This PR adds a Persistence key to the Habitat type, which allows users to setup persistent storage.

TODO

• Mounting a PersistentVolume in each Pod.
• Validate Size field, so that the Habitat object creation fails if it's invalid. Right now, the Habitat gets created, while the Deployment fails. Either here or as part of Use Schema for validation instead of ad-hoc code #185.
• React on PV/PVC updates/deletions/creations
  • is this an aspect where a StatefulSet would require less work?
    • No, we would still have to tie the Habitat to the PV.
• Figure out if we want to support changing the size of the PV
• Figure out if we want to add support for custom StorageClasses
  • The downside is that the user can get overwhelmed
  • The upside is that it's the precondition for dynamic provisioning
• Figure out if Spec.Persistence.Enabled is a good idea. Normally, the field presence is enough to signal intent: the presence of Spec.Persistence would mean that the user wants it.
• Figure out which storage plugin we can/should use
  • HostPath provides no data replication on multi-node clusters
• Add README

Closes #181.
Ref https://github.com/kinvolk/habitat-operator/pull/195.

[asymmetric]

Adding reviewers for early feedback, feel free to ignore if you're only interesed in the finished product :)

[jeremymv2]

@asymmetric here's an interesting article that describes how even though PersistentVolumes are certainly available for use in Deployments, they make most sense in StatefulSets.

https://blog.thecodeteam.com/2017/08/16/technical-dive-statefulsets-deployments-kubernetes/

Thoughts?

[lilic]

As I mentioned before I think going the way of StatefulSets would be better IMHO.

One of the problems I see is with this is the ReadWriteOnce which is the access kind you are using, and that means the volume can be mounted as read-write by only a single node. And because most types support only ReadWriteOnce access mode, we could only support a few volume plugins.

[lilic]

I don't think you want a Volume, judging by this PR description and the comment above you want type PersistentVolume.

[asymmetric]

No, this is a Pod's Volume. In a manifest it would be the volumes key:

kind: Pod
apiVersion: v1
metadata:
  name: mypod
spec:
  containers:
    - name: myfrontend
      image: dockerfile/nginx
      volumeMounts:
      - mountPath: "/var/www/html"
        name: mypd
  volumes:
    - name: mypd
      persistentVolumeClaim:
        claimName: myclaim

[asymmetric]

But I can rename the variable from pv to v to make this less confusing.

[asymmetric]

@lilic Isn't the AccessModes issue orthogonal to Deployment vs StatefulSet? Meaning, StatefulSets would have the same issue, because under the hood they also use PersistentVolumes, with their limitations.

[lilic]

@asymmetric In a StatefulSet each pod gets their own PersistentVolumeClaim whereas in a Deployment each pod has the same PersistentVolumeClaim. This is why StatefulSet has a PersistentVolumeClaimTemplate as it creates PersistentVolumeClaims for each Pod off of that template. This is why the AccessMode is relevant when using a Deployment, it would be multiple Pods using the same Volume.

[asymmetric]

@lilic Ah I see, thanks! This is a good reason why StatefulSets could make more sense, I agree. Nice to have finally found one out! :)

I'll see if it's doable to implement this behaviour with Deployments first, and otherwise I think we have settled the argument.

[lilic]

@asymmetric

I'll see if it's doable to implement this behaviour with Deployments first, and otherwise I think we have settled the argument.

Hmm...can you explain that a bit? Wouldn't that be just trying to reimplement StatefulSets?

[asymmetric]

@lilic yes, but I was trying to figure out if it's possible to do that without the other changes in semantics (mainly ordered creation/removal). It seems pointless though, guess we have to accept those as a tradeoff.

[lilic]

@asymmetric According to the docs from 1.7 onwards that works fine https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-management-policies

[asymmetric]

Closing in favor of StatefulSets.