Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement persistence with StatefulSets #201

Merged
merged 93 commits into from
Mar 21, 2018
Merged
Show file tree
Hide file tree
Changes from 87 commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
8591882
Add Persistence field to Habitat type
Feb 21, 2018
b4d958d
Create PVC
Feb 21, 2018
2244992
Extract PVC creation to helper method
Feb 21, 2018
3c7a657
Create PVC before Deployment
Feb 21, 2018
36fea97
Mount PersistentVolume in Deployment
Feb 21, 2018
25b4684
Make Persistence field pointer
Feb 23, 2018
64b642f
Add example for persistence
Feb 23, 2018
9b5d269
Rename directory
Feb 26, 2018
8ca2d45
Create StatfulSet instead of Deployment
Feb 26, 2018
ddcbb6d
Avoid creating PVC
Feb 26, 2018
b2ece1a
Add Selector field to StatefulSet
Feb 27, 2018
418837d
Create Headless Service
Feb 27, 2018
23e6b07
Add ServiceName field to StatefulSet
Feb 27, 2018
975696c
Add OwnerReference to StatefulSet in Service
Feb 27, 2018
989af04
Make persistence example leader-follower
Feb 27, 2018
578b823
Disable protected mode in redis container
Feb 27, 2018
e6826ab
Remove Headless Service
Feb 27, 2018
06d7daa
Add OwnerReference to Habitat in ConfigMap
Feb 27, 2018
88c2bcb
Set PodManagementPolicy flag
Feb 28, 2018
16cf8f6
Regenerate deepcopy functions
Feb 28, 2018
3361d92
Use constants
Feb 28, 2018
08cac69
Make slice creation more idiomatic
Feb 28, 2018
0b91640
order
Feb 28, 2018
9ecdcaf
Add RBAC rule for StatefulSets
Feb 28, 2018
833a720
Remove RBAC rule for Deployments
Feb 28, 2018
255cca3
Add StorageClassName key to Habitat type
Mar 5, 2018
92b099b
Remove PersistentVolume from example
Mar 5, 2018
6690c0b
Change StorageClassName in example
Mar 5, 2018
ffe1f86
Add comments for example fields
Mar 5, 2018
0e9765e
Add comments for Persistence struct fields
Mar 5, 2018
7449f74
Add label to PersistentVolumeClaim
Mar 5, 2018
7581d97
Add README for persistent storage example
Mar 5, 2018
5cfff30
Change mountPath in persisted example
Mar 5, 2018
e8e70de
Fix typo in error message
Mar 5, 2018
3d69f43
Rename ConfigMap handling function
Mar 5, 2018
a3a5ca1
Add PersistentVolume watching
Mar 5, 2018
ee9f83b
Add PersistentVolumeClaim watching
Mar 5, 2018
a9c2c66
Move StatefulSet logic to its own module
Mar 6, 2018
e7da267
Move PersistentVolume logic to its own module
Mar 6, 2018
feb2760
Move PersistentVolumeClaim logic to its own module
Mar 6, 2018
cf5efb8
Move persistentVolumeName constant
Mar 6, 2018
1f9475a
Rename checkPVC method
Mar 6, 2018
8a576bf
Reword TODO
Mar 6, 2018
538aaff
Add TODO
Mar 6, 2018
51a487d
Fix incorrect assignments
Mar 6, 2018
bb0bcb8
Move method upwards
Mar 6, 2018
4f3e1dc
Update RBAC for E2E tests
Mar 6, 2018
5efcdc1
Use constant from apiv1 instead of string literal
Mar 6, 2018
8914288
Output APIVersion with Sprintf
Mar 6, 2018
85e59be
Renam wApp to web in E2E tests
Mar 6, 2018
768e8f1
Remove number suffix from E2E test example
Mar 6, 2018
c7f421c
Use redis in standalone E2E test
Mar 6, 2018
4a8d061
Use defer to delete Service
Mar 6, 2018
93e7fa4
Add E2E tests for persistence
Mar 6, 2018
d534996
Add "section" comment in E2E test
Mar 6, 2018
2943d14
Delete persistent storage at end of tests
Mar 6, 2018
28f4578
Use runtime.Object in helper function
Mar 7, 2018
17597cf
Skip persistence tests
Mar 7, 2018
1b4c0d5
Add comment about redis protected mode
Mar 12, 2018
a1de425
Rename Persistence to PersistentStorage
Mar 12, 2018
2b263c2
Copy deeply nested key to variable
Mar 12, 2018
e0f0273
Run deepcopy code generator script
Mar 12, 2018
93f82dc
Update year in copyright notice for new files
asymmetric Mar 14, 2018
d002da3
Add context to error
asymmetric Mar 14, 2018
2f3adef
Replace link to docs/api.md with types.go
asymmetric Mar 14, 2018
fb9d9f8
Remove docs/api.md
asymmetric Mar 14, 2018
b6be7bc
Improve grammar in Persistentstorage comment
asymmetric Mar 14, 2018
b6a8401
Use fields from Habitat object
asymmetric Mar 14, 2018
d9399c9
Remove confusing comment
asymmetric Mar 14, 2018
71e0638
Add OwnerReference from StatefulSet to Habitat
asymmetric Mar 14, 2018
9caf4c0
Remove custom logic for StatefulSet deletion
asymmetric Mar 14, 2018
6041ff4
Replace kinvolk wiht habitat-sh
asymmetric Mar 14, 2018
951f9fe
Add PV and PVC caches to WaitForCacheSync call
asymmetric Mar 19, 2018
3aff775
Extract deeply nested keys to variables
asymmetric Mar 19, 2018
6dfc118
Extract else branch out of conditional
asymmetric Mar 19, 2018
89e4eea
Improve documentation of findHabForPVC method
asymmetric Mar 19, 2018
0573853
Clarify TODO in findHabForPVC
asymmetric Mar 19, 2018
78468bb
Rename findHabForPVC to checkHabForPVC
asymmetric Mar 19, 2018
f4b9093
Add cluster-level permissions for operator
asymmetric Mar 19, 2018
8d7720c
Remove rbac.yml
asymmetric Mar 19, 2018
d17efa3
Calculate APIVersion and Kind
asymmetric Mar 19, 2018
96d7aa1
Update PersistentStorage README
asymmetric Mar 19, 2018
8e9e0e2
Stop caching PVs and PVCs
asymmetric Mar 20, 2018
5d16382
Revert "Update PersistentStorage README"
asymmetric Mar 20, 2018
1ed628c
Fix key name in README
asymmetric Mar 20, 2018
4a6f295
Revert "Add cluster-level permissions for operator"
asymmetric Mar 20, 2018
64e7f10
Visually separate internal imports
asymmetric Mar 20, 2018
ca9e0c8
Always request the user to create their own SC
asymmetric Mar 21, 2018
4f327c9
Add RBAC rules for minikube
asymmetric Mar 21, 2018
efe0e8f
Remove stale link ot docs/api.md
asymmetric Mar 21, 2018
f3af7cd
Add link to PersistentVolume in example
asymmetric Mar 21, 2018
b8876df
Remove unneeded ClusterRoleBinding
asymmetric Mar 21, 2018
49da9a4
Add warning about minikube --vm-driver=none
asymmetric Mar 21, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ The Habitat operator is a Kubernetes controller designed to solve running and au

To learn more about Habitat, please visit the [Habitat website](https://www.habitat.sh/).

For a more detailed description of the Habitat operator API have a look at the [API documentation](https://github.com/habitat-sh/habitat-operator/blob/master/docs/api.md).
For a more detailed description of the Habitat type have a look [here](https://github.com/habitat-sh/habitat-operator/blob/master/pkg/apis/habitat/v1beta1/types.go).

## Prerequisites

Expand Down
39 changes: 0 additions & 39 deletions docs/api.md

This file was deleted.

37 changes: 37 additions & 0 deletions examples/persisted/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Persistent Storage example
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the whole point of persistence is that you can mount already existing volumes, can you maybe write up some steps of how a user can do that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explicitly avoided to do that because it's a big topic and I would only make a poorer job at explaining it than the official docs do, which I linked. I anyway added some commits after your comment which should hopefully make things clearer.

BTW, it's not necessary that the PV has been created beforehand, if the cluster supports dynamic provisioning.

TL;DR: IMHO explaining how persistence works in Kubernetes is beside the scope of this README.


Habitat objects are translated by the operator into `StatefulSet` objects, which
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Maybe add a note that the user needs to know about basic understanding of persistent volumes in K8s and link to it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I'll add a link to this

provide optional support for persistent storage.

In order to enable persistent storage for your Habitat object, you need to:

* create a
[`StorageClass`](https://kubernetes.io/docs/concepts/storage/storage-classes/) object in your cluster
* add the `spec.persistentStorage` key to the Habitat object's manifest
* specify the `name` of the aforementioned `StorageClass` object under
`spec.persistence.storageClassName` in the Habitat object's manifest

## Workflow

Before deploying the example, create a `StorageClass` object, specifying the
type of volume your cluster is able to provision.

**NOTE**: If you're deploying the example on GKE, a standard
`StorageClass` for `GCEPersistentDisk` has already been defined, so you can skip
the above step

Once the `StorageClass` has been created, run the example:

kubectl create -f examples/persisted/habitat.yml

When you want to delete the Habitat, run:

kubectl delete -f examples/persisted/habitat.yml

**NOTE**: Any `PersistentVolume` created by the operator will **NOT** be
automatically removed. This is the default behaviour of Kubernetes and is
intended as a safeguard against accidental data deletion.

If you want to explicitly delete the `PersistentVolume`, run:

kubectl delete pvc -l habitat-name=example-persistent-habitat
28 changes: 28 additions & 0 deletions examples/persisted/habitat.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
apiVersion: habitat.sh/v1beta1
kind: Habitat
metadata:
name: example-persistent-habitat
spec:
# the core/redis habitat service packaged as a Docker image
image: kinvolk/redis-hab
count: 3
# the presence of this key activates persistent storage
persistentStorage:
# the size of the volume that will be mounted in each Pod
size: 1Gi
# a StorageClass object with name "standard" must be created beforehand by
# the cluster administrator
storageClassName: standard
# the location under which the volume will be mounted
mountPath: /tmp/foobar
env:
- name: HAB_REDIS
# this is needed to make redis accept connections from other hosts
# see https://redis.io/topics/security#protected-mode
# NOTE: do not use this setting in a production environment
value: '{ "protected-mode": "no" }'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a note of warning here.

service:
name: redis
topology: leader
# if not present, defaults to "default"
group: foobar
2 changes: 1 addition & 1 deletion examples/rbac/rbac.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ rules:
- apiGroups:
- apps
resources:
- deployments
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we not need Volume or something similar here as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No I don't think so, as it's not the operator that's creating Volumes, it's the StatefulSet.

Copy link
Contributor Author

@asymmetric asymmetric Mar 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You were right btw, we did have to: the PV/PVC ListWatchers running in the operator means we need to assign list and watch permissions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I am guessing as this is not needed anymore the above comment is not valid, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly.

- statefulsets
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources:
Expand Down
16 changes: 16 additions & 0 deletions pkg/apis/habitat/v1beta1/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,20 @@ type HabitatSpec struct {
// The EnvVar type is documented at https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#envvar-v1-core.
// Optional.
Env []corev1.EnvVar `json:"env,omitempty"`
// Optional.
PersistentStorage *PersistentStorage `json:"persistentStorage,omitempty"`
}

// PersistentStorage contains the details of the persistent storage that the
// cluster should provision.
type PersistentStorage struct {
// Size is the volume's size.
// It uses the same format as Kubernetes' size fields, e.g. 10Gi
Size string `json:"size"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are missing the omitempty, or are these are required? Then we should document that as we don't have the api docs.md anymore. :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean? Right now we only explicitly document that a field is optional, otherwise it's mandatory.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, just before in the docs.md file it was more clear at least to me, what is required. But if it makes sense to you 👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICT that's also how its' done in upstream.

// MountPath is the path at which the PersistentVolume will be mounted.
MountPath string `json:"mountPath"`
// StorageClassName is the name of the StorageClass that the StatefulSet will request.
StorageClassName string `json:"storageClassName"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Can we name this just ClassName as it's under the PersistantStorage anyways. WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave it the same name it has in the PersistentVolume resource, because it maps 1:1 with that field. My hope is that it will make it easier for people to understand how things tie together.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, that makes sense!

}

type HabitatStatus struct {
Expand Down Expand Up @@ -102,6 +116,8 @@ const (

TopologyStandalone Topology = "standalone"
TopologyLeader Topology = "leader"

HabitatKind = "Habitat"
)

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
Expand Down
25 changes: 25 additions & 0 deletions pkg/apis/habitat/v1beta1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading