kdep stands for Kube Deploy and is a thin wrapper around Helm that does two things:
- Automatically provisions secrets from Vault during deployment
- Simplifies chart configuration for multiple environments
- How it works
- Key features
- Installation
- Usage
- Overview & conventions
- Values files
- Working with secrets
- Related work
- Questions & suggestions
kdep
is written in bash and is the command you call to install and upgrade your app (instead of helm install/upgrade
).
When called, it:
- Looks at the values files in your chart to see which secrets to fetch from Vault.
- Reads those secrets and saves them as Kubernetes secrets, optionally running them through a template.
- Configures your chart to point to these Kubernetes secrets on the fly.
- Deploys your chart via Helm.
- The secrets are never stored to disk.
- The name of each generated Kubernetes secret includes a hash of the secret itself, meaning:
- Deployments and rollbacks are immutable - a secret change is only propagated through a new deploy.
- Pods are automatically restarted when the name of the secret they reference changes.
- One command for both installs and upgrades.
- Tiny footprint (200 loc bash) makes it easily customizable.
Download a release and unzip it into your PATH.
Required dependencies which you may already have:
kdep [-d] [-v] [-i] [-t temp-name] [-n namespace] path/to/chart/[[region-]env-]values.yaml
Flags:
-d Debug. Skips install/upgrade by adding '--dry-run --debug' to helm upgrade command
-v Output values. Skips install/upgrade. This option makes the script create the secrets
in Kubernetes and then output the generated values file that would be passed to
helm upgrade.
-i Ignore if the passed in file doesn't exist. Useful mostly for cicd when we want to just
pass in a blanket command like 'kdep chart/useast-dev-values.yaml' not worrying
whether 'useast-dev-values.yaml' exists or only 'values.yaml' is present for example.
-t temp-name Temporary release name. Launch a chart meant to be deleted shortly after, useful for
testing. This flag specifies a custom release name. When - is specified, a release
name will be generated.
-n namespace Override the default namespace. The default namespace is inferred from the first word of
the chart name. When an argument is provided, the namespace will be added as a prefix
to the release name, which by default is the name of the chart.
- Logical structure
- Microservices
- Every microservice lives in its own git repository.
- Closely related microservices are grouped together and thought of as "Apps".
- "Apps" (groups of microservices)
- Every "App" has its own git repo.
- The "App" git repo consists of Helm chart(s) - one chart for each of the microservices that make up this "App".
- Charts in this repo are named using lowercase letters, numbers, and dashes, something like api-order-history.
- The chart's first word ("api" from above example) is the Kubernetes namespace that this chart is deployed to.
- Generally, all of the charts in a particular "App" belong to one namespace, and this namespace is used only by this "App", though this is not a requirement.
- Inside each chart are staggered values files which define two things:
- The configuration used for deployment to different clusters.
- The secrets this chart needs from Vault.
- Microservices
- Continuous Integration (CI)
- A CI job (Jenkins/Travis/etc) is configured for each microservice repository.
- The CI job generally runs tests, builds a docker image, tags it with the git commit hash, and pushes it to a docker registry.
- The last step of the CI job is to create a Pull Request to the "App" repository, updating the image tag in the relevant chart.
- Continuous Deployment (CD)
- Structure
- Pull Requests to an "App" repository can be deployed to test clusters and tested.
- Merges into an "App" repository can be deployed to staging/production clusters.
- An "App" repository can have multiple branches, for example, one for staging and one for production.
- Approaches to deploying PRs and merges
- Push-based approach
- The idea is to push the changed charts into the clusters from a tool that runs outside the clusters.
- Can be done using various tools like Jenkins/Travis/Spinnaker/etc.
- Pull-based approach (cluster level)
- The idea is something runs inside that cluster keeping it up to date.
- Can be done using something like Brigade
- Pull-based approach (application level)
- The idea is every "App" in every cluster updates itself
- Can be done via quickcd
- This is the approach used by the team behind
kdep
- Push-based approach
- Structure
Part of most charts is a file named values.yaml
. This file contains the default viles to be inserted into the chart's templates. This works well for deploying to one cluster but when deploying to multiple clusters, the values can be slightly different for each. A couple examples would be a different ingress domain, or different database connection parameters, etc. To store this information declaratively, we introduce the concept of staggered values files with the following naming convention: [[region-]env-]values.yaml
.
Staggered values files work as follows: inside values.yaml
we store all the most common configuration. Then in something like prod-values.yaml
we store any differing values to be applied to clusters in the production environment. Then in something like aus-prod-values.yaml
we store any values specific to production clusters running in the Australia region. Settings in a more specific values file will override the settings in the more general ones. These files are all optional and used only when needed - if all deployments of an app are the same, a single values file will suffice.
An example of how the staggered values files work:
Let's say we want to launch the api-order-history
chart into the production cluster in Australia. Let's assume this chart is already in our current working directory. The command for this scenario would be:
kdep api-order-history/aus-prod-values.yaml
When invoked with the above arguments, kdep
will first read values.yaml
and store the settings in memory. It would then read prod-values.yaml
and merge the settings from it into the currently stored settings, overriding any differences. It would then read aus-prod-values.yaml
and once again merge it into the currently stored settings, overriding any differences. In the end, the merged settings will be fed to the helm install
command, which will provision the chart in the cluster we're set up to talk to. Internally, kdep
uses kubectl and helm and therefore, talks to the cluster that these commands are configured to talk to in your current environment.
One thing to note for the example above is that if the file aus-prod-values.yaml
does not exist, kdep
will throw an error as it expects to be fed a real file path which we find to be more useful for development. When using kdep in a CI/CD context and deploying various charts, the -i
flag is useful to ignore this error and allows kdep
to be called in a standard way for all charts.
Before calling Helm to deploy a chart, kdep loads required secrets from Vault. It then creates those secrets in Kubernetes under the namespace of the chart. The name of each secret includes a hash of the secret itself, making for a unique name for each unique secret. After all the secrets are provisioned, kdep injects the generated secret names into the chart so that the Kubernetes secrets can be consumed by pods, as an env vars or volume mounts. Optionally, a configuration file template may be used into which kdep will inject secrets and then save the entire result as a Kubernetes secret that can be consumed by a pod through a volume mount.
Two authenticate with Vault there are two options. Option one is via the env var VAULT_TOKEN
. If VAULT_TOKEN
is not set, kdep uses option two, which gets two secrets from Kubernetes, vault-role-id
, and vault-secret-id
, which were once manually created. With this option, kdep will exchange the AppRole credentials (the two secrets) for a token and use the token to communicate with Vault from then on.
This is done in a values file via the kdep.secrets
key, for example:
kdep:
version: 1
secrets:
sec1: /generic/user/roman/myservice/sec1
sec2: /generic/user/roman/myservice/sec2
envSecret: /generic/user/roman/myservice/sec3
The above defines three secrets and specifies Vault paths for each. During deployment and before passing the values to Helm, kdep will fetch the referenced secrets, create them in the cluster, and turn the above into:
kdep:
version: 1
secrets:
sec1: name-of-dynamically-created-secret-in-cluster-sdf42
sec2: name-of-dynamically-created-secret-in-cluster-4sdg6
envSecret: name-of-dynamically-created-secret-in-cluster-f42zc
In other words, the secret paths are replaced by the generated secret names. This means that in the chart's template, this secret can be referenced like this: {{ .Values.kdep.secrets.sec1 }}
.
There are two main ways for a pod to consume these secrets - by environment variable or by a filesystem mount.
Here is an abridged example of both:
apiVersion: apps/v1beta2
kind: Deployment
metadata: ...
spec:
template:
...
spec:
containers:
- name: myservice
image: "nginx:latest"
env:
- name: SECRET_FROM_ENV
valueFrom:
secretKeyRef:
name: {{ .Values.kdep.secrets.envSecret }}
key: value
volumeMounts:
- name: param1
mountPath: "/tmp/sec1"
subPath: value
- name: param2
mountPath: "/tmp/sec2"
subPath: value
volumes:
- name: param1
secret:
secretName: {{ .Values.kdep.secrets.sec1 }}
- name: param2
secret:
secretName: {{ .Values.kdep.secrets.sec2 }}
Sometimes, an application requires secrets in a configuration file. One way to accomplish this is to provide each secret to the container and then construct a special configuration file in the container's entrypoint script on startup. Kdep offers an alternative, perhaps simpler, solution. Instead of saving the configuration file template into the container, to be populated by the entrypoint script, save it to the chart folder. It will then be populated by kdep with secrets from Vault, saved as a secret in the cluster, and mounted in a pod as a volume.
Config file templates are specified in a values file via the kdep.files
key:
kdep:
version: 1
files:
main_conf:
template: main.conf.ctmpl
secretPaths:
hostname: /generic/user/roman/mysvc/hostname
mysqlPassword: /generic/user/roman/mysvc/mysqlpassword
kafkaBrokers: /generic/user/roman/mysvc/kafkabrokers
During deployment and before passing the values to Helm, kdep will run main.conf.ctmpl
template through consul-template
, save the output as a secret in the cluster, and turn the above into:
kdep:
version: 1
files:
main_conf: name-of-dynamically-created-secret-in-cluster-hje56
Let us disect the kdep.files
specification above:
- main_conf: this is a name used to refer to this config file secret in the chart's templates (see below)
- template: this key specifies the filename of the configuration file template that will be filled with secrets. This file should be placed in the chart folder (same folder as Chart.yaml)
- secretPaths: this maps Vault paths to friendly secret names. The Vault path can also be a "folder", for example if you intend to iterate through all items of a Vault "folder" inside the template.
The templating language used in the config file templates is that of consul-template and is very flexible.
Here is an example of a config file template that uses the friendly names defined above:
service_hostname = {{ with secret "${hostname}" }}{{ .Data.value }}{{ end }}
mysql_password = {{ with secret "${mysqlPassword}" }}{{ .Data.value }}{{ end }}
kafka_servers = { {{ range secrets "${kafkaBrokers}" }}
{{ with secret (printf "${kafkaBrokers}/%s" .) }}"{{ .Data.value }}";{{ end }}{{ end }}
}
During deployment, kdep
will call consul-template
, which will turn the above into the below, and store it as a secret in the cluster.
service_hostname = mainserver.local
mysql_password = 123456
kafka_servers = {
"1.2.3.4";
"5.6.7.8";
}
The details of the templating engine can be found here: https://github.com/hashicorp/consul-template#templating-language
Consuming config file secrets from a pod is the same as consuming regular secrets described above, here is an example:
apiVersion: apps/v1beta2
kind: Deployment
metadata:
...
spec:
template:
...
spec:
containers:
- name: myservice
image: "nginx:latest"
volumeMounts:
- name: fileconf
mountPath: "/tmp/main.conf"
subPath: value
volumes:
- name: fileconf
secret:
secretName: {{ .Values.kdep.files.main_conf }}
The above configuration will make the populated version of main.conf.ctmpl
available to the container at the path /tmp/main.conf
.
- https://github.com/GoogleContainerTools/skaffold
- https://github.com/roboll/kube-vault-controller
Please create an issue.