sts-annotator

How it works
- Limitations
Installation
Configurations
Tips
- Generating TLS certificate

sts-annotator is Kubernetes admission controllers, which is an API providing custom admission review in your cluster. The goal of this controller is to allow pods to get POD_INDEX and POD_REPLICAS from Environment Variables. sts-annotator achieve it by adding annotations to any pods managed by Statefulset at the time of CREATE and UPDATE. The POD_INDEX is the pod's ordinal index suffixed to the Statefulset Pod Name.

If you want to understand basics, read A Guide to Kubernetes Admission Controllers . If you want to understand required specifications over the custom APIs, read Dynamic Admission Control .

How it works

sts-annotator add annotations to any pods managed by Statefulset at the time of CREATE or UPDATE. It looks like below.

A Statefulset POD (api-somethings-v1-0) is successfully annotated

kind: Pod
apiVersion: v1
metadata:
  name: api-somethings-v1-sts-0
  namespace: default
  annotations:
    sts-annotator/pod-index: "0"
    sts-annotator/pod-replicas: "2"

You can get the annotations at your Environment variable, accessing to the annotations by fieldRef.

How to get the pod-index and pod-replicas from Env Vars

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: "api-somethings-v1-sts"
spec:
  template:
    spec:
      containers:
        - env:
            - name: POD_INDEX
              valueFrom:
                fieldRef:
                  fieldPath: metadata.annotations['sts-annotator/pod-index']
            - name: POD_REPLICAS
              valueFrom:
                fieldRef:
                  fieldPath: metadata.annotations['sts-annotator/pod-replicas']

You can see the Environment variables correctly loaded when ssh into the pod.

A Statefulset POD (api-somethings-v1-0) is successfully load env vars

# POD: api-somethings-v1-sts-0, Statefulset Replicas: 3
$ env | grep POD
POD_REPLICAS=2
POD_INDEX=0

Limitations

sts-annotator can add annotations to pods when they are initially created and never update it. So that it won't update the metadata.annotations['sts-annotator/pod-replicas'] of existing pods when you change the number of replicas in Statefulset unless you recreate all pods. If this is critical for your application, please raise an issue. We are thinking to add an option of auto remediation, which will re-create all pods under the updated Statefulset.

Installation

Prerequisites

k8s cluster is running

Quick Install

We used to use a set of static yaml files to deploy sts-annotator but Helm is supported now and recommend you to use Helm to install sts-annotator now.

If you want to quickly test sts-annotator, please follow the steps below. The quick install uses the pre-generated certificate and requires sts-annotator deployed in sts namespace.

# clone source code and move into the directory
$ git clone https://github.com/mk811/sts-annotator.git
$ cd sts-annotator

# the namespace name must be sts. Other namespace name is not supported by the TLS certificate
$ kubectl create ns sts
$ helm repo add mk811 https://raw.githubusercontent.com/mk811/sts-annotator/master/chart
$ helm install test mk811/sts-annotator  -f test/values.yaml -n sts

Custom Install

You can install sts-annotator with your own configurations.

To deploy sts-annotator to your namespaces, you need to generate a server certificate for sts-annotator by yourself. This is simply because MutatingWebhookConfiguration sends TLS encrypted request to sts-annotator. Please see Generating TLS certificate to find the way to generate TLS certificate.

Once you get cert, key, certificate authority, please create a values.yaml file in your local machine and set base64 encoded cert, key and CA in the file. Also, you can add more configurations as you like. Please see the Configurations section for available parameters.

# values.yaml
tls:
  cert: "LS0tLS1..."
  key: "LS0tL..."
  caCert: "LS0tL..."

Now, let's move onto the installation.

# Create a namespace for the sts-annotator.
$ kubectl create namespace <your-namespace>

# In this step, helm uses the values.yaml created step above.
$ helm repo add mk811 https://raw.githubusercontent.com/mk811/sts-annotator/master/chart
$ helm install <name e.g. `test`> mk811/sts-annotator -f values.yaml -n <your-namespace>

NAME: test
LAST DEPLOYED: Mon Aug  2 17:49:58 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

Configurations

Default values for all configurations are defined in values.yaml.

sts-annotator configurations

config is a section for configuring sts-annotator behavior.

field	description
server.mode	if you set 'debug', the api will run with 'debug' mode.
server.log.level	log level, either of 'error' , 'info' or 'debug',
server.port	port where the `sts-annotator` listens to
server.tls.cert	path to the cert file for `sts-annotator.<namespace>.svc.cluster`
server.tls.key	path to the private key for `sts-annotator.<namespace>.svc.cluster`
k8s.url	combination of url and port where your master api listens to
k8s.token	path to the k8s token file, used to call master api
k8s.tls.caCert	path to the ca file, used to call master api

sts-annotator deployment configurations

app is a section for configuring sts-annotator deployment configurations

field	description
replicas	number of replicas of sts-annotator
labels	optional labels for sts-annotator deployment and pod
image	sts-annotator image
tag	sts-annotator image version
env	default set of environment variables
imagePullPolicy	k8s imagePullPolicy for sts-annotator deployment
resources	k8s resources for sts-annotator deployment
affinity	k8s affinity configuration for sts-annotator deployment
livenessProbe	k8s livenessProbe configuration for sts-annotator deployment
readinessProbe	k8s readinessProbe configuration for sts-annotator deployment
volumes	k8s volume configuration for sts-annotator deployment
volumeMounts	k8s volumeMounts configuration for sts-annotator deployment

MutatingWebhookConfiguration Configurations

mutationWebhook is a section for configuring MutatingWebhookConfiguration resource.

field	description
apiVersion	`apiVersion` of `MutatingWebhookConfiguration`. `v1` or `v1beta`. `v1` is recommended as it is latest version.
failurePolicy	failurePolicy defines how unrecognized errors and timeout errors from the admission webhook are handled. Allowed values are `Ignore` or `Fail`.
sideEffects	The .webhooks[].sideEffects field should be set to None if a webhook doesn't have any side effect.

TLS Certificate Configurations

tls is a section for configuring TLS certificate used within sts-annotator server. Default values for the fields under TLS section aren't defined as it should be different on your environment.

field	description
cert	BASE64 Encoded Certificate
key	BASE64 Encoded Private Key for the Certificate
caCert	BASE64 Encoded CA Certificate

Tips

Generating TLS certificate

If you have k8s admin permission (or appropriate permissions), you can generate valid server certificate for sts-annotator in k8s cluster. Please refer to k8s official docs for details. The following steps are the sample commands used to generate certificate test/test.crt and test.key.

Note, we verified the following commands on MacOS only. base64 or other MacOS native commands may be vary if you are using Windows or Linux systems.

Create a Certificate Signing Request

Please update hosts for your namespace where the sts-annotator is deployed. The sample below expects sts-annotator is deployed in sts namespace. If you deploy it to default namespace, the hosts should be: ["sts-annotator.default.svc.cluster.local", "sts-annotator.default.svc", "sts-annotator"].

cat <<EOF | cfssl genkey - | cfssljson -bare server
{
  "hosts": [
    "sts-annotator.sts.svc.cluster.local",
    "sts-annotator.sts.svc",
    "sts-annotator"
  ],
  "CN": "system:node:sts-annotator.sts.pod.cluster.local",
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
  "names": [
    {
      "O": "system:nodes"
    }
  ]
}
EOF

Create a Certificate Signing Request object to send to the Kubernetes API

cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: sts-annotator
spec:
  request: $(cat server.csr | base64 | tr -d '\n')
  signerName: kubernetes.io/kubelet-serving
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

This command generates two files; it generates server.csr containing the PEM encoded pkcs#10 certification request, and server-key.pem containing the PEM encoded key to the certificate that is still to be created.

Let's base64 encode the key as we will use it in installation.

$ cat server-key.pem | base64 > server-key.base64.pem

Also, now you should be able to see the certificate signing request.

kubectl get csr sts-annotator

Approve CSR

If you have admin permission, you can approve the CSR by command below.
```
kubectl certificate approve sts-annotator
```
Get server certificate (base64 encoded)
```
kubectl get csr sts-annotator -o jsonpath='{.status.certificate}' > server.base64.crt
```
Now, you can find the server-key.pem in your local and you can download certificate.
Get Certificate Authority signed the Certificate. (If you don't know how to get the Certificate Authority)

Here are some tips to get Kubernetes Certificate Authority.
1. get through kubectl command
  There is a chance to get Certificate Authority by the command below. However, you might not be able to get it. (At least, I couldn't get CA for some environments)
```
kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' > ca.base64.crt
```
1. retrieve from pod
  Pick up any pod you prefer and run the command below. You can copy the Kubernetes Master API Certificate Authority from POD.
```
POD_NAME="<your/pod/name>"
NAMESPACE="<your/namespace>"
kubectl exec sts-annotator-869bb7f4b-9cdc6  -- cat "/run/secrets/kubernetes.io/serviceaccount/ca.crt" | base64 > ca.base64.crt
```
It is also okay to generate new certificates by following here or you can go with the steps above. In either case, please discuss the certificate lifecycle management with your system admin for production use.

Done! Finally, you have server certificate, key and CA in your local.

$ ls | grep base64
ca.base64.crt
server-key.base64.pem
server.base64.crt

Name		Name	Last commit message	Last commit date
Latest commit History 20 Commits
build		build
chart		chart
scripts		scripts
test		test
.gitignore		.gitignore
.releaserc.json		.releaserc.json
README.md		README.md
config.go		config.go
controllers.go		controllers.go
controllers_test.go		controllers_test.go
go.mod		go.mod
go.sum		go.sum
main.go		main.go
main_test.go		main_test.go
models.go		models.go
services.go		services.go
services_test.go		services_test.go
utils.go		utils.go

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

sts-annotator

How it works

Limitations

Installation

Prerequisites

Quick Install

Custom Install

Configurations

sts-annotator configurations

sts-annotator deployment configurations

MutatingWebhookConfiguration Configurations

TLS Certificate Configurations

Tips

Generating TLS certificate

About

Releases 1

Packages

Contributors 2

Languages

kikyomits/sts-annotator

Folders and files

Latest commit

History

Repository files navigation

sts-annotator

How it works

Limitations

Installation

Prerequisites

Quick Install

Custom Install

Configurations

sts-annotator configurations

sts-annotator deployment configurations

MutatingWebhookConfiguration Configurations

TLS Certificate Configurations

Tips

Generating TLS certificate

About

Resources

Stars

Watchers

Forks

Releases 1

Packages 0

Contributors 2

Languages

Packages