Skip to content

Latest commit

 

History

History
352 lines (276 loc) · 20.6 KB

10-managing-clusters.adoc

File metadata and controls

352 lines (276 loc) · 20.6 KB
Raw

Managing Clusters

Table of Contents

Foreword

Before proceeding, one should make themselves familiar with custom resource definitions and with the API spec document (in particular with the AerospikeCluster custom resource definition).

Prerequisites

Before creating an Aerospike cluster with aerospike-operator, one should make sure that their Kubernetes cluster has the required resources. The first thing one should make sure is that one’s Kubernetes cluster has at least as many nodes as the number of Aerospike nodes one intends to deploy. For example, if one wants to create an Aerospike cluster with two nodes, one must have two Kubernetes nodes in the cluster. This is because aerospike-operator enforces inter-pod anti-affinity, and as such will never co-locate two Aerospike pods in the same Kubernetes node.

⚠️
 Running aerospike-operator with the --debug=true flag effectively disables inter-pod anti-affinity, and is strongly discouraged outside testing environments.

After making sure that enough Kubernetes nodes are available, one should also make sure that these nodes have enough RAM to meet the demands of an Aerospike node. How much RAM needs to be available depends on several factors, but at the bare minimum it must be equal to the value of the memorySize field of the Aerospike namespace that the Aerospike cluster will manage.

⚠️
 aerospike-operator sets resource requests on every pod based on the value of the memorySize field. This, along with the fact that aerospike-operator enforces inter-pod anti-affinity, means that there must be at least .spec.nodeCount Kubernetes nodes in the Kubernetes cluster, and that each of these nodes must have at least .spec.namespaces[0].memorySize gibibytes of free memory. Failing to meet these prerequisites will cause pods associated with an AerospikeCluster resource not to be scheduled.

Finally, one should make sure that an adequate storage class is configured in the Kubernetes cluster. aerospike-operator dynamically provisions a persistent volume per namespace per Aerospike node, and as such expects a storage class supporting dynamic provisioning to be available. The size of said volume is equal to the value of the .spec.namespaces[0].storage.size field of the AerospikeCluster resource.

⚠️
 One should carefully capacity plan storage based at least on the estimated amount and size of the records in the Aerospike namespace, on the desired replication factor and on the desired number of nodes in the Aerospike cluster. One should also take into account that one should not exceed 50-60% capacity on the storage device [1].
ℹ️
 As an example, an Aerospike cluster with three nodes managing an Aerospike namespace with a replication factor of two requires approximately 67GiB of disk space per Aerospike node in order to store 100,000,000 keys of type string, each with 1024 characters. Hence, and taking the official recommendations [1] into account, the value used for .spec.namespaces[0].storage.size for such an Aerospike cluster should be between 112G and 134G.

Creating an Aerospike cluster

The interface for creating Aerospike clusters using aerospike-operator is the AerospikeCluster custom resource definition. This custom resource definition allows for specifying the desired topology of the cluster, as well as configuration properties for the Aerospike namespace it will manage.

The AerospikeCluster custom resource definition is namespaced, meaning that every Aerospike cluster is created and managed within a Kubernetes namespace. This makes it easy to manage multiple Aerospike clusters within the same Kubernetes cluster.

An example of an AerospikeCluster custom resource can be found below:

apiVersion: aerospike.travelaudience.com/v1alpha2
kind: AerospikeCluster
metadata:
  name: as-cluster-0
  namespace: kubernetes-namespace-0
spec:
  version: "4.2.0.3"
  nodeCount: 2
  namespaces:
  - name: as-namespace-0
    replicationFactor: 2
    memorySize: 4G
    storage:
      type: file
      size: 16G
      storageClassName: ssd

Creating such a resource will make aerospike-operator create an Aerospike cluster named as-cluster-0 inside the kubernetes-namespace-0 Kubernetes namespace [2]. The resulting cluster will:

  • Have two nodes (pods) running Aerospike 4.2.0.3 [3].

  • Manage an Aerospike namespace called as-namespace-0.

⚠️
 As described in the API spec document, and even though .spec.namespaces is an array, aerospike-operator enforces the existence of a single Aerospike namespace per Aerospike cluster. To create multiple Aerospike namespaces one should create multiple Aerospike clusters.

In its turn, the as-namespace-0 Aerospike namespace managed by this Aerospike cluster will:

  • Be configured to have a replication factor of two.

  • Use 4GiB of RAM for indexes and data.

  • Operate in the file storage mode using a 16GiB persistent volume provisioned using the ssd storage class.

To create the abovementioned Aerospike cluster, one may use kubectl as shown below:

$ kubectl create -f - <<EOF
apiVersion: aerospike.travelaudience.com/v1alpha2
kind: AerospikeCluster
metadata:
  name: as-cluster-0
  namespace: kubernetes-namespace-0
spec:
  version: "4.2.0.3"
  nodeCount: 2
  namespaces:
  - name: as-namespace-0
    replicationFactor: 2
    memorySize: 4G
    storage:
      type: file
      size: 16G
      storageClassName: ssd
EOF
aerospikecluster.aerospike.travelaudience.com "as-cluster-0" created

After a few seconds, listing pods in the kubernetes-namespace-0 Kubernetes namespace will reveal two pods:

$ kubectl -n kubernetes-namespace-0 get pod
NAME             READY     STATUS    RESTARTS   AGE
as-cluster-0-0   2/2       Running   0          2m
as-cluster-0-1   2/2       Running   0          2m

Each of these pods corresponds to an Aerospike node of the as-cluster-0 Aerospike cluster, and features two containers: aerospike-server (the Aerospike server itself) and asprom (an exporter of Aerospike metrics in Prometheus format [4]). Inspecting the logs for the aerospike-server container of any of these pods will reveal a working Aerospike cluster with size two and a namespace named as-namespace-0:

$ kubectl -n kubernetes-namespace-0 logs -f as-cluster-0-0 aerospike-server
Jul 02 2018 14:01:23 GMT: INFO (as): (as.c:319) <><><><><><><><><><>  Aerospike Community Edition build 4.2.0.3  <><><><><><><><><><>
(...)
Jul 02 2018 14:02:03 GMT: INFO (info): (ticker.c:171) NODE-ID bb90a023c0a580a CLUSTER-SIZE 2
(...)
Jul 02 2018 14:02:03 GMT: INFO (info): (ticker.c:408) {as-namespace-0} objects: all 0 master 0 prole 0 non-replica 0
(...)

aerospike-operator will also create a headless service in the kubernetes-namespace-0 Kubernetes namespace that can be used to discover Aerospike nodes and connect to the Aerospike cluster:

$ kubectl -n kubernetes-namespace-0 get svc
NAME           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                      AGE
as-cluster-0   ClusterIP   None         <none>        3000/TCP,3002/TCP,9145/TCP   2m

At this point, pointing an Aerospike client at as-cluster-0.kubernetes-namespace-0.svc.cluster.local will yield an output similar to the following, indicating a successful connection:

$ kubectl run --rm -i -t --restart Never \
    --image aerospike/aerospike-tools:3.15.3.10 \
    aerospike-tools \
    -- \
    asinfo -h as-cluster-0.kubernetes-namespace-0.svc.cluster.local
1 :  node
     BB907003C0A580A
2 :  statistics
     cluster_size=2;(...)
(...)

Inspecting an Aerospike cluster

As aerospike-operator works towards bringing the current state of an Aerospike cluster in line with the desired state, it will output useful information about the operations it performs against said cluster. This information is stored in the form of Kubernetes events associated with the target AerospikeCluster resource. To access the events associated with a specific AerospikeCluster resource, one can use kubectl as shown below:

$ kubectl -n kubernetes-namespace-0 describe aerospikecluster as-cluster-0
Name:         as-cluster-0
Namespace:    kubernetes-namespace-0
(...)
Events:
  Type    Reason       Age   From              Message
  ----    ------       ----  ----              -------
  Normal  NodeStarted  2m    aerospikecluster  aerospike started on pod kubernetes-namespace-0/as-cluster-0-0
  Normal  NodeStarted  2m    aerospikecluster  aerospike started on pod kubernetes-namespace-0/as-cluster-0-1

Listing Aerospike clusters

To list all Aerospike clusters in a given Kubernetes namespace, one may use kubectl as shown below:

$ kubectl -n kubernetes-namespace-0 get aerospikeclusters
NAME           VERSION   NODE COUNT   AGE
as-cluster-0   4.2.0.3   2            19m

One may also use the asc shorthand instead of aerospikeclusters, for brevity:

$ kubectl -n kubernetes-namespace-0 get asc
NAME           VERSION   NODE COUNT   AGE
as-cluster-0   4.2.0.3   2            19m

To list all Aerospike clusters in the current Kubernetes cluster (i.e. across all Kubernetes namespaces), one may run

$ kubectl get asc --all-namespaces
NAMESPACE                NAME           VERSION   NODE COUNT   AGE
kubernetes-namespace-0   as-cluster-0   4.2.0.3   2            19m
kubernetes-namespace-1   as-cluster-1   4.2.0.5   3            4m

Creating and deleting Aerospike namespaces

As described in the API spec document, an Aerospike cluster managed by aerospike-operator is limited to having exactly one Aerospike namespace. Hence, to create a new Aerospike namespace one must create a new AerospikeCluster resource. Similarly, to delete an existing Aerospike namespace one must delete the AerospikeCluster resource that contains it.

Updating the Aerospike configuration

In order to ensure a correct and consistent behaviour, aerospike-operator must take full ownership of every Aerospike cluster’s configuration file. This means that the aerospike.conf file used to configure Aerospike is generated and managed by aerospike-operator. It CANNOT be edited by the user. That being said, the AerospikeCluster custom resource definition exposes some configuration properties that can be tweaked by the user.

⚠️
 The fact that the configuration for an Aerospike cluster is fully managed by aerospike-operator means that it is currently not possible to set the value of configuration properties such as high-water-memory-pct or cold-start-empty to a value of the user’s choosing.

Some of the configuration properties exposed by the AerospikeCluster custom resource definition, such as replicationFactor, can only be set when creating the Aerospike cluster. Some other properties, such as memorySize, can be tweaked on a live Aerospike cluster.

When a configuration change to a live Aerospike cluster is detected, aerospike-operator will perform a rolling restart [5] on the cluster. This means that pods in the Aerospike cluster will be deleted and re-created one by one. In order to avoid data loss, aerospike-operator waits for all migrations on the a given pod to finish before deleting and recreating it, and will reuse existing persistent volumes containing namespace data when creating the new pod.

⚠️
 Since every Aerospike node must be cold-started [6], applying a configuration update to an Aerospike cluster can take up to several hours. The actual amount of time depends on factors such as the amount of data stored by each node and whether the restart causes evictions to occur. Configuration updates should be carefully planned before being applied.
❗
 Update operations against a given AerospikeCluster resource MUST NOT target the .status field or any of its subfields. In particular, this means that updates to AerospikeCluster resources should ALWAYS be done using kubectl edit or kubectl patch and double-checked for changes to .status. Commands such as kubectl replace may cause the .status field to be updated inadvertently, and may leave the target AerospikeCluster resource in an inconsistent or inoperable state.

Scaling an Aerospike cluster

As load increases or decreases, one may want to scale a given Aerospike cluster up or down. Scaling an Aerospike cluster can be done using the kubectl scale command. For instance, in the example above, the following command will cause aerospike-operator to create a new Aerospike node:

$ kubectl scale asc as-cluster-0 --replicas=3

Scaling an Aerospike cluster can also be done by directly editing the associated AerospikeCluster resource in order to update the value of the .spec.nodeCount field. For instance, setting .spec.nodeCount to three in the example above will also cause aerospike-operator to create a new Aerospike node:

$ kubectl -n kubernetes-namespace-0 edit asc as-cluster-0
(...)
aerospikecluster.aerospike.travelaudience.com "as-cluster-0" edited
$ kubectl -n kubernetes-namespace-0 get pod
NAME             READY     STATUS    RESTARTS   AGE
as-cluster-0-0   2/2       Running   0          8m
as-cluster-0-1   2/2       Running   0          8m
as-cluster-0-2   2/2       Running   0          2m
❗
 Update operations against a given AerospikeCluster resource MUST NOT target the .status field or any of its subfields. In particular, this means that updates to AerospikeCluster resources should ALWAYS be done using kubectl edit or kubectl patch and double-checked for changes to .status. Commands such as kubectl replace may cause the .status field to be updated inadvertently, and may leave the target AerospikeCluster resource in an inconsistent or inoperable state.

At this point, inspecting the logs for the new as-cluster-0-2 pod will reveal that it has successfully joined the existing cluster:

(...)
Jul 02 2018 14:18:40 GMT: INFO (info): (ticker.c:171) NODE-ID bb908003c0a580a CLUSTER-SIZE 3
Jul 02 2018 14:18:40 GMT: INFO (info): (ticker.c:247)    cluster-clock: skew-ms 0
Jul 02 2018 14:18:40 GMT: INFO (info): (ticker.c:277)    system-memory: free-kbytes 7193812 free-pct 93 heap-kbytes (2217693,2219016,2297856) heap-efficiency-pct 96.5
Jul 02 2018 14:18:40 GMT: INFO (info): (ticker.c:291)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
Jul 02 2018 14:18:40 GMT: INFO (info): (ticker.c:313)    fds: proto (0,7,7) heartbeat (2,3,1) fabric (48,48,0)
Jul 02 2018 14:18:40 GMT: INFO (info): (ticker.c:322)    heartbeat-received: self 0 foreign 801
Jul 02 2018 14:18:40 GMT: INFO (info): (ticker.c:353)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (0,0)
(...)

In a similar way, setting .spec.nodeCount back to two will cause aerospike-operator to delete the as-cluster-0-2 pod:

$ kubectl -n kubernetes-namespace-0 edit asc as-cluster-0
(...)
aerospikecluster.aerospike.travelaudience.com "as-cluster-0" edited
$ kubectl -n kubernetes-namespace-0 get pod
NAME             READY     STATUS        RESTARTS   AGE
as-cluster-0-0   2/2       Running       0          10m
as-cluster-0-1   2/2       Running       0          10m
as-cluster-0-2   0/2       Terminating   0          4m
⚠️
 It is not possible to set .spec.nodeCount to a value that is smaller than the value of the replication factor of the managed Aerospike namespace (i.e. the value of .spec.namespaces[0].replicationFactor). For instance, if a given Aerospike cluster manages an Aerospike namespace with a replication factor of three, it is not possible to scale said cluster down to less than three Aerospike nodes.

Deleting an Aerospike cluster

Deleting an Aerospike cluster is done by deleting the associated AerospikeCluster custom resource:

$ kubectl -n kubernetes-namespace-0 delete asc as-cluster-0
❗
 Deleting an AerospikeCluster custom resource will cause all nodes and data in the target Aerospike cluster to be deleted without notice. All data in the target Aerospike cluster will be effectively lost unless a previous backup exists. Persistent volumes associated with the Aerospike cluster will also be deleted.
❗
 When deleting an AerospikeCluster using kubectl delete one MUST make sure that the value of the --cascade flag is set to true. This is the default value for this command, and MUST NOT be changed. Running kubectl delete --cascade=false against an AerospikeCluster resource will cause existing dependent resources (pods, services, etc…​) to be left untouched (i.e. orphaned), requiring manual cleanup by an operator to be deleted from the Kubernetes cluster.
❗
 When deleting and recreating an AerospikeCluster using kubectl replace --force one MUST make sure that the value of the --cascade flag is set to true. This is NOT the default value for this command, and MUST be explicitly set. Running kubectl replace --force without --cascade=true against an AerospikeCluster resource will cause existing dependent resources (pods, services, etc…​) to be left untouched (i.e. orphaned), requiring manual cleanup by an operator to be deleted from the Kubernetes cluster.

Defining node selector for an Aerospike cluster

One may need to make sure that pods in an Aerospike cluster are scheduled onto specific Kubernetes nodes. For addressing that, node selectors are available.

In order to use node selectors, one needs to label at least one of the Kubernetes nodes in one’s cluster:

$ kubectl label nodes <node-name> app=aerospike
node "<node-name>" labeled

In order to define a node selector for an Aerospike cluster, one sets AerospikeCluster.spec.nodeSelector property to match the node label set before:

$ kubectl create -f - <<EOF
apiVersion: aerospike.travelaudience.com/v1alpha2
kind: AerospikeCluster
metadata:
  name: as-cluster-0
spec:
  version: "4.2.0.10"
  nodeCount: 1
  nodeSelector:
    app: aerospike
  namespaces:
  - name: as-namespace-0
    replicationFactor: 1
    memorySize: 1G
    defaultTTL: 0s
    storage:
      type: file
      size: 1G
EOF
aerospikecluster.aerospike.travelaudience.com "as-cluster-0" created

Defining tolerations for an Aerospike cluster

In order to define tolerations for an Aerospike cluster, one sets AerospikeCluster.spec.tolerations property:

$ kubectl create -f - <<EOF
apiVersion: aerospike.travelaudience.com/v1alpha2
kind: AerospikeCluster
metadata:
  name: as-cluster-0
spec:
  version: "4.2.0.10"
  nodeCount: 2
  tolerations:
  - key: "node.kubernetes.io/memory-pressure"
    operator: "Exists"
    effect: "NoSchedule"
  namespaces:
  - name: as-namespace-0
    replicationFactor: 2
    memorySize: 1G
    defaultTTL: 0s
    storage:
      type: file
      size: 1G
EOF
aerospikecluster.aerospike.travelaudience.com "as-cluster-0" created
1. As mentioned in https://www.aerospike.com/docs/operations/plan/capacity#total-storage-required-for-cluster
2. The Kubernetes namespace, if different from default, must be created before creating the AerospikeCluster resource.
3. Pods created by aerospike-operator are based on the official aerospike/aerospike-server:<tag> image
4. https://github.com/alicebob/asprom.
5. As described in https://discuss.aerospike.com/t/general-questions-on-rolling-restart/5130.
6. As described in https://www.aerospike.com/docs/operations/manage/aerospike/cold_start.