rcs-ocm-deployer
is an operator designed to deploy Capp
(ContainerApp) workloads created on the Hub Cluster on the most suitable Managed Cluster using OCM
(Open Cluster Management) APIs of Placement
and ManifestWork
. It also includes a OCM AddOn
to sync the status of Capp
between the Managed Clusters and the Hub Cluster.
RCS
(Run Container Service) is an open-source implementation of Container as a Service solution.
It utilizes the OCM
project to create and manage workloads across multiple clusters, providing cloud-like features and technologies to users running on-premise.
It offers an auto cluster scheduler based on cluster usage and availability, requires basic configuration, and provides an auto-scaler template based on a single metric, among other features.
RCS
aims to simplify and streamline the management of containerized applications, making it easier for developers to focus on writing code.
This operator works together with the container-app-operator
. For more information about Capp
and about its API, please check the Capp repo.
This project uses the Placement
and ManifestWork
APIs of the Open Cluster Management (OCM) project. For more information please refer to the OCM documentation.
-
The client creates the
Capp
CR in a Namespace on the Hub Cluster. -
The
rcs-ocm-deployer
controller on Hub Cluster watches forCapp
CRs and reconciles them. -
The controller chooses the Managed Cluster with the highest score to deploy the
Capp
workload using thePlacement
API. The score is computed by thercs-score
addon which creates anAddOnPlacementScore
. -
Each Managed Cluster on the Hub Cluster has a dedicated namespace. The controller creates a
ManifestWork
object on the Hub Cluster in the namespace of the chosen Managed Cluster. -
The Managed Cluster pulls the
ManifestWork
from the Hub Cluster using thework agent
and creates aCapp
CR on the Managed Cluster. -
A
Capp
controller runs on Managed Cluster, watches forCapp
CRs and reconciles them. -
A status is returned from the Managed Cluster to the Hub Cluster using an
OCM AddOn
.
-
placement
: The controller adds an annotation containing the chosen Managed Cluster to deploy theCapp
workload on, in accordance to theplacementDecision
and the desiredSite
. -
sync
: The controller controls the lifecycle of theManifestWork
CR in the namespace of the chosen Managed Cluster. TheManifestWork
contains theCapp
CR as well as all theSecrets
andVolumes
referenced in theCapp
CR, thus making sure that all theSecrets
andVolumes
also exist on the Managed Cluster, in the same namespace theCapp CR
exists in on the Hub Cluster.
The following should be installed on your Linux machine:
Simply run the following to get a Hub cluster ready with all the prereq needed to have rcs-ocm-deployer
deployed on it, and 2 Managed Cluster with container-app-operator
already installed on them, with all add-ons installed as well.
$ make local-quickstart CAPP_RELEASE=<release> ADDON_RELEASE=<release>
CAPP_RELEASE
and ADDON_RELEASE
are optional parameters. If unspecified, the script will install conatiner-app-operator
and rcs-ocm-addons
using the latest development version.
To just deploy the rcs-ocm-deployer
on a cluster which already includes all the needed prereq, use the Helm chart defined at chars/rcs-ocm-deployer
:
$ helm upgrade --install rcs-deployer --namespace rcs-deployer-system --create-namespace oci://ghcr.io/dana-team/helm-charts/rcs-ocm-deployer --version <release>
The OCM project contains a script that allows spinning up a KinD
-based environment, containing of a Hub Cluster and 2 Managed Clusters.
You can clone the repository and set up the environment. This script would create 3 clusters with the following Kubernetes contexts:
kind-hub
kind-cluster1
kind-cluster2
To switch between contexts, use:
$ kubectl config use-context <context-name>
Run the script:
$ git clone https://github.com/open-cluster-management-io/OCM
$ bash ./OCM/solutions/setup-dev-environment/local-up.sh
ManagedClusterSet
is an OCM cluster-scoped API in the Hub Cluster for grouping a few managed clusters into a "set".
To create a ManagedClusterSet
, run the following command on the Hub Cluster:
$ clusteradm create clusterset <clusterSet-name>
To add a Managed Cluster to the ManagedClusterSet
, run the following command on the Hub Cluster:
$ clusteradm clusterset set <clusterSet-name> --clusters <cluster-name>
To bind the cluster set to a namespace, run the following command on the Hub Cluster:
$ clusteradm clusterset bind <clusterSet-name> --namespace <clusterSet-namespace>
For the controller to work, it is needed to create a Placement
CR. This Placement
selects all clusters and is used to install the addons
on all the Managed Clusters:
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
name: all-clusters
namespace: <clusterSet-namespace>
spec:
clusterSets:
- <clusterSet-name>
You can also create a Placement such that the clusters are selected using the customized scores. This uses the AddonPlacementScore
which is deployed later:
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
name: <placement-name>
namespace: <clusterSet-namespace>
spec:
numberOfClusters: 1
clusterSets:
- <clusterSet-name>
prioritizerPolicy:
mode: Exact
configurations:
- scoreCoordinate:
type: AddOn
addOn:
resourceName: rcs-score
scoreName: cpuAvailable
weight: 1
The Capp
CRD needs to be installed on the Hub Cluster:
$ git clone https://github.com/dana-team/container-app-operator
$ cd container-app-operator
$ make install
To use rcs-ocm-deployer
, you need to have cert-manager
installed on your cluster. Follow the instruction here.
$ make deploy IMG=ghcr.io/dana-team/rcs-ocm-deployer:<release>
The rcs-ocm-deployer
operator utilizes the RCSConfig
CRD to manage its configuration and deployment options.
An instance of the RCSConfig
CRD named rcs-config
should exist in the rcs-deployer-system
namespace. This CRD instance contains the necessary configuration for the operator.
apiVersion: rcs.dana.io/v1alpha1
kind: RCSConfig
metadata:
name: rcs-config
namespace: rcs-deployer-system
spec:
placements:
- placement-1st
- placement-2nd
# Add more placement names as needed
placementsNamespace: your-placements-namespace
Ensure that the spec section includes a list of placements
and specifies the placementsNamespace
as required for your setup.
The addons
are managed in a separate repository, called rcs-ocm-addons
.
Deploy the addons the Hub
cluster:
$ make deploy-addons ADDON_RELEASE=<release>
Patch the ClusterManagementAddon
CR of the status addon, called capp-status
to add the created Placement
to the add-on, so that it's deployed on all clusters.
$ kubectl patch clustermanagementaddon capp-status-addon --type merge -p '{"spec":{"installStrategy":{"type":"Placements","placements":[{"name":"all-clusters","namespace":"<clusterSet-namespace>"}]}}}'
The controller will automatically install the add-on agent
on all Managed/Spoke Clusters. Validate the add-on agent is installed on a Managed/Spoke` cluster:
$ kubectl -n open-cluster-management-agent-addon get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
capp-status-addon-agent 1/1 1 1 2m24s
You can also validate and check the status of the add-on on the Hub cluster:
$ kubectl -n <managed-cluster> get managedclusteraddon
NAME AVAILABLE DEGRADED PROGRESSING
capp-status-addon True
Deploy the AddOnDeploymentConfig
which uses customizedVariables
to pass the max
value and min
value of the customized scores as environment variables to the add-on deployment on the Managed Clusters.
apiVersion: addon.open-cluster-management.io/v1alpha1
kind: AddOnDeploymentConfig
metadata:
name: rcs-score-deploy-config
namespace: open-cluster-management-hub
spec:
agentInstallNamespace: open-cluster-management-agent-addon
customizedVariables:
- name: MAX_CPU_COUNT
value: "<MAX_CPU_COUNT>"
- name: MIN_CPU_COUNT
value: "<MIN_CPU_COUNT>>"
- name: MAX_MEMORY_BYTES
value: "<MAX_MEMORY_BYTES>"
- name: MIN_MEMORY_BYTES
value: "<MIN_MEMEORY_BYTES>"
Patch the ClusterManagementAddon
CR of the score, called rcs-score
to add the created Placement
and AddonDeploymentConfigs
to the add-on, so that it's deployed on all clusters in the Placement
with the default configuration.
kubectl patch clustermanagementaddon rcs-score --type merge -p \
'{"spec":{"installStrategy":{"type":"Placements","placements":[{"name":"<placement-name>","namespace":"<clusterSet-namespace>","configs":[{"group":"addon.open-cluster-management.io","resource":"addondeploymentconfigs","name":"rcs-score-deploy-config","namespace":"open-cluster-management-hub"}]}]}}}'
apiVersion: rcs.dana.io/v1alpha1
kind: Capp
metadata:
name: capp-sample
namespace: capp-sample
spec:
configurationSpec:
template:
spec:
containers:
- env:
- name: APP_NAME
value: capp-env-var
image: 'quay.io/danateamorg/example-python-app:v1-flask'
name: capp-sample
volumeMounts:
- name: test-nfspvc
mountPath: /data
routeSpec:
hostname: capp.dev
tlsEnabled: true
tlsSecret: cappTlsSecretName
volumesSpec:
nfsVolumes:
- server: test
path: /test
name: test-nfspvc
capacity:
storage: 200Gi
logSpec:
type: elastic
host: 10.11.12.13
index: main
user: elastic
passwordSecret: es-elastic-user
scaleMetric: concurrency
state: enabled