KAS Installer allows the deployment and configuration of Managed Kafka Service in a single K8s cluster.
- Prerequisites
- Description
- Usage
- Installation Modes
- Fleet Manager Parameter Customization
- SSO Providers
- Custom Components
- Using rhoas CLI
- Legacy Scripts
- Running E2E Test Suite (experimental)
- jq
- curl
- gsed for macOS e.g. via
brew install gsed
- OpenShift. In the future there are plans to make it compatible
with native K8s. Currently an OpenShift dedicated based environment is needed
(Currently needs to be a multi-zone cluster if you want to create a Kafka
instance through the fleet manager by using
managed_kafka.sh
). - git
- opm required to build custom kas-fleetshard OLM bundle from source (see Custom Components)
- oc
- kubectl
- openssl CLI tool
- rhoas CLI (https://github.com/redhat-developer/app-services-cli)
- A user with administrative privileges in the OpenShift cluster and is logged in using
oc
orkubectl
- brew coreutils (Mac only)
- OSD Cluster with the following specs. Clusters with fewer/smaller compute nodes may work, but have not been verified with kas-installer.
- Plan
developer.x1
- 6 compute nodes
- Size: m5.2xlarge
- MultiAz: N/A
- Plan
standard.x1
- 9 compute nodes (3 per zone)
- Size: m5.2xlarge
- MultiAz: True
- Plan
standard.x2
- 12 compute nodes (4 per zone)
- Size: m5.2xlarge
- MultiAz: True
- Plan
KAS Installer deploys and configures the following components that are part of Managed Kafka Service:
- MAS SSO
- KAS Fleet Manager
- Observability Operator (via KAS Fleet Manager)
- sharded-nlb IngressController
- KAS Fleet Shard and Strimzi Operators (via KAS Fleet Manager)
It deploys and configures the components to the cluster set in the user's kubeconfig file.
Additionally, a single Data Plane cluster is configured ready to be used, in the same cluster set in the user's kubeconfig file.
- Create and fill the KAS installer configuration file
kas-installer.env
. Minimally, the values identified as [required] in kas-installer-defaults.env must be configured. - make sure you have run
oc login --server=<api cluster url|https://api.xxx.openshiftapps.com:6443>
to your target OSD cluster. You will be asked a password or a token - Run the KAS installer
kas-installer.sh
to deploy and configure Managed Kafka Service - Run
uninstall.sh
to remove KAS from the cluster. You should remove any deployed Kafkas before runnig this script.
Troubleshooting: If the installer crashed due to configuration error in kas-installer.env
, you often can rerun the installer after fixing the config issue.
It is not necessary to run uninstall before retrying.
Deploying a cluster with kas-fleet-manager in standalone
is the default for kas-installer (when OCM_SERVICE_TOKEN
is not defined).
In this mode, the fleet manager deploys the data plane components from OLM bundles.
NOTE:
In standalone mode, predefined bundles are used for Strimzi and KAS Fleetshard operators. To use a different bundle
you'll need to build a dev bundle and set either STRIMZI_OLM_INDEX_IMAGE
or KAS_FLEETSHARD_OLM_INDEX_IMAGE
environment variables.
Installation with OCM mode allows kas-fleet-manager to deploy the data plane components as OCM addons. This mode can be
used by setting OCM_SERVICE_TOKEN
to your OCM offline token and also setting
OCM_CLUSTER_ID
to the idenfier for the OSD cluster used in deployment.
NOTE:
In OCM mode, it may take up to 10 minutes after the kas-installer.sh
completes before the addon installations are ready. Until ready,
the fleet-manager API will reject new Kafka requests.
The kas-installer.sh
process will check for the presence of an executable named kas-fleet-manager-service-template-params
in
the project root. When available, it will be executed with the expection that key/value pairs will be output to stdout. The output
will be used when processing the kas-fleet-manager's service-template.yml.
The name of the executable is intentially missing an extension to indicate that any language may be used that is known to the user
to be supported in their own environment.
Note that oc process
requires that individual parameters are specified on a single line.
For example, to provide a custom KAS_FLEETSHARD_OPERATOR_SUBSCRIPTION_CONFIG
parameter value to the fleet manager template,
something like the following may be used for a kas-fleet-manager-service-template-params
executable:
#!/bin/bash
# Declare the subscription config using multi-line YAML
MY_CUSTOM_SUBSCRIPTION_CONFIG='---
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "1Gi"
cpu: "500m"
env:
- name: SSO_ENABLED
value: "true"
- name: MANAGEDKAFKA_ADMINSERVER_EDGE_TLS_ENABLED
value: "true"
- name: STANDARD_KAFKA_CONTAINER_CPU
value: 500m
- name: STANDARD_KAFKA_CONTAINER_MEMORY
value: 4Gi
- name: STANDARD_KAFKA_JVM_XMS
value: 1G
- name: STANDARD_ZOOKEEPER_CONTAINER_CPU
value: 500m
'
# Quota override for my organization
MY_CUSTOM_ORG_QUOTA='
- id: <YOUR ORG HERE>
max_allowed_instances: 10
any_user: true
registered_users: []
'
# Serialize to a single line as JSON (subset of YAML) to standard output
echo "KAS_FLEETSHARD_OPERATOR_SUBSCRIPTION_CONFIG='$(echo "${MY_CUSTOM_SUBSCRIPTION_CONFIG}" | yq e -o=json -I=0)'"
# Disable organization quotas (allows deployment of developer instances)
echo "REGISTERED_USERS_PER_ORGANISATION='[]'"
# Custom organization quota (allows deployment of standard instances), disabled/commented here
#echo "REGISTERED_USERS_PER_ORGANISATION='$(echo "${MY_CUSTOM_ORG_QUOTA}" | yq e -o=json -I=0)'"
Similar to the kas-fleet-manager-service-template-params
previously described, the kas-installer.sh
process will check for the presence of an executable named kas-fleet-manager-secrets-template-params
in
the project root. When available, it will be executed with the expection that key/value pairs will be output to stdout. The output
will be used when processing the kas-fleet-manager's secrets-template.yml.
The default kas-fleet-manager configuration enables the deployment of two instance types, standard
and developer
. Permission
to create an instance of a particular type depends on the organization's quota for the user creating the instance.
- Creating a
developer
instance type requires that the user creating the instance does not have an instance quota. Use parameter customization to set theREGISTERED_USERS_PER_ORGANISATION
property to an empty array[]
. - Creating a
standard
instance type requires that the user creating the instance does have an instance quota. Use parameter customization to set theREGISTERED_USERS_PER_ORGANISATION
property to an array containing the user's org. See theMY_CUSTOM_ORG_QUOTA
variable in the sample script for an example.
To create a standard.x2
(or other non-default types) instance type, the plan
must be provided to the Kafka create request. If using the managed_kafka.sh
script, the --plan
argument may be used:
managed_kafka.sh --create mykafka --plan standard.x2
In order to allow clients to deploy Kafka clusters to multiple clusters and/or zones, the following parameters must be customized:
- KUBE_CONFIG in
kas-fleet-manager-secrets-template-params
(standalone only) - SUPPORTED_CLOUD_PROVIDERS in
kas-fleet-manager-service-template-params
- CLUSTER_LIST in
kas-fleet-manager-service-template-params
The following sections describe the contents of these parameters in detail.
The SUPPORTED_CLOUD_PROVIDERS parameter contains a list of supported cloud providers in a yaml format. See deploying-kas-fleet-manager-to-openshift.md for more details.
The CLUSTER_LIST parameter contains a list clusters for kas fleet manager. See data-plane-osd-cluster-options.md for more details.
When running in standalone mode, the KUBE_CONFIG contains cluster connection information for each cluster to which Kafka can be deployed. There should be one entry in KUBE_CONFIG for each entry in CLUSTER_LIST. For example, to generate a KUBE_CONFIG for two clusters:
oc login -u kubeadmin -p <first_cluster_password> <first_cluster_url>
oc config view --minify --raw > /tmp/c1.yaml
oc login -u kubeadmin -p <second_cluster_password> <second_cluster_url>
oc config view --minify --raw > /tmp/c2.yaml
(KUBECONFIG=/tmp/c1.yaml:/tmp/c2.yaml oc config view --raw)
The KUBE_CONFIG value must be base64 encoded. For example, the following may be used to encode the value in the kas-fleet-manager-secrets-template-params
:
echo "KUBE_CONFIG='$(echo "${KUBE_CONFIG}" | yq -o=json -I=0 | ${BASE64} -w0)'"
Configuration of kas-fleet-manager's SSO providers is done by setting the SSO_PROVIDER_TYPE
configuration variable. When not set, the default provider is mas_sso
. To use RH SSO,
the variable may be set to redhat_sso
and additional configuration can be provided for REDHAT_SSO_HOSTNAME
(default sso.stage.redhat.com), REDHAT_SSO_REALM
(default redhat-external
),
REDHAT_SSO_CLIENT_ID
(required), and REDHAT_SSO_CLIENT_SECRET
(required). See the description for each variable in the kas-installer-defaults.env
file for more information.
Custom-built components are supported for kas-fleet-manager and kas-fleetshard.
- kas-fleet-manager: see the documentation for
KAS_FLEET_MANAGER_IMAGE_BUILD
in the kas-installer-defaults.env file. - kas-fleetshard: prior to running
kas-installer.sh
, executeoperators/generate-kas-fleetshard-olm-bundle.sh
, passing the required parameters (seeoperators/generate-kas-fleetshard-olm-bundle.sh --help
for details). The script will build kas-fleetshard from source and an OLM bundle index, optionally updating thekas-installer.env
file with the configuration. Runningkas-installer.sh
will deploy the custom-built operator.
Use ./rhoas_login.sh
as a short cut to login to the CLI. Login using the username you specified as RH_USERNAME in the env file. The password is the same as the RH_USERNAME value.
Alternatively (and likely preferably), use ./login-all.sh
which makes sure you are also logged to your OpenShift Dedicated cluster.
There are a couple of things that are expected not to work when using the RHOAS CLI with a kas-installer installed instance. These are noted below.
- To create an account, run
rhoas service-account create --short-description foo --file-format properties
. - To list existing service accounts, run
rhoas service-account list
. - To remove an existing service account, run
rhoas service-account delete --id=<ID of service account>
.
- To create a cluster, run
rhoas kafka create --bypass-checks --provider aws --region us-east-1 --name <clustername>
. Note that--bypass-checks
is required as the T&Cs endpoint will not exist in your environment. The provider and region must be passed on the command line. - To list existing clusters, run
rhoas kafka list
- To remove an existing cluster, run
rhoas kafka delete --name <clustername>
.
To use these cli featurs, you must set MANAGEDKAFKA_ADMINSERVER_EDGE_TLS_ENABLED=true
in your kas-installer.env
so that the admin-server will run over TLS (edge terminated).
- To create a topic
rhoas kafka topic create --name=foo
- To grant access
rhoas kafka acl grant-access --topic=foo --all-accounts --producer
etc.
Please favour using the rhoas command line. These scripts will be remove at some point soon.
The service_account.sh
script supports creating, listing, and deleting service accounts.
- To create an account, run
service_account.sh --create
. The new service account information will be printed to the console. Be sure to retain theclientID
andclientSecret
values to use when generating an access token or for connecting to Kafka directly. - To list existing service accounts, run
service_account.sh --list
. - To remove an existing service account, run
service_account.sh --delete <ID of service account>
.
- Run
get_access_token.sh
using theclientID
andclientSecret
as the first and second arguments. The generated access token and its expiration date and time will be printed to the console.
The managed_kafka.sh
script supports creating, listing, and deleting Kafka clusters.
- To create a cluster, run
managed_kafka.sh --create <cluster name>
. Progress will be printed as the cluster is prepared and provisioned. - To list existing clusters, run
managed_kafka.sh --list
. - To remove an existing cluster, run
managed_kafka.sh --delete <cluster ID>
. - To patch an existing cluster (for instance changing a strimzi version), run
managed_kafka.sh --admin --patch <cluster ID> '{ "strimzi_version": "strimzi-cluster-operator.v0.23.0-3" }'
- To use kafka bin scripts against pre existing kafka cluster, run
managed_kafka.sh --certgen <kafka id> <Service_Account_ID> <Service_Account_Secret>
. If you do not pass the <Service_Account_ID> <Service_Account_Secret> arguments, the script will attempt to create a Service_Account for you. The cert generation is already performed at the end of--create
. Point the--command-config flag
to the generated app-services.properties in the working directory.
- If there is already 2 service accounts pre-existing you must delete 1 of them for this script to work
To use the Kafka Cluster that is created with the managed_kafka.sh
script with command line tools like kafka-topics.sh
or kafka-console-consumer.sh
do the following.
-
Generate the certificate and
app-services.properties
file, runmanaged_kafka.sh --certgen <instance-id>
whereinstance-id
can found by runningmanaged_kafka.sh --list
and also bootstrap host to the cluster in same response. -
Run the following to give the current user the permissions to create a topic and group. For the
<service-acct>
for below script take the service account from generatedapp-services.properties
filecurl -vs -H"Authorization: Bearer $(./get_access_token.sh --owner)" http://admin-server-$(./managed_kafka.sh --list | jq -r .items[0].bootstrap_server_host | awk -F: '{print $1}')/api/v1/acls -XPOST -H'Content-type: application/json' --data '{"resourceType":"GROUP", "resourceName":"*", "patternType":"LITERAL", "principal":"User:<service-acct>", "operation":"ALL", "permission":"ALLOW"}'
then for Topic
curl -vs -H"Authorization: Bearer $(./get_access_token.sh --owner)" http://admin-server-$(./managed_kafka.sh --list | jq -r .items[0].bootstrap_server_host | awk -F: '{print $1}')/api/v1/acls -XPOST -H'Content-type: application/json' --data '{"resourceType":"TOPIC", "resourceName":"*", "patternType":"LITERAL", "principal":"User:<service-acct>", "operation":"ALL", "permission":"ALLOW"}'
-
Then execute the your tool like
kafka-topics.sh --bootstrap-server <bootstrap-host>:443 --command-config app-services.properties --topic foo --create --partitions 9
-
if you created separate service account using above instructions, edit the
app-services.properties
file and update the username and password withclientID
andclientSecret
-
Install all cluster components using
kas-installer.sh
-
Clone the e2e-test-suite repository locally and change directory to the test suite project root
-
Generate the test suite configuration with
${KAS_INSTALLER_DIR}/e2e-test-config.sh > config.json
. When usingkas-installer.sh
with theredhat_sso
provider type, the user accounts to be used by the E2E tests must be preconfigured. Thee2e-test-config.sh
script will set all variables named likeE2E_USER_
in the test suite'sconfig.json
file. TheE2E_USER_
prefix is not included in the JSON configuration.For example, an environment variable
E2E_USER_PRIMARY_USERNAME='my-primary-user'
will be added toconfig.json
as key/value pair"PRIMARY_USERNAME": "my-primary-user"
. -
Execute individual test classes:
./hack/testrunner.sh test KafkaAdminPermissionTest
./hack/testrunner.sh test KafkaInstanceAPITest
./hack/testrunner.sh test KafkaCLITest