diff --git a/config.toml b/config.toml index db893f5ae..435387733 100644 --- a/config.toml +++ b/config.toml @@ -27,7 +27,7 @@ alpine_js_version = "2.2.1" favicon = "favicon.png" [params.versions] -docs = ["2.1", "2.0", "1.5", "1.4"] +docs = ["2.2", "2.1", "2.0", "1.5", "1.4"] # Site fonts. For more options see https://fonts.google.com. [[params.fonts]] diff --git a/content/blog/migrating-to-github-container-registry.md b/content/blog/migrating-to-github-container-registry.md new file mode 100644 index 000000000..2fc427798 --- /dev/null +++ b/content/blog/migrating-to-github-container-registry.md @@ -0,0 +1,71 @@ ++++ +title = "Migrating our container images to GitHub Container Registry" +date = 2021-03-26 +author = "KEDA Maintainers" ++++ + +We provide **various ways to [deploy KEDA](https://keda.sh/docs/latest/deploy/) in your cluster** including by using [Helm chart](https://github.com/kedacore/charts), [Operator Hub](https://operatorhub.io/operator/keda) and raw YAML specifications. + +These deployment options all rely on the container images that we provide which are available on **[Docker Hub](https://hub.docker.com/u/kedacore), the industry standard for public container images**. + +However, we have found that Docker Hub is no longer the best place for our container images and are migrating to GitHub Container Registry (Preview). + +## Why are making this change? + +### Docker Hub is introducing rate limiting and image retention + +Over the past couple of years, Docker Hub has become the industry standard for hosting public container images. This has become a big burden for Docker to manage all the traffic and decided in 2020 to make some changes: + +- Anonymous image pulls are being rate limited +- Unused images will no longer be retained + +Because we want to ensure that our end-users can use KEDA without any issues, we want to make them available to anyone without any limitations. + +Learn more about these changes in [Docker's FAQ](https://www.docker.com/pricing/resource-consumption-updates) and our issue on [GitHub](https://github.com/kedacore/keda/issues/995). + +### Gaining insights on KEDA adoption + +As maintainers, **we find it hard to measure the adoption of KEDA** to understand how many end-users are using older versions of KEDA and what the growth is over time. + +Docker Hub provides a vague total pull count per container image, but it does not give in-depth details concerning the tags and what the pull growth is over time. + +In GitHub Container Registry, however, **metrics are provided out-of-the-box on a per-tag basis** allowing us to better understand what our customers are using and make better decisions when we no longer support a given version. + +### Bringing our artifacts closer to home + +Lastly, we want to **bring our artifacts closer to our home on GitHub**. By using more of the GitHub ecosystem, we believe that this integration will only improve and get tighter integration with our releases and such. + +## What is changing? + +Our container images are being published on [GitHub Container Registry](https://github.com/orgs/kedacore/packages?type=source) for end-users to pull them. + +Because of this, the names of our container images are changing: + +| Component | New Image (GitHub Container Registry) | Legacy Image (Docker Hub) | +| :------------- | :---------------------------------------- | --------------------------------- | +| Metrics Server | `ghcr.io/kedacore/keda-metrics-apiserver` | `kedacore/keda-metrics-apiserver` | +| Operator | `ghcr.io/kedacore/keda` | `kedacore/keda` | + +## When is this taking place? + +As of v2.2, we have started publishing our new container images to GitHub Container Registry in parallel to Docker Hub. + +This allows customers to already migrate to our new registry and consume our artifacts there. + +**Once GitHub Container Registry becomes generally available (GA), we will no longer publish new versions to Docker Hub.** + +## What is the impact for end-users? + +**If you are using one of our deployment options, end-users are not be impacted.** + +Since v2.2, we are using GitHub Container Registry by default and you are good to go. + +If you are using your own deployment mechanism, then you will have to pull the container images from GitHub Container Registry instead. + +## Join the conversation + +Do you have questions or remarks? Feel free to join the conversation on [GitHub Discussions](https://github.com/kedacore/keda/discussions/1700). + +Thanks for reading, and happy scaling! + +KEDA Maintainers. \ No newline at end of file diff --git a/content/docs/1.4/deploy.md b/content/docs/1.4/deploy.md index abe33fd3d..29610741f 100644 --- a/content/docs/1.4/deploy.md +++ b/content/docs/1.4/deploy.md @@ -94,7 +94,7 @@ kubectl apply -f ./ You can also find the same YAML declarations in our `/deploy` directory on our [GitHub repo](https://github.com/kedacore/keda) if you prefer to clone it. ```sh -git clone https://github.com/kedacore/keda && cd keda +git clone https://github.com/kedacore/keda && cd keda && git checkout tags/v1.4.1 kubectl apply -f deploy/crds/keda.k8s.io_scaledobjects_crd.yaml kubectl apply -f deploy/crds/keda.k8s.io_triggerauthentications_crd.yaml diff --git a/content/docs/1.5/deploy.md b/content/docs/1.5/deploy.md index bb6450d8e..cb379f1c1 100644 --- a/content/docs/1.5/deploy.md +++ b/content/docs/1.5/deploy.md @@ -94,7 +94,7 @@ kubectl apply -f ./ You can also find the same YAML declarations in our `/deploy` directory on our [GitHub repo](https://github.com/kedacore/keda) if you prefer to clone it. ```sh -git clone https://github.com/kedacore/keda && cd keda +git clone https://github.com/kedacore/keda && cd keda && git checkout tags/v1.5.0 kubectl apply -f deploy/crds/keda.k8s.io_scaledobjects_crd.yaml kubectl apply -f deploy/crds/keda.k8s.io_triggerauthentications_crd.yaml diff --git a/content/docs/2.0/operate/cluster.md b/content/docs/2.0/operate/cluster.md index 095bf32c3..389a1c274 100644 --- a/content/docs/2.0/operate/cluster.md +++ b/content/docs/2.0/operate/cluster.md @@ -26,4 +26,15 @@ Here is an overview of all KEDA deployments and the supported replicas: | Deployment | Support Replicas | Reasoning | |----------------|-------------------------|-------------------------------| | Operator | 1 | | -| Metrics Server | 1 | Limitation in [k8s custom metrics server](https://github.com/kubernetes-sigs/custom-metrics-apiserver/issues/70) | \ No newline at end of file +| Metrics Server | 1 | Limitation in [k8s custom metrics server](https://github.com/kubernetes-sigs/custom-metrics-apiserver/issues/70) | + +## Firewall requirements + +KEDA requires to be accessible inside the cluster to be able to autoscale. + +Here is an overview of the required ports that need to be accessible for KEDA to work: + +| Port | Why? | Remarks | +| ------ | -------------------------------------------- | ---------------------------------------------------- | +| `443` | Used by Kubernetes API server to get metrics | Required for all platforms, except for Google Cloud. | +| `6443` | Used by Kubernetes API server to get metrics | Only required for Google Cloud | \ No newline at end of file diff --git a/content/docs/2.1/deploy.md b/content/docs/2.1/deploy.md index 9777b014d..a83a0a326 100644 --- a/content/docs/2.1/deploy.md +++ b/content/docs/2.1/deploy.md @@ -37,7 +37,7 @@ Deploying KEDA with Helm is very simple: ```sh kubectl create namespace keda - helm install keda kedacore/keda --namespace keda + helm install keda kedacore/keda --version 2.1.0 --namespace keda ``` ### Uninstall @@ -48,10 +48,10 @@ If you want to remove KEDA from a cluster you can run one of the following: ```sh helm uninstall -n keda keda -kubectl delete -f https://raw.githubusercontent.com/kedacore/keda/main/config/crd/bases/keda.sh_scaledobjects.yaml -kubectl delete -f https://raw.githubusercontent.com/kedacore/keda/main/config/crd/bases/keda.sh_scaledjobs.yaml -kubectl delete -f https://raw.githubusercontent.com/kedacore/keda/main/config/crd/bases/keda.sh_triggerauthentications.yaml -kubectl delete -f https://raw.githubusercontent.com/kedacore/keda/main/config/crd/bases/keda.sh_clustertriggerauthentications.yaml +kubectl delete -f https://raw.githubusercontent.com/kedacore/keda/v2.1.0/config/crd/bases/keda.sh_scaledobjects.yaml +kubectl delete -f https://raw.githubusercontent.com/kedacore/keda/v2.1.0/config/crd/bases/keda.sh_scaledjobs.yaml +kubectl delete -f https://raw.githubusercontent.com/kedacore/keda/v2.1.0/config/crd/bases/keda.sh_triggerauthentications.yaml +kubectl delete -f https://raw.githubusercontent.com/kedacore/keda/v2.1.0/config/crd/bases/keda.sh_clustertriggerauthentications.yaml ``` ## Deploying with Operator Hub {#operatorhub} diff --git a/content/docs/2.1/operate/cluster.md b/content/docs/2.1/operate/cluster.md index 928baaf83..851125b2b 100644 --- a/content/docs/2.1/operate/cluster.md +++ b/content/docs/2.1/operate/cluster.md @@ -4,17 +4,6 @@ description = "Guidance & requirements for running KEDA in your cluster" weight = 100 +++ -## High Availability - -KEDA does not provide support for high-availability due to upstream limitations. - -Here is an overview of all KEDA deployments and the supported replicas: - -| Deployment | Support Replicas | Reasoning | -|----------------|-------------------------|-------------------------------| -| Operator | 1 | | -| Metrics Server | 1 | Limitation in [k8s custom metrics server](https://github.com/kubernetes-sigs/custom-metrics-apiserver/issues/70) | - ## Cluster capacity requirements The KEDA runtime require the following resources in a production-ready setup: @@ -28,6 +17,28 @@ These are used by default when deploying through YAML. > πŸ’‘ For more info on CPU and Memory resource units and their meaning, see [this](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes) link. +## Firewall requirements + +KEDA requires to be accessible inside the cluster to be able to autoscale. + +Here is an overview of the required ports that need to be accessible for KEDA to work: + +| Port | Why? | Remarks | +| ------ | -------------------------------------------- | ---------------------------------------------------- | +| `443` | Used by Kubernetes API server to get metrics | Required for all platforms, except for Google Cloud. | +| `6443` | Used by Kubernetes API server to get metrics | Only required for Google Cloud | + +## High Availability + +KEDA does not provide support for high-availability due to upstream limitations. + +Here is an overview of all KEDA deployments and the supported replicas: + +| Deployment | Support Replicas | Reasoning | +|----------------|-------------------------|-------------------------------| +| Operator | 1 | | +| Metrics Server | 1 | Limitation in [k8s custom metrics server](https://github.com/kubernetes-sigs/custom-metrics-apiserver/issues/70) | + ## HTTP Timeouts Some scalers issue HTTP requests to external servers (i.e. cloud services). Each applicable scaler uses its own dedicated HTTP client with its own connection pool, and by default each client is set to time out any HTTP request after 3 seconds. diff --git a/content/docs/2.2/concepts/authentication.md b/content/docs/2.2/concepts/authentication.md index 5a8e8462a..690adbe25 100644 --- a/content/docs/2.2/concepts/authentication.md +++ b/content/docs/2.2/concepts/authentication.md @@ -19,7 +19,7 @@ Some metadata parameters will not allow resolving from a literal value, and will ### Example -If using the [RabbitMQ scaler](https://keda.sh/docs/2.1/scalers/rabbitmq-queue/), the `host` parameter may include passwords so is required to be a reference. You can create a secret with the value of the `host` string, reference that secret in the deployment, and map it to the `ScaledObject` metadata parameter like below: +If using the [RabbitMQ scaler](https://keda.sh/docs/2.2/scalers/rabbitmq-queue/), the `host` parameter may include passwords so is required to be a reference. You can create a secret with the value of the `host` string, reference that secret in the deployment, and map it to the `ScaledObject` metadata parameter like below: ```yaml apiVersion: v1 diff --git a/content/docs/2.2/deploy.md b/content/docs/2.2/deploy.md index 8747afb20..4658fbf1c 100644 --- a/content/docs/2.2/deploy.md +++ b/content/docs/2.2/deploy.md @@ -72,15 +72,15 @@ Locate installed KEDA Operator in `keda` namespace, then remove created `KedaCon If you want to try KEDA on [Minikube](https://minikube.sigs.k8s.io) or a different Kubernetes deployment without using Helm you can still deploy it with `kubectl`. - We provide sample YAML declaration which includes our CRDs and all other resources in a file which is available on the [GitHub releases](https://github.com/kedacore/keda/releases) page. -Run the following command (if needed, replace the version, in this case `2.0.0`, with the one you are using): +Run the following command (if needed, replace the version, in this case `2.2.0`, with the one you are using): ```sh -kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.0.0/keda-2.0.0.yaml +kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.2.0/keda-2.2.0.yaml ``` - Alternatively you can download the file and deploy it from the local path: ```sh -kubectl apply -f keda-2.0.0.yaml +kubectl apply -f keda-2.2.0.yaml ``` - You can also find the same YAML declarations in our `/config` directory on our [GitHub repo](https://github.com/kedacore/keda) if you prefer to clone it. @@ -88,27 +88,27 @@ kubectl apply -f keda-2.0.0.yaml ```sh git clone https://github.com/kedacore/keda && cd keda -VERSION=2.0.0 make deploy +VERSION=2.2.0 make deploy ``` ### Uninstall -- In case of installing from released YAML file just run the following command (if needed, replace the version, in this case `2.0.0`, with the one you are using): +- In case of installing from released YAML file just run the following command (if needed, replace the version, in this case `2.2.0`, with the one you are using): ```sh -kubectl delete -f https://github.com/kedacore/keda/releases/download/v2.0.0/keda-2.0.0.yaml +kubectl delete -f https://github.com/kedacore/keda/releases/download/v2.2.0/keda-2.2.0.yaml ``` - If you have downloaded the file locally, you can run: ```sh -kubectl delete -f keda-2.0.0.yaml +kubectl delete -f keda-2.2.0.yaml ``` - You would need to run these commands from within the directory of the cloned [GitHub repo](https://github.com/kedacore/keda): ```sh -VERSION=2.0.0 make undeploy +VERSION=2.2.0 make undeploy ``` ## Deploying KEDA on MicroK8s {#microk8s} diff --git a/content/docs/2.2/operate/cluster.md b/content/docs/2.2/operate/cluster.md index b5ef5fc5a..b8f356436 100644 --- a/content/docs/2.2/operate/cluster.md +++ b/content/docs/2.2/operate/cluster.md @@ -4,40 +4,42 @@ description = "Guidance & requirements for running KEDA in your cluster" weight = 100 +++ -## High Availability - -KEDA does not provide support for high-availability due to upstream limitations. - -Here is an overview of all KEDA deployments and the supported replicas: - -| Deployment | Support Replicas | Reasoning | -|----------------|-------------------------|-------------------------------| -| Operator | 1 | | -| Metrics Server | 1 | Limitation in [k8s custom metrics server](https://github.com/kubernetes-sigs/custom-metrics-apiserver/issues/70) | - ## Cluster capacity requirements The KEDA runtime require the following resources in a production-ready setup: | Deployment | CPU | Memory | -|----------------|-------------------------|-------------------------------| -| Operator | Limit: 1, Request: 100m | Limit: 1000Mi, Request: 100Mi | +| -------------- | ----------------------- | ----------------------------- | | Metrics Server | Limit: 1, Request: 100m | Limit: 1000Mi, Request: 100Mi | +| Operator | Limit: 1, Request: 100m | Limit: 1000Mi, Request: 100Mi | These are used by default when deploying through YAML. > πŸ’‘ For more info on CPU and Memory resource units and their meaning, see [this](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes) link. +## Firewall requirements + +KEDA requires to be accessible inside the cluster to be able to autoscale. + +Here is an overview of the required ports that need to be accessible for KEDA to work: + + +| Port | Why? | Remarks | +| ------ | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `443` | Used by Kubernetes API server to get metrics | Required for all platforms because it uses Control Plane → port 443 on the Service IP range communication.

This is not applicable for Google Cloud. | +| `6443` | Used by Kubernetes API server to get metrics | Only required for Google Cloud because it uses Control Plane → port 6443 on the Pod IP range for communication | + + ## High Availability KEDA does not provide support for high-availability due to upstream limitations. Here is an overview of all KEDA deployments and the supported replicas: -| Deployment | Support Replicas | Reasoning | -|----------------|-------------------------|-------------------------------| -| Operator | 1 | | -| Metrics Server | 1 | Limitation in [k8s custom metrics server](https://github.com/kubernetes-sigs/custom-metrics-apiserver/issues/70) | +| Deployment | Support Replicas | Reasoning | +| -------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------- | +| Metrics Server | 1 | Limitation in [k8s custom metrics server](https://github.com/kubernetes-sigs/custom-metrics-apiserver/issues/70) | +| Operator | 1 | | ## HTTP Timeouts diff --git a/content/docs/2.2/operate/events.md b/content/docs/2.2/operate/events.md index e99afcb64..adb16b9e4 100644 --- a/content/docs/2.2/operate/events.md +++ b/content/docs/2.2/operate/events.md @@ -7,6 +7,7 @@ weight = 100 ## Kubernetes Events emitted by KEDA KEDA emits the following [Kubernetes Events](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#event-v1-core): + | Event | Type | Description | |-------------------------------------|-----------|-----------------------------------------------------------------------------------------------------------------------------| | `ScaledObjectReady` | `Normal` | On the first time a ScaledObject is ready, or if the previous ready condition status of the object was `Unknown` or `False` | diff --git a/content/docs/2.2/scalers/aws-sqs.md b/content/docs/2.2/scalers/aws-sqs.md index 9b0412002..b52f77ead 100644 --- a/content/docs/2.2/scalers/aws-sqs.md +++ b/content/docs/2.2/scalers/aws-sqs.md @@ -26,7 +26,10 @@ triggers: **Parameter list:** - `queueURL` - Full URL for the SQS Queue -- `queueLength` - Target value for queue length passed to the scaler. Example: if one pod can handle 10 messages, set the queue length target to 10. If the actual `ApproximateNumberOfMessages` in the SQS Queue is 30, the scaler scales to 3 pods. (default: 5) +- `queueLength` - Target value for queue length passed to the scaler. Example: if one pod can handle 10 messages, set the queue length target to 10. If the actual messages in the SQS Queue is 30, the scaler scales to 3 pods. (default: 5) + +> For the purposes of scaling, "actual messages" is equal to `ApproximateNumberOfMessages` + `ApproximateNumberOfMessagesNotVislble`, since `NotVisible` in SQS terms means the message is still in-flight/processing. + - `awsRegion` - AWS Region for the SQS Queue - `identityOwner` - Receive permissions on the SQS Queue via Pod Identity or from the KEDA operator itself (see below). @@ -34,7 +37,7 @@ triggers: ### Authentication Parameters -> These parameters are relevant only when `identityOwner` is set to `pod`. +> These parameters are relevant only when `identityOwner` is set to `pod`. You can use `TriggerAuthentication` CRD to configure the authenticate by providing either a role ARN or a set of IAM credentials. @@ -63,7 +66,7 @@ metadata: data: AWS_ACCESS_KEY_ID: AWS_SECRET_ACCESS_KEY: ---- +--- apiVersion: keda.sh/v1alpha1 kind: TriggerAuthentication metadata: diff --git a/content/docs/2.2/scalers/metrics-api.md b/content/docs/2.2/scalers/metrics-api.md index f0403524c..e1fd938eb 100644 --- a/content/docs/2.2/scalers/metrics-api.md +++ b/content/docs/2.2/scalers/metrics-api.md @@ -61,6 +61,8 @@ because many applications implement basic auth with a username as apikey and pas - `cert`: Certificate for client authentication. This is a required field. - `key`: Key for client authentication. Optional. This is a required field. +> πŸ’‘ **NOTE:**It's also possible to set the CA certificate regardless of the selected `authMode` (also without any authentication). This might be usefull if you are using an enterprise CA. + ### Example Here is a full example of scaled object definition using Metric API trigger: @@ -103,7 +105,7 @@ The above example expects that the API endpoint will return response similar to ``` Assuming such response, Metrics API trigger will figure out that current metric value is 12. -> πŸ’‘ **NOTE:**The value of the metric must be json number type. The value is casted to **integer**. +> πŸ’‘ **NOTE:**The value of the metric can either be an integral unquoted json number type (e.g. 123). Or a quantity (e.g. "123", "1.23", "10Mi"). Here is an example of a metric scaler with API Key based authentication, diff --git a/content/docs/2.2/scalers/rabbitmq-queue.md b/content/docs/2.2/scalers/rabbitmq-queue.md index e4b09149b..92ce1b554 100644 --- a/content/docs/2.2/scalers/rabbitmq-queue.md +++ b/content/docs/2.2/scalers/rabbitmq-queue.md @@ -17,7 +17,8 @@ triggers: metadata: host: amqp://localhost:5672/vhost # Optional. If not specified, it must be done by using TriggerAuthentication. protocol: auto # Optional. Specifies protocol to use, either amqp or http, or auto to autodetect based on the `host` value. Default value is auto. - queueLength: '20' # Optional. Queue length target for HPA. Default: 20 messages + mode: QueueLength # QueueLength or MessageRate + value: 100 # message backlog or publish/sec. target per instance queueName: testqueue vhostName: / # Optional. If not specified, use the vhost in the `host` connection string. # Alternatively, you can use existing environment variables to read configuration from: @@ -27,14 +28,15 @@ triggers: **Parameter list:** -- `host`: Host of RabbitMQ with format `amqp://:/vhost`. The resolved host should follow a format like `amqp://guest:password@localhost:5672/vhost` or +- `host`: Host of RabbitMQ with format `://:/vhost`. The resolved host should follow a format like `amqp://guest:password@localhost:5672/vhost` or `http://guest:password@localhost:15672/vhost`. When using a username/password consider using `hostFromEnv` or a TriggerAuthentication. - `queueName`: Name of the queue to read message from. Required. -- `queueLength`: Target value for queue length passed to the scaler. Example: if one pod can handle 10 messages, set the queue length target to 10. If the actual number of messages in the queue is 30, the scaler scales to 3 pods. Default is 20. Optional. +- `mode`: QueueLength to trigger on number of messages in the queue. MessageRate to trigger on the publish rate into the queue. Required. +- `value`: Message backlog or Publish/sec. rate to trigger on - `protocol`: Protocol to be used for communication. Either `auto`, `http`, or `amqp`. It should correspond with the `host` value. Optional, will autodetect based on the `host` URL if possible. - `vhostName`: Vhost to use for the connection, overrides any vhost set in the connection string from `host`/`hostFromEnv`. - +- `queueLength`: DEPRECATED! Use `mode: QueueLength` and `value: ##` instead. Target value for queue length passed to the scaler. Example: if one pod can handle 10 messages, set the queue length target to 10. If the actual number of messages in the queue is 30, the scaler scales to 3 pods. Default is 20 unless `publishRate` is specified, in which case `queueLength` is disabled for this trigger. Some parameters could be provided using environmental variables, instead of setting them directly in metadata. Here is a list of parameters you can use to retrieve values from environment variables: @@ -42,8 +44,12 @@ Some parameters could be provided using environmental variables, instead of sett > πŸ’‘ **Note:** `host`/`hostFromEnv` has an optional vhost name after the host slash which will be used to scope API request. +> πŸ’‘ **Note:** `mode: MessageRate` requires protocol `http`. + > ⚠ **Important:** if you have unacknowledged messages and want to have these counted for the scaling to happen, make sure to utilize the `http` REST API interface which allows for these to be counted. +> ⚠ **Important:** If scaling against both is desired then the `ScaledObject` should have two triggers, one for `mode: QueueLength` and the other for `mode: MessageRate`. HPA will scale based on the largest result considering each of the two triggers independently. + ### Authentication Parameters TriggerAuthentication CRD is used to connect and authenticate to RabbitMQ: @@ -89,12 +95,53 @@ spec: metadata: protocol: amqp queueName: testqueue - queueLength: "20" + mode: QueueLength + value: "20" + authenticationRef: + name: keda-trigger-auth-rabbitmq-conn +``` + +#### HTTP protocol (`QueueLength`): + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: keda-rabbitmq-secret +data: + host: # base64 encoded value of format http://guest:password@localhost:15672/vhost +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-rabbitmq-conn + namespace: default +spec: + secretTargetRef: + - parameter: host + name: keda-rabbitmq-secret + key: host +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: rabbitmq-scaledobject + namespace: default +spec: + scaleTargetRef: + name: rabbitmq-deployment + triggers: + - type: rabbitmq + metadata: + protocol: http + queueName: testqueue + mode: QueueLength + value: "20" authenticationRef: name: keda-trigger-auth-rabbitmq-conn ``` -#### HTTP protocol: +#### HTTP protocol (`MessageRate` and `QueueLength`): ```yaml apiVersion: v1 @@ -128,7 +175,16 @@ spec: metadata: protocol: http queueName: testqueue - queueLength: "20" + mode: QueueLength + value: "20" + authenticationRef: + name: keda-trigger-auth-rabbitmq-conn + - type: rabbitmq + metadata: + protocol: http + queueName: testqueue + mode: MessageRate + value: "100" authenticationRef: name: keda-trigger-auth-rabbitmq-conn ``` diff --git a/content/docs/2.3/_index.md b/content/docs/2.3/_index.md new file mode 100644 index 000000000..a3c18da2f --- /dev/null +++ b/content/docs/2.3/_index.md @@ -0,0 +1,8 @@ ++++ +title = "The KEDA Documentation" +weight = 1 ++++ + +Welcome to the documentation for **KEDA**, the Kubernetes Event-driven Autoscaler. Use the navigation to the left to learn more about how to use KEDA and its components. + +Additions and contributions to these docs are managed on [the keda-docs GitHub repo](https://github.com/kedacore/keda-docs). diff --git a/content/docs/2.3/concepts/_index.md b/content/docs/2.3/concepts/_index.md new file mode 100644 index 000000000..1dd5e9366 --- /dev/null +++ b/content/docs/2.3/concepts/_index.md @@ -0,0 +1,48 @@ ++++ +title = "KEDA Concepts" +description = "What KEDA is and how it works" +weight = 1 ++++ + +## What is KEDA? + +**KEDA** is a [Kubernetes](https://kubernetes.io)-based Event Driven Autoscaler. With KEDA, you can drive the scaling of any container in Kubernetes based on the number of events needing to be processed. + +**KEDA** is a single-purpose and lightweight component that can be added into any Kubernetes cluster. KEDA works alongside standard Kubernetes components like the [Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) and can extend functionality without overwriting or duplication. With KEDA you can explicitly map the apps you want to use event-driven scale, with other apps continuing to function. This makes KEDA a flexible and safe option to run alongside any number of any other Kubernetes applications or frameworks. + +## How KEDA works + +KEDA performs two key roles within Kubernetes: + +1. **Agent** β€” KEDA activates and deactivates Kubernetes [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment) to scale to and from zero on no events. This is one of the primary roles of the `keda-operator` container that runs when you install KEDA. +1. **Metrics** β€” KEDA acts as a [Kubernetes metrics server](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-custom-metrics) that exposes rich event data like queue length or stream lag to the Horizontal Pod Autoscaler to drive scale out. It is up to the Deployment to consume the events directly from the source. This preserves rich event integration and enables gestures like completing or abandoning queue messages to work out of the box. The metric serving is the primary role of the `keda-operator-metrics-apiserver` container that runs when you install KEDA. + +## Architecture + +The diagram below shows how KEDA works in conjunction with the Kubernetes Horizontal Pod Autoscaler, external event sources, and Kubernetes' [etcd](https://etcd.io) data store: + +![KEDA architecture](/img/keda-arch.png) + +### Event sources and scalers + +KEDA has a wide range of [**scalers**](/scalers) that can both detect if a deployment should be activated or deactivated, and feed custom metrics for a specific event source. The following scalers are available: + +{{< scalers-compact >}} + +### Custom Resources (CRD) + +When you install KEDA, it creates four custom resources: + +1. `scaledobjects.keda.sh` +1. `scaledjobs.keda.sh` +1. `triggerauthentications.keda.sh` +1. `clustertriggerauthentications.keda.sh` + +These custom resources enable you to map an event source (and the authentication to that event source) to a Deployment, StatefulSet, Custom Resource or Job for scaling. +- `ScaledObjects` represent the desired mapping between an event source (e.g. Rabbit MQ) and the Kubernetes Deployment, StatefulSet or any Custom Resource that defines `/scale` subresource. +- `ScaledJobs` represent the mapping between event source and Kubernetes Job. +- `ScaledObject`/`ScaledJob` may also reference a `TriggerAuthentication` or `ClusterTriggerAuthentication` which contains the authentication configuration or secrets to monitor the event source. + +## Deploy KEDA + +See the [Deployment](../deploy) documentation for instructions on how to deploy KEDA into any cluster using tools like [Helm](../deploy/#helm). diff --git a/content/docs/2.3/concepts/authentication.md b/content/docs/2.3/concepts/authentication.md new file mode 100644 index 000000000..5a8e8462a --- /dev/null +++ b/content/docs/2.3/concepts/authentication.md @@ -0,0 +1,253 @@ ++++ +title = "Authentication" +weight = 500 ++++ + +Often a scaler will require authentication or secrets and config to check for events. + +KEDA provides a few secure patterns to manage authentication flows: + +* Configure authentication per `ScaledObject` +* Re-use per-namespace credentials or delegate authentication with `TriggerAuthentication` +* Re-use global credentials with `ClusterTriggerAuthentication` + +## Defining secrets and config maps on ScaledObject + +Some metadata parameters will not allow resolving from a literal value, and will instead require a reference to a secret, config map, or environment variable defined on the target container. + +> πŸ’‘ ***TIP:*** *If creating a deployment yaml that references a secret, be sure the secret is created before the deployment that references it, and the scaledObject after both of them to avoid invalid references.* + +### Example + +If using the [RabbitMQ scaler](https://keda.sh/docs/2.1/scalers/rabbitmq-queue/), the `host` parameter may include passwords so is required to be a reference. You can create a secret with the value of the `host` string, reference that secret in the deployment, and map it to the `ScaledObject` metadata parameter like below: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: {secret-name} +data: + {secret-key-name}: YW1xcDovL3VzZXI6UEFTU1dPUkRAcmFiYml0bXEuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbDo1Njcy #base64 encoded per secret spec +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {deployment-name} + namespace: default + labels: + app: {deployment-name} +spec: + selector: + matchLabels: + app: {deployment-name} + template: + metadata: + labels: + app: {deployment-name} + spec: + containers: + - name: {deployment-name} + image: {container-image} + envFrom: + - secretRef: + name: {secret-name} +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: {scaled-object-name} + namespace: default +spec: + scaleTargetRef: + name: {deployment-name} + triggers: + - type: rabbitmq + metadata: + queueName: hello + host: {secret-key-name} + queueLength : '5' +``` + +If you have multiple containers in a deployment, you will need to include the name of the container that has the references in the `ScaledObject`. If you do not include a `envSourceContainerName` it will default to the first container. KEDA will attempt to resolve references from secrets, config maps, and environment variables of the container. + +While this method works for many scenarios, there are some downsides. This method makes it difficult to efficiently share auth config across `ScaledObjects`. It also doesn’t support referencing a secret directly, only secrets that are referenced by the container. This method also doesn't support a model where other types of authentication may work - namely "pod identity" where access to a source could be acquired with no secrets or connection strings. For these and other reasons, we also provide a `TriggerAuthentication` resource to define authentication as a separate resource to a `ScaledObject`, which can reference secrets directly or supply configuration like pod identity. + +### The downsides + +While this method works for many scenarios, there are some downsides: + +* **Difficult to efficiently share auth** config across `ScaledObjects` +* **No support for referencing a secret directly**, only secrets that are referenced by the container +* **No support for other types of authentication flows** such as *pod identity* where access to a source could be acquired with no secrets or connection strings + +For these and other reasons, we also provide a `TriggerAuthentication` resource to define authentication as a separate resource to a `ScaledObject`. This allows you to reference secrets directly, configure to use pod identity or use authentication object managed by a different team. + +## Re-use credentials and delegate auth with TriggerAuthentication + +`TriggerAuthentication` allows you to describe authentication parameters separate from the `ScaledObject` and the deployment containers. It also enables more advanced methods of authentication like "pod identity", authentication re-use or allowing IT to configure the authentication. + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: {trigger-auth-name} + namespace: default # must be same namespace as the ScaledObject +spec: + podIdentity: + provider: none | azure | gcp | spiffe | aws-eks | aws-kiam # Optional. Default: none + secretTargetRef: # Optional. + - parameter: {scaledObject-parameter-name} # Required. + name: {secret-name} # Required. + key: {secret-key-name} # Required. + env: # Optional. + - parameter: {scaledObject-parameter-name} # Required. + name: {env-name} # Required. + containerName: {container-name} # Optional. Default: scaleTargetRef.envSourceContainerName of ScaledObject + hashiCorpVault: # Optional. + address: {hashicorp-vault-address} # Required. + authentication: token | kubernetes # Required. + role: {hashicorp-vault-role} # Optional. + mount: {hashicorp-vault-mount} # Optional. + credential: # Optional. + token: {hashicorp-vault-token} # Optional. + serviceAccount: {path-to-service-account-file} # Optional. + secrets: # Required. + - parameter: {scaledObject-parameter-name} # Required. + key: {hasicorp-vault-secret-key-name} # Required. + path: {hasicorp-vault-secret-path} # Required. +``` + +Based on the requirements you can mix and match the reference types providers in order to configure all required parameters. + +Every parameter you define in `TriggerAuthentication` definition does not need to be included in the `metadata` of the trigger for your `ScaledObject` definition. To reference a `TriggerAuthentication` from a `ScaledObject` you add the `authenticationRef` to the trigger. + +```yaml +# some Scaled Object +# ... + triggers: + - type: {scaler-type} + metadata: + param1: {some-value} + authenticationRef: + name: {trigger-authencation-name} # this may define other params not defined in metadata +``` + +## Authentication scopes: Namespace vs. Cluster + +Each `TriggerAuthentication` is defined in one namespace and can only be used by a `ScaledObject` in that same namespace. For cases where you want to share a single set of credentials between scalers in many namespaces, you can instead create a `ClusterTriggerAuthentication`. As a global object, this can be used from any namespace. To set a trigger to use a `ClusterTriggerAuthentication`, add a `kind` field to the authentication reference: + +```yaml + authenticationRef: + name: {trigger-authencation-name} + kind: ClusterTriggerAuthentication +``` + +By default, Secrets loaded from a `secretTargetRef` must be in the same namespace as KEDA is deployed in (usually `keda`). This can be overridden by setting a `KEDA_CLUSTER_OBJECT_NAMESPACE` environment variable for the `keda-operator` container. + +Defining a `ClusterTriggerAuthentication` works almost identically to a `TriggerAuthentication`, except there is no `metadata.namespace` value: + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ClusterTriggerAuthentication +metadata: + name: {cluster-trigger-auth-name} +spec: + # As before ... +``` + +## Authentication parameters + +Authentication parameters can be pulled in from many sources. All of these values are merged together to make the authentication data for the scaler. + +### Environment variable(s) + +You can pull information via one or more environment variables by providing the `name` of the variable for a given `containerName`. + +```yaml +env: # Optional. + - parameter: region # Required - Defined by the scale trigger + name: my-env-var # Required. + containerName: my-container # Optional. Default: scaleTargetRef.envSourceContainerName of ScaledObject +``` + +**Assumptions:** `containerName` is in the same resource as referenced by `scaleTargetRef.name` in the ScaledObject, unless specified otherwise. + +### Secret(s) + +You can pull one or more secrets into the trigger by defining the `name` of the Kubernetes Secret and the `key` to use. + +```yaml +secretTargetRef: # Optional. + - parameter: connectionString # Required - Defined by the scale trigger + name: my-keda-secret-entity # Required. + key: azure-storage-connectionstring # Required. +``` + +**Assumptions:** `namespace` is in the same resource as referenced by `scaleTargetRef.name` in the ScaledObject, unless specified otherwise. + +### Hashicorp Vault secret(s) + +You can pull one or more Hashicorp Vault secrets into the trigger by defining the autentication metadata such as Vault `address` and the `authentication` method (token | kubernetes). If you choose kubernetes auth method you should provide `role` and `mount` as well. +`credential` defines the Hashicorp Vault credentials depending on the authentication method, for kubernetes you should provide path to service account token (/var/run/secrets/kubernetes.io/serviceaccount/token) and for token auth method provide the token. +`secrets` list defines the mapping between the path and the key of the secret in Vault to the parameter. + +```yaml +hashiCorpVault: # Optional. + address: {hashicorp-vault-address} # Required. + authentication: token | kubernetes # Required. + role: {hashicorp-vault-role} # Optional. + mount: {hashicorp-vault-mount} # Optional. + credential: # Optional. + token: {hashicorp-vault-token} # Optional. + serviceAccount: {path-to-service-account-file} # Optional. + secrets: # Required. + - parameter: {scaledObject-parameter-name} # Required. + key: {hasicorp-vault-secret-key-name} # Required. + path: {hasicorp-vault-secret-path} # Required. +``` + +### Pod Authentication Providers + +Several service providers allow you to assign an identity to a pod. By using that identity, you can defer authentication to the pod & the service provider, rather than configuring secrets. + +Currently we support the following: + +```yaml +podIdentity: + provider: none | azure | gcp | spiffe | aws-eks | aws-kiam # Optional. Default: none +``` + +#### Azure Pod Identity + +Azure Pod Identity is an implementation of [**Azure AD Pod Identity**](https://github.com/Azure/aad-pod-identity) which let's you bind an [**Azure Managed Identity**](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/) to a Pod in a Kubernetes cluster as delegated access - *Don't manage secrets, let Azure AD do the hard work*. + +You can tell KEDA to use Azure AD Pod Identity via `podIdentity.provider`. + +```yaml +podIdentity: + provider: azure # Optional. Default: false +``` + +Azure AD Pod Identity will give access to containers with a defined label for `aadpodidbinding`. You can set this label on the KEDA operator deployment. This can be done for you during deployment with Helm with `--set podIdentity.activeDirectory.identity={your-label-name}`. + +#### EKS Pod Identity Webhook for AWS + +[**EKS Pod Identity Webhook**](https://github.com/aws/amazon-eks-pod-identity-webhook), which is described more in depth [here](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/), allows you to provide the role name using an annotation on a service account associated with your pod. + +You can tell KEDA to use EKS Pod Identity Webhook via `podIdentity.provider`. + +```yaml +podIdentity: + provider: aws-eks # Optional. Default: false +``` + +#### Kiam Pod Identity for AWS + +[**Kiam**](https://github.com/uswitch/kiam/) lets you bind an AWS IAM Role to a pod using an annotation on the pod. + +You can tell KEDA to use Kiam via `podIdentity.provider`. + +```yaml +podIdentity: + provider: aws-kiam # Optional. Default: false +``` diff --git a/content/docs/2.3/concepts/external-scalers.md b/content/docs/2.3/concepts/external-scalers.md new file mode 100644 index 000000000..9a144d94f --- /dev/null +++ b/content/docs/2.3/concepts/external-scalers.md @@ -0,0 +1,592 @@ ++++ +title = "External Scalers" +weight = 500 ++++ + +While KEDA ships with a set of [built-in scalers](../scalers), users can also extend KEDA through a GRPC service that implements the same interface as the built-in scalers. + +Built-in scalers run in the KEDA process/pod, while external scalers require an externally managed GRPC server that's accessible from KEDA with optional [TLS authentication](https://grpc.io/docs/guides/auth/). KEDA itself acts as a GRPC client and it exposes similar service interface for the built-in scalers, so external scalers can fully replace built-in ones. + +This document describes the external scaler interfaces and how to implement them in Go, Node, and .NET; however for more details on GRPC refer to [the official GRPC documentation](https://grpc.io/docs/) + +Want to learn about existing external scalers? Explore our [external scaler community](https://github.com/kedacore/external-scalers). + +## Overview + +### Built-in scalers interface + +Built-in scalers implement one of the following go interfaces: + +```go +type Scaler interface { + GetMetrics(ctx context.Context, metricName string, metricSelector labels.Selector) ([]external_metrics.ExternalMetricValue, error) + GetMetricSpecForScaling() []v2beta2.MetricSpec + IsActive(ctx context.Context) (bool, error) + Close() error +} + +type PushScaler interface { + Scaler + + Run(ctx context.Context, active chan<- bool) +} +``` + +The `Scaler` interface defines 4 methods: +- `IsActive` is called on `pollingInterval` defined in the ScaledObject/ScaledJob CRDs and scaling to 1 happens if this returns true +- `Close` is called to allow the scaler to clean up connections or other resources. +- `GetMetricSpec` returns the target value for the HPA definition for the scaler. For more details refer to [Implementing `GetMetricSpec`](#5-implementing-getmetricspec) +- `GetMetrics` returns the value of the metric refered to from `GetMetricSpec`. For more details refer to [Implementing `GetMetrics`](#6-implementing-getmetrics) + +> Refer to the [HPA docs](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/) for how HPA calculates `replicaCount` based on metric value and target value. KEDA uses the metric target type `AverageValue` for external metrics. This will cause the metric value returned by the external scaler to be divided by the number of replicas. + +The `PushScaler` interface adds `Run` method. The `Run` method receives a push channel (`active`), that the scaler can push `true` to any time to force scaling action independently from `pollingInterval`. + + +### External Scaler GRPC interface + +KEDA comes with 2 external scalers [`external`](../scalers/external.md) and [`external-push`](../scalers/external-push.md). + +The configuration in the ScaledObject points to a GRPC service endpoint that implements the following GRPC contract [`externalscaler.proto`](https://github.com/kedacore/keda/blob/main/pkg/scalers/externalscaler/externalscaler.proto): + +```proto +service ExternalScaler { + rpc IsActive(ScaledObjectRef) returns (IsActiveResponse) {} + rpc StreamIsActive(ScaledObjectRef) returns (stream IsActiveResponse) {} + rpc GetMetricSpec(ScaledObjectRef) returns (GetMetricSpecResponse) {} + rpc GetMetrics(GetMetricsRequest) returns (GetMetricsResponse) {} +} +``` + +- `GetMetrics` and `GetMetricsSpec` mirror their counterparts in the `Scaler` interface for creating HPA definition +- `IsActive` maps to the `IsActive` method on the `Scaler` interface +- `StreamingIsActive` maps to the `Run` method on the `PushScaler` interface. + +Few things to notice: +- Lack of `Close` method as the GRPC connection defines the lifetime of the scaler +- `IsActive`, `StreamIsActive`, and `GetMetricsSpec` are called with a `ScaledObjectRef` that contains the scaledObject name/namespace as well as the content of `metadata` defined in the trigger. + +For example the following `ScaledObject` + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: scaledobject-name + namespace: scaledobject-namespace +spec: + scaleTargetRef: + name: deployment-name + triggers: + - type: external + metadata: + scalerAddress: service-address.svc.local:9090 + key1: value1 + key2: value2 +``` + +KEDA will attempt a connection to `service-address.svc.local:9090` and calls `IsActive`, `StreamIsActive`, and `GetMetricsSpec` with the following `ScaledObjectRef` + +```json +{ + "name": "scaledobject-name", + "namespace": "scaledobject-namespace", + "scalerMetadata": { + "scalerAddress": "service-address.svc.local:9090", + "key1": "value1", + "key2": "value2" + } +} +``` + +For `StreamingIsActive` KEDA establishes the connection to the GRPC server and expects `IsActive` events to be streamed as a response. + +## Implementing KEDA external scaler GRPC interface + +### Implementing an external scaler: + +#### 1. Download [`externalscaler.proto`](https://github.com/kedacore/keda/blob/main/pkg/scalers/externalscaler/externalscaler.proto) + +#### 2. Prepare project: + +{{< collapsible "Golang" >}} + +2.1. Download [`./protoc`](https://github.com/protocolbuffers/protobuf/releases) for your platform + +2.2. get `protoc-gen-go` + +```bash +go get github.com/golang/protobuf/protoc-gen-go@v1.3.2 +``` + +2.3. Prepare project + +```bash +go mod init example.com/external-scaler/sample +mkdir externalscaler +protoc externalscaler.proto --go_out=plugins=grpc:externalscaler +``` +{{< /collapsible >}} + +{{< collapsible "C#" >}} +2.1. Create a new project + +```bash +dotnet new console -o ExternalScalerSample +cd ExternalScalerSample + +# add Grpc.AspNetCore +dotnet add package Grpc.AspNetCore +dotnet add package Newtonsoft.Json + +# Create a Protos and Services folders +mkdir Protos +mkdir Services +``` + +2.2. Move `externalscaler.proto` to `Protos` folder + +2.3. Compile `externalscaler.proto` using this in `ExternalScalerSample.csproj` + +```xml + + + +``` + +{{< /collapsible >}} + +{{< collapsible "Javascript" >}} +2.1. Prepare the project + +```bash +npm install --save grpc request +``` +{{< /collapsible >}} + +
+ +#### 3. Implementing `IsActive` + +Just like `IsActive(ctx context.Context) (bool, error)` in the go interface, `IsActive` method in the GRPC interface is called every `pollingInterval` with a `ScaledObjectRef` object that contains the scaledObject name, namespace, and scaler metadata. This section implements an external scaler that queries earthquakes from https://earthquake.usgs.gov/ and scales the deployment if there has been more than 2 earthquakes with `magnitude > 1.0` around a particular longitude/latitude in the previous day + +`ScaledObject` + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: scaledobject-name + namespace: scaledobject-namespace +spec: + scaleTargetRef: + name: deployment-name + triggers: + - type: external + metadata: + scalerAddress: earthquake-scaler:9090 + longitude: "-122.335167" + latitude: "47.608013" +``` + +{{< collapsible "Golang" >}} + +Full implementation can be found here: https://github.com/kedacore/external-scaler-samples + +`main.go` +```golang +func (e *ExternalScaler) IsActive(ctx context.Context, scaledObject *pb.ScaledObjectRef) (*pb.IsActiveResponse, error) { + // request.Scalermetadata contains the `metadata` defined in the ScaledObject + longitude := scaledObject.ScalerMetadata["longitude"] + latitude := scaledObject.ScalerMetadata["latitude"] + + if len(longitude) == 0 || len(latitude) == 0 { + return nil, status.Error(codes.InvalidArgument, "longitude and latitude must be specified") + } + + startTime := time.Now().AddDate(0, 0, -1).Format("2006-01-02") + endTime := time.Now().Format("2006-01-02") + radiusKM := 500 + query := fmt.Sprintf("format=geojson&starttime=%s&endtime=%s&longitude=%s&latitude=%s&maxradiuskm=%d", startTime, endTime, longitude, latitude, radiusKM) + + resp, err := http.Get(fmt.Sprintf("https://earthquake.usgs.gov/fdsnws/event/1/query?%s", query)) + if err != nil { + return nil, status.Error(codes.Internal, err.Error()) + } + + defer resp.Body.Close() + body, err := ioutil.ReadAll(resp.Body) + if err != nil { + return nil, status.Error(codes.Internal, err.Error()) + } + + payload := USGSResponse{} + err = json.Unmarshal(body, &payload) + if err != nil { + return nil, status.Error(codes.Internal, err.Error()) + } + + // count how many earthquakes with mag > 1.0 + count := 0 + for _, f := range payload.Features { + if f.Properties.Mag > 1.0 { + count++ + } + } + + // return true if there is more than 2 + return &pb.IsActiveResponse{ + Result: count > 2, + }, nil +} +``` +{{< /collapsible >}} + +{{< collapsible "C#" >}} +Full implementation can be found here: https://github.com/kedacore/external-scaler-samples + +`Services/ExternalScalerService.cs` +```csharp +public class ExternalScalerService : ExternalScaler.ExternalScalerBase +{ + private static readonly HttpClient _client = new HttpClient(); + + public override async Task IsActive(ScaledObjectRef request, ServerCallContext context) + { + // request.Scalermetadata contains the `metadata` defined in the ScaledObject + if (!request.ScalerMetadata.ContainsKey("latitude") || + !request.ScalerMetadata.ContainsKey("longitude")) { + throw new ArgumentException("longitude and latitude must be specified"); + } + + var longitude = request.ScalerMetadata["longitude"]; + var latitude = request.ScalerMetadata["latitude"]; + var startTime = DateTime.UtcNow.AddDays(-1).ToString("yyyy-MM-dd"); + var endTime = DateTime.UtcNow.ToString("yyyy-MM-dd"); + var radiusKm = 500; + var query = $"format=geojson&starttime={startTime}&endtime={endTime}&longitude={longitude}&latitude={latitude}&maxradiuskm={radiusKm}"; + + var resp = await _client.GetAsync($"https://earthquake.usgs.gov/fdsnws/event/1/query?{query}"); + resp.EnsureSuccessStatusCode(); + var payload = JsonConvert.DeserializeObject(await resp.Content.ReadAsStringAsync()); + + return new IsActiveResponse + { + // return true if there is more than 2 Earthquakes with mag > 1.0 + Result = payload.features.Count(f => f.properties.mag > 1.0) > 2 + }; + } +} +``` +{{< /collapsible >}} + +{{< collapsible "Javascript" >}} +`index.js` +```js +const grpc = require('grpc') +const request = require('request') +const externalScalerProto = grpc.load('externalscaler.proto') + +const server = new grpc.Server() +server.addService(externalScalerProto.externalscaler.ExternalScaler.service, { + isActive: (call, callback) => { + const longitude = call.request.scalerMetadata.longitude + const latitude = call.request.scalerMetadata.latitude + if (!longitude || !latitude) { + callback({ + code: grpc.status.INVALID_ARGUMENT, + details: 'longitude and latitude must be specified', + }) + } else { + const now = new Date() + const yesterday = new Date(new Date().setDate(new Date().getDate()-1)); + + const startTime = `${yesterday.getUTCFullYear()}-${yesterday.getUTCMonth()}-${yesterday.getUTCDay()}` + const endTime = `${now.getUTCFullYear()}-${now.getUTCMonth()}-${now.getUTCDay()}` + const radiusKm = 500 + const query = `format=geojson&starttime=${startTime}&endtime=${endTime}&longitude=${longitude}&latitude=${latitude}&maxradiuskm=${radiusKm}` + + request.get({ + url: `https://earthquake.usgs.gov/fdsnws/event/1/query?${query}`, + json: true, + }, (err, resp, data) => { + if (err) { + callback({ + code: grpc.status.INTERNAL, + details: err, + }) + } else if (resp.statusCode !== 200) { + callback({ + code: grpc.status.INTERNAL, + details: `expected status 200, got ${resp.statusCode}` + }) + } else { + // count how many earthquakes with mag > 1.0 + let count = 0 + data.features.forEach(i => { + if (i.properties.mag > 1.0) { + count++ + } + }) + callback(null, { + result: count > 2, + }) + } + }) + } + } +}) + +server.bind('127.0.0.1:9090', grpc.ServerCredentials.createInsecure()) +console.log('Server listening on 127.0.0.1:9090') + +server.start() +``` +{{< /collapsible >}} + +
+ +#### 4. Implementing `StreamingIsActive` + +Unlike `IsActive`, `StreamingIsActive` is called once when KEDA reconciles the `ScaledObject`, and expects the external scaler to push `IsActiveResponse` whenever the scaler needs KEDA to activate the deployment. + +This implementation creates a timer and queries USGS APIs on that timer, effectively ignoring `pollingInterval` set in the scaledObject. Alternatively any other asynchronous event can be used instead of a timer, like an HTTP request, or a network connection. + + +{{< collapsible "Golang" >}} +```golang +func (e *ExternalScaler) StreamIsActive(scaledObject *pb.ScaledObjectRef, epsServer pb.ExternalScaler_StreamIsActiveServer) error { + longitude := scaledObject.ScalerMetadata["longitude"] + latitude := scaledObject.ScalerMetadata["latitude"] + + if len(longitude) == 0 || len(latitude) == 0 { + return status.Error(codes.InvalidArgument, "longitude and latitude must be specified") + } + + for { + select { + case <-epsServer.Context().Done(): + // call cancelled + return nil + case <-time.Tick(time.Hour * 1): + earthquakeCount, err := getEarthQuakeCount(longitude, latitude) + if err != nil { + // log error + } else if earthquakeCount > 2 { + err = epsServer.Send(&pb.IsActiveResponse{ + Result: true, + }) + } + } + } +} +``` +{{< /collapsible >}} + +{{< collapsible "C#" >}} +```csharp +public override Task StreamIsActive(ScaledObjectRef request, IServerStreamWriter responseStream, ServerCallContext context) +{ + if (!request.ScalerMetadata.ContainsKey("latitude") || + !request.ScalerMetadata.ContainsKey("longitude")) + { + throw new ArgumentException("longitude and latitude must be specified"); + } + + var longitude = request.ScalerMetadata["longitude"]; + var latitude = request.ScalerMetadata["latitude"]; + var key = $"{longitude}|{latitude}"; + + while (!context.CancellationToken.IsCancellationRequested) + { + var earthquakeCount = await GetEarthQuakeCount(longitude, latitude); + if (earthquakeCount > 2) { + await responseStream.WriteAsync(new IsActiveResponse + { + Result = true + }); + } + + await Task.Delay(TimeSpan.FromHours(1)); + } +} +``` +{{< /collapsible >}} + +{{< collapsible "Javascript" >}} +```js +server.addService(externalScalerProto.externalscaler.ExternalScaler.service, { + // ... + streamIsActive: (call, callback) => { + const longitude = call.request.scalerMetadata.longitude + const latitude = call.request.scalerMetadata.latitude + if (!longitude || !latitude) { + callback({ + code: grpc.status.INVALID_ARGUMENT, + details: 'longitude and latitude must be specified', + }) + } else { + const interval = setInterval(() => { + getEarthquakeCount((err, count) => { + if (err) { + console.error(err) + } else if (count > 2) { + call.write({ + result: true, + }) + } + }) + }, 1000 * 60 * 60); + + call.on('end', () => { + clearInterval(interval) + }) + } + } +}) +``` +{{< /collapsible >}} + +
+ +#### 5. Implementing `GetMetricSpec` + +`GetMetricSpec` returns the `target` value for [the HPA definition for the scaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/). This scaler will define a static target of 10, but the threshold value is often specified in the metadata for other scalers. + +{{< collapsible "Golang" >}} +```golang +func (e *ExternalScaler) GetMetricSpec(context.Context, *pb.ScaledObjectRef) (*pb.GetMetricSpecResponse, error) { + return &pb.GetMetricSpecResponse{ + MetricSpecs: []*pb.MetricSpec{{ + MetricName: "earthquakeThreshold", + TargetSize: 10, + }}, + }, nil +} +``` +{{< /collapsible >}} + +{{< collapsible "C#" >}} +```csharp +public override Task GetMetricSpec(ScaledObjectRef request, ServerCallContext context) +{ + var resp = new GetMetricSpecResponse(); + + resp.MetricSpecs.Add(new MetricSpec + { + MetricName = "earthquakeThreshold", + TargetSize = 10 + }); + + return Task.FromResult(resp); +} +``` +{{< /collapsible >}} + +{{< collapsible "Javascript" >}} +```js +server.addService(externalScalerProto.externalscaler.ExternalScaler.service, { + // ... + getMetricSpec: (call, callback) => { + callback(null, { + metricSpecs: [{ + metricName: 'earthquakeThreshold', + targetSize: 10, + }] + }) + } +}) +``` +{{< /collapsible >}} + +
+ +#### 6. Implementing `GetMetrics` + +`GetMetrics` returns the value of the metric refered to from `GetMetricSpec`, in this example it's `earthquakeThreshold`. + +{{< collapsible "Golang" >}} +```golang +func (e *ExternalScaler) GetMetrics(_ context.Context, metricRequest *pb.GetMetricsRequest) (*pb.GetMetricsResponse, error) { + longitude := metricRequest.ScaledObjectRef.ScalerMetadata["longitude"] + latitude := metricRequest.ScaledObjectRef.ScalerMetadata["latitude"] + + if len(longitude) == 0 || len(latitude) == 0 { + return nil, status.Error(codes.InvalidArgument, "longitude and latitude must be specified") + } + + earthquakeCount, err := getEarthQuakeCount(longitude, latitude, 1.0) + if err != nil { + return nil, status.Error(codes.Internal, err.Error()) + } + + return &pb.GetMetricsResponse{ + MetricValues: []*pb.MetricValue{{ + MetricName: "earthquakeThreshold", + MetricValue: int64(earthquakeCount), + }}, + }, nil +} +``` +{{< /collapsible >}} + +{{< collapsible "C#" >}} +```csharp +public override async Task GetMetrics(GetMetricsRequest request, ServerCallContext context) +{ + if (!request.ScaledObjectRef.ScalerMetadata.ContainsKey("latitude") || + !request.ScaledObjectRef.ScalerMetadata.ContainsKey("longitude")) + { + throw new ArgumentException("longitude and latitude must be specified"); + } + + var longitude = request.ScaledObjectRef.ScalerMetadata["longitude"]; + var latitude = request.ScaledObjectRef.ScalerMetadata["latitude"]; + + var earthquakeCount = await GetEarthQuakeCount(longitude, latitude); + + var resp = new GetMetricsResponse(); + resp.MetricValues.Add(new MetricValue + { + MetricName = "earthquakeThreshold", + MetricValue_ = earthquakeCount + }); + + return resp; +} +``` +{{< /collapsible >}} + +{{< collapsible "Javascript" >}} +```js +server.addService(externalScalerProto.externalscaler.ExternalScaler.service, { + // ... + getMetrics: (call, callback) => { + const longitude = call.request.scaledObjectRef.scalerMetadata.longitude + const latitude = call.request.scaledObjectRef.scalerMetadata.latitude + if (!longitude || !latitude) { + callback({ + code: grpc.status.INVALID_ARGUMENT, + details: 'longitude and latitude must be specified', + }) + } else { + getEarthquakeCount((err, count) => { + if (err) { + callback({ + code: grpc.status.INTERNAL, + details: err, + }) + } else { + callback(null, { + metricValues: [{ + metricName: 'earthquakeThreshold', + metricValue: count, + }] + }) + } + }) + } + } +}) +``` +{{< /collapsible >}} diff --git a/content/docs/2.3/concepts/scaling-deployments.md b/content/docs/2.3/concepts/scaling-deployments.md new file mode 100644 index 000000000..e9bf65b54 --- /dev/null +++ b/content/docs/2.3/concepts/scaling-deployments.md @@ -0,0 +1,174 @@ ++++ +title = "Scaling Deployments, StatefulSets & Custom Resources" +weight = 200 ++++ + +## Overview + +### Scaling of Deployments and StatefulSets + +Deployments and StatefulSets are the most common way to scale workloads with KEDA. + +It allows you to define the Kubernetes Deployment or StatefulSet that you want KEDA to scale based on a scale trigger. KEDA will monitor that service and based on the events that occur it will automatically scale your resource out/in accordingly. + +Behind the scenes, KEDA acts to monitor the event source and feed that data to Kubernetes and the HPA (Horizontal Pod Autoscaler) to drive rapid scale of a resource. Each replica of a resource is actively pulling items from the event source. With KEDA and scaling Deployments/StatefulSet you can scale based on events while also preserving rich connection and processing semantics with the event source (e.g. in-order processing, retries, deadletter, checkpointing). + +For example, if you wanted to use KEDA with an Apache Kafka topic as event source, the flow of information would be: + +* When no messages are pending processing, KEDA can scale the deployment to zero. +* When a message arrives, KEDA detects this event and activates the deployment. +* When the deployment starts running, one of the containers connects to Kafka and starts pulling messages. +* As more messages arrive on the Kafka Topic, KEDA can feed this data to the HPA to drive scale out. +* Each replica of the deployment is actively processing messages. Very likely, each replica is processing a batch of messages in a distributed manner. + +### Scaling of Custom Resources + +With KEDA you can scale any workload defined as any `Custom Resource` (for example `ArgoRollout` [resource](https://argoproj.github.io/argo-rollouts/)). The scaling behaves the same way as scaling for arbitrary Kubernetes `Deployment` or `StatefulSet`. + +The only constraint is that the target `Custom Resource` must define `/scale` [subresource](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#scale-subresource). + +## ScaledObject spec + +This specification describes the `ScaledObject` Custom Resource definition which is used to define how KEDA should scale your application and what the triggers are. The `.spec.ScaleTargetRef` section holds the reference to the target resource, ie. `Deployment`, `StatefulSet` or `Custom Resource`. + +[`scaledobject_types.go`](https://github.com/kedacore/keda/blob/main/api/v1alpha1/scaledobject_types.go) + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: {scaled-object-name} +spec: + scaleTargetRef: + apiVersion: {api-version-of-target-resource} # Optional. Default: apps/v1 + kind: {kind-of-target-resource} # Optional. Default: Deployment + name: {name-of-target-resource} # Mandatory. Must be in the same namespace as the ScaledObject + envSourceContainerName: {container-name} # Optional. Default: .spec.template.spec.containers[0] + pollingInterval: 30 # Optional. Default: 30 seconds + cooldownPeriod: 300 # Optional. Default: 300 seconds + minReplicaCount: 0 # Optional. Default: 0 + maxReplicaCount: 100 # Optional. Default: 100 + advanced: # Optional. Section to specify advanced options + restoreToOriginalReplicaCount: true/false # Optional. Default: false + horizontalPodAutoscalerConfig: # Optional. Section to specify HPA related options + behavior: # Optional. Use to modify HPA's scaling behavior + scaleDown: + stabilizationWindowSeconds: 300 + policies: + - type: Percent + value: 100 + periodSeconds: 15 + triggers: + # {list of triggers to activate scaling of the target resource} +``` + +> πŸ’‘ **NOTE:** You can find all supported triggers [here](/scalers). + +### Details +```yaml + scaleTargetRef: + apiVersion: {api-version-of-target-resource} # Optional. Default: apps/v1 + kind: {kind-of-target-resource} # Optional. Default: Deployment + name: {name-of-target-resource} # Mandatory. Must be in the same namespace as the ScaledObject + envSourceContainerName: {container-name} # Optional. Default: .spec.template.spec.containers[0] +``` + +The reference to the resource this ScaledObject is configured for. This is the resource KEDA will scale up/down and setup an HPA for, based on the triggers defined in `triggers:`. + +To scale Kubernetes Deployments only `name` is needed to be specified, if one wants to scale a different resource such as StatefulSet or Custom Resource (that defines `/scale` subresource), appropriate `apiVersion` (following standard Kubernetes convetion, ie. `{api}/{version}`) and `kind` need to be specified. + +`envSourceContainerName` is an optional property that specifies the name of container in the target resource, from which KEDA should try to get environment properties holding secrets etc. If it is not defined, KEDA will try to get environment properties from the first Container, ie. from `.spec.template.spec.containers[0]`. + +**Assumptions:** Resource referenced by `name` (and `apiVersion`, `kind`) is in the same namespace as the ScaledObject + +--- + +```yaml + pollingInterval: 30 # Optional. Default: 30 seconds +``` + +This is the interval to check each trigger on. By default KEDA will check each trigger source on every ScaledObject every 30 seconds. + +**Example:** in a queue scenario, KEDA will check the queueLength every `pollingInterval`, and scale the resource up or down accordingly. + +--- + +```yaml + cooldownPeriod: 300 # Optional. Default: 300 seconds +``` + +The period to wait after the last trigger reported active before scaling the resource back to 0. By default it's 5 minutes (300 seconds). + +The `cooldownPeriod` only applies after a trigger occurs; when you first create your `Deployment` (or `StatefulSet`/`CustomResource`), KEDA will immediately scale it to `minReplicaCount`. Additionally, the KEDA `cooldownPeriod` only applies when scaling to 0; scaling from 1 to N replicas is handled by the [Kubernetes Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-cooldowndelay). + +**Example:** wait 5 minutes after the last time KEDA checked the queue and it was empty. (this is obviously dependent on `pollingInterval`) + +--- + +```yaml + minReplicaCount: 0 # Optional. Default: 0 +``` + +Minimum number of replicas KEDA will scale the resource down to. By default it's scale to zero, but you can use it with some other value as well. KEDA will not enforce that value, meaning you can manually scale the resource to 0 and KEDA will not scale it back up. However, when KEDA itself is scaling the resource it will respect the value set there. + +--- + +```yaml + maxReplicaCount: 100 # Optional. Default: 100 +``` + +This setting is passed to the HPA definition that KEDA will create for a given resource. + +--- + +```yaml +advanced: + restoreToOriginalReplicaCount: true/false # Optional. Default: false +``` + +This property specifies whether the target resource (`Deployment`, `StatefulSet`,...) should be scaled back to original replicas count, after the `ScaledObject` is deleted. +Default behavior is to keep the replica count at the same number as it is in the moment of `ScaledObject's` deletion. + +For example a `Deployment` with `3 replicas` is created, then `ScaledObject` is created and the `Deployment` is scaled by KEDA to `10 replicas`. Then `ScaledObject` is deleted: + 1. if `restoreToOriginalReplicaCount = false` (default behavior) then `Deployment` replicas count is `10` + 2. if `restoreToOriginalReplicaCount = true` then `Deployment` replicas count is set back to `3` (the original value) + +--- + +```yaml +advanced: + horizontalPodAutoscalerConfig: # Optional. Section to specify HPA related options + behavior: # Optional. Use to modify HPA's scaling behavior + scaleDown: + stabilizationWindowSeconds: 300 + policies: + - type: Percent + value: 100 + periodSeconds: 15 +``` + +**`horizontalPodAutoscalerConfig:`** + +**`horizontalPodAutoscalerConfig.behavior`:** + +Starting from Kubernetes v1.18 the autoscaling API allows scaling behavior to be configured through the HPA behavior field. This way one can directly affect scaling of 1<->N replicas, which is internally being handled by HPA. KEDA would feed values from this section directly to the HPA's `behavior` field. Please follow [Kubernetes documentation](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior) for details. + +**Assumptions:** KEDA must be running on Kubernetes cluster v1.18+, in order to be able to benefit from this setting. + +## Long-running executions + +One important consideration to make is how this pattern can work with long running executions. Imagine a deployment triggers on a RabbitMQ queue message. Each message takes 3 hours to process. It's possible that if many queue messages arrive, KEDA will help drive scaling out to many replicas - let's say 4. Now the HPA makes a decision to scale down from 4 replicas to 2. There is no way to control which of the 2 replicas get terminated to scale down. That means the HPA may attempt to terminate a replica that is 2.9 hours into processing a 3 hour queue message. + +There are two main ways to handle this scenario. + +### Leverage the container lifecycle + +Kubernetes provides a few [lifecycle hooks](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/) that can be leveraged to delay termination. Imagine a replica is scheduled for termination and is 2.9 hours into processing a 3 hour message. Kubernetes will send a [`SIGTERM`](https://www.gnu.org/software/libc/manual/html_node/Termination-Signals.html) to signal the intent to terminate. Rather than immediately terminating, a deployment can delay termination until processing the current batch of messages has completed. Kubernetes will wait for a `SIGTERM` response or the `terminationGracePeriodSeconds` before killing the replica. + +> πŸ’‘ **NOTE:**There are other ways to delay termination, including the [`preStop` Hook](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks) + +Using this method can preserve a replica and enable long-running executions. However, one downside of this approach is while delaying termination, the pod phase will remain in the `Terminating` state. That means a pod that is delaying termination for a very long duration may show `Terminating` during that entire period of delay. + +### Run as jobs + +The other alternative to handling long running executions is by running the event driven code in Kubernetes Jobs instead of Deployments or Custom Resources. This approach is discussed [in the next section](../scaling-jobs). diff --git a/content/docs/2.3/concepts/scaling-jobs.md b/content/docs/2.3/concepts/scaling-jobs.md new file mode 100644 index 000000000..361563974 --- /dev/null +++ b/content/docs/2.3/concepts/scaling-jobs.md @@ -0,0 +1,201 @@ ++++ +title = "Scaling Jobs" +weight = 300 ++++ + + +## Overview + +As an alternate to [scaling event-driven code as deployments](../scaling-deployments) you can also run and scale your code as Kubernetes Jobs. The primary reason to consider this option is to handle processing long running executions. Rather than processing multiple events within a deployment, for each detected event a single Kubernetes Job is scheduled. That job will initialize, pull a single event from the message source, and process to completion and terminate. + +For example, if you wanted to use KEDA to run a job for each message that lands on a RabbitMQ queue, the flow may be: + +1. When no messages are awaiting processing, no jobs are created. +1. When a message arrives on the queue, KEDA creates a job. +1. When the job starts running, it pulls *a single* message and processes it to completion. +1. As additional messages arrive, additional jobs are created. Each job processes a single message to completion. +1. Periodically remove completed/failed job by the `SuccessfulJobsHistoryLimit` and `FailedJobsHistoryLimit.` + +## ScaledJob spec + +This specification describes the `ScaledJob` custom resource definition which is used to define how KEDA should scale your application and what the triggers are. + +[`scaledjob_types.go`](https://github.com/kedacore/keda/blob/main/api/v1alpha1/scaledjob_types.go) + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledJob +metadata: + name: {scaled-job-name} +spec: + jobTargetRef: + parallelism: 1 # [max number of desired pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism) + completions: 1 # [desired number of successfully finished pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism) + activeDeadlineSeconds: 600 # Specifies the duration in seconds relative to the startTime that the job may be active before the system tries to terminate it; value must be positive integer + backoffLimit: 6 # Specifies the number of retries before marking this job failed. Defaults to 6 + template: + # describes the [job template](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) + pollingInterval: 30 # Optional. Default: 30 seconds + successfulJobsHistoryLimit: 5 # Optional. Default: 100. How many completed jobs should be kept. + failedJobsHistoryLimit: 5 # Optional. Default: 100. How many failed jobs should be kept. + envSourceContainerName: {container-name} # Optional. Default: .spec.JobTargetRef.template.spec.containers[0] + maxReplicaCount: 100 # Optional. Default: 100 + scalingStrategy: + strategy: "custom" # Optional. Default: default. Which Scaling Strategy to use. + customScalingQueueLengthDeduction: 1 # Optional. A parameter to optimize custom ScalingStrategy. + customScalingRunningJobPercentage: "0.5" # Optional. A parameter to optimize custom ScalingStrategy. + triggers: + # {list of triggers to create jobs} +``` + +You can find all supported triggers [here](../scalers). + +## Details + +```yaml + jobTargetRef: + parallelism: 1 # Max number of desired instances ([docs](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)) + completions: 1 # Desired number of successfully finished instances ([docs](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)) + activeDeadlineSeconds: 600 # Specifies the duration in seconds relative to the startTime that the job may be active before the system tries to terminate it; value must be positive integer + backoffLimit: 6 # Specifies the number of retries before marking this job failed. Defaults to 6 +``` + +--- + +```yaml + pollingInterval: 30 # Optional. Default: 30 seconds +``` + +This is the interval to check each trigger on. By default, KEDA will check each trigger source on every ScaledJob every 30 seconds. + +--- + +```yaml + successfulJobsHistoryLimit: 5 # Optional. Default: 100. How many completed jobs should be kept. + failedJobsHistoryLimit: 5 # Optional. Default: 100. How many failed jobs should be kept. +``` + +The `successfulJobsHistoryLimit` and `failedJobsHistoryLimit` fields are optional. These fields specify how many completed and failed jobs should be kept. By default, they are set to 100. + +This concept is similar to [Jobs History Limits](https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#jobs-history-limits) allowing you to learn what the outcomes of your jobs are. + +The actual number of jobs could exceed the limit in a short time. However, it is going to resolve in the cleanup period. Currently, the cleanup period is the same as the Polling interval. + +--- + + +```yaml + envSourceContainerName: {container-name} # Optional. Default: .spec.JobTargetRef.template.spec.containers[0] +``` + +This optional property specifies the name of container in the Job, from which KEDA should try to get environment properties holding secrets etc. If it is not defined it, KEDA will try to get environment properties from the first Container, ie. from `.spec.JobTargetRef.template.spec.containers[0]`. + +--- + +```yaml + maxReplicaCount: 100 # Optional. Default: 100 +``` + +The max number of pods that is created within a single polling period. If there are running jobs, the number of running jobs will be deducted. This table is an example of the scaling logic. + +| Queue Length | Max Replica Count | Target Average Value | Running Job Count | Number of the Scale | +| ------- | ------ | ------- | ------ | ----- | +| 10 | 3 | 1 | 0 | 3 | +| 10 | 3 | 2 | 0 | 3 | +| 10 | 3 | 1 | 1 | 2 | +| 10 | 100 | 1 | 0 | 10 | +| 4 | 3 | 5 | 0 | 1 | + +* **Queue Length:** The number of items in the queue. +* **Target Average Value:** The number of messages that will be consumed on a job. It is defined on the scaler side. e.g. `queueLength` on `Azure Storage Queue` scaler. +* **Running Job Count:** How many jobs are running. +* **Number of the Scale:** The number of the job that is created. + +--- + +```yaml +scalingStrategy: + strategy: "default" # Optional. Default: default. Which Scaling Strategy to use. +``` + +Select a Scaling Strategy. Possible values are `default`, `custom`, or `accurate`. The default value is `default`. + +**default** +This logic is the same as Job for V1. The number of the scale will be calculated as follows. + +_The number of the scale_ + +```go +queueLength - runningJobCount +``` + +**custom** +You can customize the default scale logic. You need to configure the following parameters. If you don't configure it, then the strategy will be `default.` + +```yaml +customScalingQueueLengthDeduction: 1 # Optional. A parameter to optimize custom ScalingStrategy. +customScalingRunningJobPercentage: "0.5" # Optional. A parameter to optimize custom ScalingStrategy. +``` + +_The number of the scale_ + +```go +queueLength - customScalingQueueLengthDeduction - (runningJobCount * customScalingRunningJobPercentage) +``` + +**accurate** +If the scaler returns `queueLength` that does not include the number of locked messages, this strategy is recommended. `Azure Storage Queue` is one example. You can use this strategy if you delete a message once your app consumes it. + +```go +if (maxScale + runningJobCount) > maxReplicaCount { + return maxReplicaCount - runningJobCount +} +return queueLength +``` +For more details, you can refer to [this PR](https://github.com/kedacore/keda/pull/1227). + +# Sample + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: rabbitmq-consumer +data: + RabbitMqHost: +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledJob +metadata: + name: rabbitmq-consumer + namespace: default +spec: + jobTargetRef: + template: + spec: + containers: + - name: rabbitmq-client + image: tsuyoshiushio/rabbitmq-client:dev3 + imagePullPolicy: Always + command: ["receive", "amqp://user:PASSWORD@rabbitmq.default.svc.cluster.local:5672", "job"] + envFrom: + - secretRef: + name: rabbitmq-consumer + restartPolicy: Never + backoffLimit: 4 + pollingInterval: 10 # Optional. Default: 30 seconds + maxReplicaCount: 30 # Optional. Default: 100 + successfulJobsHistoryLimit: 3 # Optional. Default: 100. How many completed jobs should be kept. + failedJobsHistoryLimit: 2 # Optional. Default: 100. How many failed jobs should be kept. + scalingStrategy: + strategy: "custom" # Optional. Default: default. Which Scaling Strategy to use. + customScalingQueueLengthDeduction: 1 # Optional. A parameter to optimize custom ScalingStrategy. + customScalingRunningJobPercentage: "0.5" # Optional. A parameter to optimize custom ScalingStrategy. + triggers: + - type: rabbitmq + metadata: + queueName: hello + host: RabbitMqHost + queueLength : '5' +``` + diff --git a/content/docs/2.3/concepts/troubleshooting.md b/content/docs/2.3/concepts/troubleshooting.md new file mode 100644 index 000000000..68b19bf15 --- /dev/null +++ b/content/docs/2.3/concepts/troubleshooting.md @@ -0,0 +1,24 @@ ++++ +title = "Troubleshooting" +weight = 600 ++++ + +## KEDA logging and telemetry + +The first place to look if something isn't behaving correctly is the logs generated from KEDA. After deploying you should have a pod with two containers running within the namespace (by default: `keda`). + +You can view the KEDA operator pod via kubectl: + +```sh +kubectl get pods -n keda +``` + +You can view the logs for the keda operator container with the following: + +```sh +kubectl logs -n keda {keda-pod-name} -c keda-operator +``` + +## Reporting issues + +If you are having issues or hitting a potential bug, please file an issue [in the KEDA GitHub repo](https://github.com/kedacore/keda/issues/new/choose) with details, logs, and steps to reproduce the behavior. diff --git a/content/docs/2.3/deploy.md b/content/docs/2.3/deploy.md new file mode 100644 index 000000000..8747afb20 --- /dev/null +++ b/content/docs/2.3/deploy.md @@ -0,0 +1,130 @@ ++++ +title = "Deploying KEDA" ++++ + +We provide a few approaches to deploy KEDA runtime in your Kubernetes clusters: + +- [Helm charts](#helm) +- [Operator Hub](#operatorhub) +- [YAML declarations](#yaml) + +> πŸ’‘ **NOTE:** KEDA requires Kubernetes cluster version 1.16 and higher + +Don't see what you need? Feel free to [create an issue](https://github.com/kedacore/keda/issues/new) on our GitHub repo. + +## Deploying with Helm {#helm} + +### Install + +Deploying KEDA with Helm is very simple: + +1. Add Helm repo + + ```sh + helm repo add kedacore https://kedacore.github.io/charts + ``` + +2. Update Helm repo + + ```sh + helm repo update + ``` + +3. Install `keda` Helm chart + + **Helm 3** + + ```sh + kubectl create namespace keda + helm install keda kedacore/keda --namespace keda + ``` + +### Uninstall + +If you want to remove KEDA from a cluster you can run one of the following: + +**Using Helm 3** + +```sh +helm uninstall -n keda keda +kubectl delete -f https://raw.githubusercontent.com/kedacore/keda/main/config/crd/bases/keda.sh_scaledobjects.yaml +kubectl delete -f https://raw.githubusercontent.com/kedacore/keda/main/config/crd/bases/keda.sh_scaledjobs.yaml +kubectl delete -f https://raw.githubusercontent.com/kedacore/keda/main/config/crd/bases/keda.sh_triggerauthentications.yaml +``` + +## Deploying with Operator Hub {#operatorhub} + +### Install + +1. On Operator Hub Marketplace locate and install KEDA operator to namespace `keda` +2. Create `KedaController` resource named `keda` in namespace `keda` +![Operator Hub installation](https://raw.githubusercontent.com/kedacore/keda-olm-operator/master/images/keda-olm-install.gif) +> πŸ’‘ **NOTE:** Further information on Operator Hub installation method can be found in the following [repository](https://github.com/kedacore/keda-olm-operator). + +### Uninstall + +Locate installed KEDA Operator in `keda` namespace, then remove created `KedaController` resource and uninstall KEDA operator. + +## Deploying using the deployment YAML files {#yaml} + +### Install + +If you want to try KEDA on [Minikube](https://minikube.sigs.k8s.io) or a different Kubernetes deployment without using Helm you can still deploy it with `kubectl`. + +- We provide sample YAML declaration which includes our CRDs and all other resources in a file which is available on the [GitHub releases](https://github.com/kedacore/keda/releases) page. +Run the following command (if needed, replace the version, in this case `2.0.0`, with the one you are using): + +```sh +kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.0.0/keda-2.0.0.yaml +``` + +- Alternatively you can download the file and deploy it from the local path: +```sh +kubectl apply -f keda-2.0.0.yaml +``` + +- You can also find the same YAML declarations in our `/config` directory on our [GitHub repo](https://github.com/kedacore/keda) if you prefer to clone it. + +```sh +git clone https://github.com/kedacore/keda && cd keda + +VERSION=2.0.0 make deploy +``` + +### Uninstall + +- In case of installing from released YAML file just run the following command (if needed, replace the version, in this case `2.0.0`, with the one you are using): + +```sh +kubectl delete -f https://github.com/kedacore/keda/releases/download/v2.0.0/keda-2.0.0.yaml +``` + +- If you have downloaded the file locally, you can run: + +```sh +kubectl delete -f keda-2.0.0.yaml +``` + +- You would need to run these commands from within the directory of the cloned [GitHub repo](https://github.com/kedacore/keda): + +```sh +VERSION=2.0.0 make undeploy +``` + +## Deploying KEDA on MicroK8s {#microk8s} + +### Install + +If you want to try KEDA v2 on [MicroK8s](https://microk8s.io/) from `1.20` channel, KEDA is included into MicroK8s addons. + +```sh +microk8s enable keda +``` + +### Uninstall + +To uninstall KEDA in MicroK8s, simply disable the addon as shown below. + +```sh +microk8s disable keda +``` \ No newline at end of file diff --git a/content/docs/2.3/faq.md b/content/docs/2.3/faq.md new file mode 100644 index 000000000..d012d6767 --- /dev/null +++ b/content/docs/2.3/faq.md @@ -0,0 +1,5 @@ ++++ +title = "FAQ" ++++ + +{{< faq20 >}} diff --git a/content/docs/2.3/migration.md b/content/docs/2.3/migration.md new file mode 100644 index 000000000..5ee8a6455 --- /dev/null +++ b/content/docs/2.3/migration.md @@ -0,0 +1,200 @@ ++++ +title = "Migration Guide" ++++ + +## Migrating from KEDA v1 to v2 + +Please note that you **can not** run both KEDA v1 and v2 on the same Kubernetes cluster. You need to [uninstall](../../1.5/deploy) KEDA v1 first, in order to [install](../deploy) and use KEDA v2. + +> πŸ’‘ **NOTE:** When uninstalling KEDA v1 make sure v1 CRDs are uninstalled from the cluster as well. + +KEDA v2 is using a new API namespace for its Custom Resources Definitions (CRD): `keda.sh` instead of `keda.k8s.io` and introduces a new Custom Resource for scaling of Jobs. See full details on KEDA Custom Resources [here](../concepts/#custom-resources-crd). + +Here's an overview of what's changed: +- [Scaling of Deployments](#scaling-of-deployments) +- [Scaling of Jobs](#scaling-of-jobs) +- [Improved flexibility & usability of trigger metadata](#improved-flexibility--usability-of-trigger-metadata) +- [Scalers](#scalers) +- [TriggerAuthentication](#triggerauthentication) + + +### Scaling of Deployments +In order to scale `Deployments` with KEDA v2, you need to do only a few modifications to existing v1 `ScaledObjects` definitions, so they comply with v2: +- Change the value of `apiVersion` property from `keda.k8s.io/v1alpha1` to `keda.sh/v1alpha1` +- Rename property `spec.scaleTargetRef.deploymentName` to `spec.scaleTargetRef.name` +- Rename property `spec.scaleTargetRef.containerName` to `spec.scaleTargetRef.envSourceContainerName` +- Label `deploymentName` (in `metadata.labels.`) is no longer needed to be specified on v2 ScaledObject (it was mandatory on older versions of v1) + +Please see the examples below or refer to the full [v2 ScaledObject Specification](../concepts/scaling-deployments/#scaledobject-spec) + + **Example of v1 ScaledObject** +```yaml +apiVersion: keda.k8s.io/v1alpha1 +kind: ScaledObject +metadata: + name: {scaled-object-name} + labels: + deploymentName: {deployment-name} +spec: + scaleTargetRef: + deploymentName: {deployment-name} + containerName: {container-name} + pollingInterval: 30 + cooldownPeriod: 300 + minReplicaCount: 0 + maxReplicaCount: 100 + triggers: + # {list of triggers to activate the deployment} +``` + +**Example of v2 ScaledObject** + +```yaml +apiVersion: keda.sh/v1alpha1 # <--- Property value was changed +kind: ScaledObject +metadata: # <--- labels.deploymentName is not needed + name: {scaled-object-name} +spec: + scaleTargetRef: + name: {deployment-name} # <--- Property name was changed + envSourceContainerName: {container-name} # <--- Property name was changed + pollingInterval: 30 + cooldownPeriod: 300 + minReplicaCount: 0 + maxReplicaCount: 100 + triggers: + # {list of triggers to activate the deployment} +``` + +### Scaling of Jobs + +In order to scale `Jobs` with KEDA v2, you need to do only a few modifications to existing v1 `ScaledObjects` definitions, so they comply with v2: +- Change the value of `apiVersion` property from `keda.k8s.io/v1alpha1` to `keda.sh/v1alpha1` +- Change the value of `kind` property from `ScaledObject` to `ScaledJob` +- Remove property `spec.scaleType` +- Remove properties `spec.cooldownPeriod` and `spec.minReplicaCount` + +You can configure `successfulJobsHistoryLimit` and `failedJobsHistoryLimit`. They will remove the old job histories automatically. + +Please see the examples below or refer to the full [v2 ScaledJob Specification](../concepts/scaling-jobs/#scaledjob-spec) + +**Example of v1 ScaledObject for Jobs scaling** +```yaml +apiVersion: keda.k8s.io/v1alpha1 +kind: ScaledObject +metadata: + name: {scaled-object-name} +spec: + scaleType: job + jobTargetRef: + parallelism: 1 + completions: 1 + activeDeadlineSeconds: 600 + backoffLimit: 6 + template: + # {job template} + pollingInterval: 30 + cooldownPeriod: 300 + minReplicaCount: 0 + maxReplicaCount: 100 + triggers: + # {list of triggers to create jobs} +``` + +**Example of v2 ScaledJob** + +```yaml +apiVersion: keda.sh/v1alpha1 # <--- Property value was changed +kind: ScaledJob # <--- Property value was changed +metadata: + name: {scaled-job-name} +spec: # <--- spec.scaleType is not needed + jobTargetRef: + parallelism: 1 + completions: 1 + activeDeadlineSeconds: 600 + backoffLimit: 6 + template: + # {job template} + pollingInterval: 30 # <--- spec.cooldownPeriod and spec.minReplicaCount are not needed + successfulJobsHistoryLimit: 5 # <--- property is added + failedJobsHistoryLimit: 5 # <--- Property is added + maxReplicaCount: 100 + triggers: + # {list of triggers to create jobs} +``` + +### Improved flexibility & usability of trigger metadata + +We've introduced more options to configure trigger metadata to give users more flexibility. + +> πŸ’‘ **NOTE:** Changes only apply to trigger metadata and don't impact usage of `TriggerAuthentication` + +Here's an overview: + +| Scaler | 1.x | 2.0 | +|--------|--------|--------| +| `azure-blob` | `connection` (**Default**: `AzureWebJobsStorage`) | `connectionFromEnv` | +| `azure-monitor` | `activeDirectoryClientId`
`activeDirectoryClientPassword` | `activeDirectoryClientId`
`activeDirectoryClientIdFromEnv`
`activeDirectoryClientPasswordFromEnv` | +| `azure-queue` | `connection` (**Default**: AzureWebJobsStorage) | `connectionFromEnv` | +| `azure-servicebus` | `connection` | `connectionFromEnv` | +| `azure-eventhub` | `storageConnection` (**Default**: `AzureWebJobsStorage`)
`connection` (**Default**: `EventHub`) | `storageConnectionFromEnv`
`connectionFromEnv` | +| `aws-cloudwatch` | `awsAccessKeyID` (**Default**: `AWS_ACCESS_KEY_ID`)
`awsSecretAccessKey` (**Default**: `AWS_SECRET_ACCESS_KEY`) | `awsAccessKeyID`
`awsAccessKeyIDFromEnv`
`awsSecretAccessKeyFromEnv` | +| `aws-kinesis-stream` | `awsAccessKeyID` (**Default**: `AWS_ACCESS_KEY_ID`)
`awsSecretAccessKey` (**Default**: `AWS_SECRET_ACCESS_KEY`) | `awsAccessKeyID`
`awsAccessKeyIDFromEnv`
`awsSecretAccessKeyFromEnv` | +| `aws-sqs-queue` | `awsAccessKeyID` (**Default**: `AWS_ACCESS_KEY_ID`)
`awsSecretAccessKey` (**Default**: `AWS_SECRET_ACCESS_KEY`) | `awsAccessKeyID`
`awsAccessKeyIDFromEnv`
`awsSecretAccessKeyFromEnv` | +| `kafka` | _(none)_ | _(none)_ +| `rabbitmq` | `apiHost`
`host` | ~~`apiHost`~~
`host`
`hostFromEnv` | +| `prometheus` | _(none)_ | _(none)_ | +| `cron` | _(none)_ | _(none)_ | +| `redis` | `address`
`host`
`port`
`password` | `address`
`addressFromEnv`
`host`
`hostFromEnv`
~~`port`~~
`passwordFromEnv` +| `redis-streams` | `address`
`host`
`port`
`password` | `address`
`addressFromEnv`
`host`
`hostFromEnv`
~~`port`~~
`passwordFromEnv` +| `gcp-pubsub` | `credentials` | `credentialsFromEnv` | +| `external` | _(any matching value)_ | _(any matching value with `FromEnv` suffix)_ +| `liiklus` | _(none)_ | _(none)_ | +| `stan` | _(none)_ | _(none)_ | +| `huawei-cloudeye` | | _(none)_ | _(none)_ | +| `postgresql` | `connection`
`password` | `connectionFromEnv`
`passwordFromEnv` | +| `mysql` | `connectionString`
`password` | `connectionStringFromEnv`
`passwordFromEnv` | + +### Scalers + +**Azure Service Bus** + - `queueLength` was renamed to `messageCount` + +**Kafka** + - `authMode` property was replaced with `sasl` and `tls` properties. Please refer [documentation](../scalers/apache-kafka/#authentication-parameters) for Kafka Authentication Parameters details. + +**RabbitMQ** + +In KEDA 2.0 the RabbitMQ scaler has only `host` parameter, and the protocol for communication can be specified by +`protocol` (http or amqp). The default value is `amqp`. The behavior changes only for scalers that were using HTTP +protocol. + +Example of RabbitMQ trigger before 2.0: +```yaml +triggers: +- type: rabbitmq + metadata: + queueLength: '20' + queueName: testqueue + includeUnacked: 'true' + apiHost: 'https://guest:password@localhost:443/vhostname' +``` + +The same trigger in 2.0: +```yaml +triggers: +- type: rabbitmq + metadata: + queueLength: '20' + queueName: testqueue + protocol: 'http' + host: 'https://guest:password@localhost:443/vhostname' +``` + +### TriggerAuthentication +In order to use Authentication via `TriggerAuthentication` with KEDA v2, you need to change: +- Change the value of `apiVersion` property from `keda.k8s.io/v1alpha1` to `keda.sh/v1alpha1` + +For more details please refer to the full +[v2 TriggerAuthentication Specification](../concepts/authentication/#re-use-credentials-and-delegate-auth-with-triggerauthentication) diff --git a/content/docs/2.3/operate/_index.md b/content/docs/2.3/operate/_index.md new file mode 100644 index 000000000..eb7e53f36 --- /dev/null +++ b/content/docs/2.3/operate/_index.md @@ -0,0 +1,11 @@ ++++ +title = "Operate" +description = "Guidance & requirements for operating KEDA" +weight = 1 ++++ + +We provide guidance & requirements around various areas to operate KEDA: + +- Cluster ([link](./cluster)) +- Integrate with Prometheus ([link](./prometheus)) +- Kubernetes Events ([link](./events)) diff --git a/content/docs/2.3/operate/cluster.md b/content/docs/2.3/operate/cluster.md new file mode 100644 index 000000000..b8f356436 --- /dev/null +++ b/content/docs/2.3/operate/cluster.md @@ -0,0 +1,60 @@ ++++ +title = "Cluster" +description = "Guidance & requirements for running KEDA in your cluster" +weight = 100 ++++ + +## Cluster capacity requirements + +The KEDA runtime require the following resources in a production-ready setup: + +| Deployment | CPU | Memory | +| -------------- | ----------------------- | ----------------------------- | +| Metrics Server | Limit: 1, Request: 100m | Limit: 1000Mi, Request: 100Mi | +| Operator | Limit: 1, Request: 100m | Limit: 1000Mi, Request: 100Mi | + +These are used by default when deploying through YAML. + +> πŸ’‘ For more info on CPU and Memory resource units and their meaning, see [this](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes) link. + +## Firewall requirements + +KEDA requires to be accessible inside the cluster to be able to autoscale. + +Here is an overview of the required ports that need to be accessible for KEDA to work: + + +| Port | Why? | Remarks | +| ------ | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `443` | Used by Kubernetes API server to get metrics | Required for all platforms because it uses Control Plane → port 443 on the Service IP range communication.

This is not applicable for Google Cloud. | +| `6443` | Used by Kubernetes API server to get metrics | Only required for Google Cloud because it uses Control Plane → port 6443 on the Pod IP range for communication | + + +## High Availability + +KEDA does not provide support for high-availability due to upstream limitations. + +Here is an overview of all KEDA deployments and the supported replicas: + +| Deployment | Support Replicas | Reasoning | +| -------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------- | +| Metrics Server | 1 | Limitation in [k8s custom metrics server](https://github.com/kubernetes-sigs/custom-metrics-apiserver/issues/70) | +| Operator | 1 | | + +## HTTP Timeouts + +Some scalers issue HTTP requests to external servers (i.e. cloud services). Each applicable scaler uses its own dedicated HTTP client with its own connection pool, and by default each client is set to time out any HTTP request after 3 seconds. + +You can override this default by setting the `KEDA_HTTP_DEFAULT_TIMEOUT` environment variable to your desired timeout in milliseconds. For example, on Linux/Mac/Windows WSL2 operating systems, you'd use this command to set to 1 second: + +```shell +export KEDA_HTTP_DEFAULT_TIMEOUT=1000 +``` + +And on Windows Powershell, you'd use this command: + +```shell +$env:KEDA_HTTP_DEFAULT_TIMEOUT=1000 +``` + +All applicable scalers will use this timeout. Setting a per-scaler timeout is currently unsupported. diff --git a/content/docs/2.3/operate/events.md b/content/docs/2.3/operate/events.md new file mode 100644 index 000000000..adb16b9e4 --- /dev/null +++ b/content/docs/2.3/operate/events.md @@ -0,0 +1,30 @@ ++++ +title = "Events" +description = "Kubernetes Events emitted by KEDA" +weight = 100 ++++ + +## Kubernetes Events emitted by KEDA + +KEDA emits the following [Kubernetes Events](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#event-v1-core): + +| Event | Type | Description | +|-------------------------------------|-----------|-----------------------------------------------------------------------------------------------------------------------------| +| `ScaledObjectReady` | `Normal` | On the first time a ScaledObject is ready, or if the previous ready condition status of the object was `Unknown` or `False` | +| `ScaledJobReady` | `Normal` | On the first time a ScaledJob is ready, or if the previous ready condition status of the object was `Unknown` or `False` | +| `ScaledObjectCheckFailed` | `Warning` | If the check validation for a ScaledObject fails | +| `ScaledJobCheckFailed` | `Warning` | If the check validation for a ScaledJob fails | +| `ScaledObjectDeleted` | `Normal` | When a ScaledObject is deleted and removed from KEDA watch | +| `ScaledJobDeleted` | `Normal` | When a ScaledJob is deleted and removed from KEDA watch | +| `KEDAScalersStarted` | `Normal` | When Scalers watch loop have started for a ScaledObject or ScaledJob | +| `KEDAScalersStopped` | `Normal` | When Scalers watch loop have stopped for a ScaledObject or a ScaledJob | +| `KEDAScalerFailed` | `Warning` | When a Scaler fails to create or check its event source | +| `KEDAScaleTargetActivated` | `Normal` | When the scale target (Deployment, StatefulSet, etc) of a ScaledObject is scaled to 1 | +| `KEDAScaleTargetDeactivated` | `Normal` | When the scale target (Deployment, StatefulSet, etc) of a ScaledObject is scaled to 0 | +| `KEDAScaleTargetActivationFailed` | `Warning` | When KEDA fails to scale the scale target of a ScaledObject to 1 | +| `KEDAScaleTargetDeactivationFailed` | `Warning` | When KEDA fails to scale the scale target of a ScaledObject to 0 | +| `KEDAJobsCreated` | `Normal` | When KEDA creates jobs for a ScaledJob | +| `TriggerAuthenticationAdded` | `Normal` | When a new TriggerAuthentication is added | +| `TriggerAuthenticationDeleted` | `Normal` | When a TriggerAuthentication is deleted | +| `ClusterTriggerAuthenticationAdded` | `Normal` | When a new ClusterTriggerAuthentication is added | +| `ClusterTriggerAuthenticationDeleted`| `Normal` | When a ClusterTriggerAuthentication is deleted | diff --git a/content/docs/2.3/operate/prometheus.md b/content/docs/2.3/operate/prometheus.md new file mode 100644 index 000000000..64c030664 --- /dev/null +++ b/content/docs/2.3/operate/prometheus.md @@ -0,0 +1,17 @@ ++++ +title = "Integrate with Prometheus" +description = "Overview of all Prometheus metrics that KEDA provides" +weight = 200 ++++ + +## Prometheus Exporter Metrics + +The KEDA Metrics Adapter exposes Prometheus metrics which can be scraped on port `9022` (this can be changed by setting the `metrics-port` argument for the Metrics Adapter) at `/metrics`. The metrics collected in the Metrics Adapter are only active when the HPA is active (> 0 replicas). + +The following metrics are being gathered: + +- `keda_metrics_adapter_scaler_error_totals` - The total number of errors encountered for all scalers +- `keda_metrics_adapter_scaled_object_error_totals`- The number of errors that have occurred for each scaled object +- `keda_metrics_adapter_scaler_errors` - The number of errors that have occurred for each scaler +- `keda_metrics_adapter_scaler_metrics_value`- The current value for each scaler's metric that would be used by the HPA in computing the target average. +- `keda_metrics_adapter_scaler_metrics_value`- The current value for each scaler's metric that would be used by the HPA in computing the target average. diff --git a/content/docs/2.3/scalers/_index.md b/content/docs/2.3/scalers/_index.md new file mode 100644 index 000000000..6e7428923 --- /dev/null +++ b/content/docs/2.3/scalers/_index.md @@ -0,0 +1,6 @@ ++++ +title = "Scalers" +weight = 2 ++++ + +KEDA **scalers** can both detect if a deployment should be activated or deactivated, and feed custom metrics for a specific event source. diff --git a/content/docs/2.3/scalers/apache-kafka.md b/content/docs/2.3/scalers/apache-kafka.md new file mode 100644 index 000000000..5e9783c04 --- /dev/null +++ b/content/docs/2.3/scalers/apache-kafka.md @@ -0,0 +1,159 @@ ++++ +title = "Apache Kafka" +layout = "scaler" +availability = "v1.0+" +maintainer = "Microsoft" +description = "Scale applications based on an Apache Kafka topic or other services that support Kafka protocol." +go_file = "kafka_scaler" ++++ + +> **Notice:** +> - By default, the number of replicas will not exceed the number of partitions on a topic. + That is, if `maxReplicaCount` is set more than number of partitions, the scaler won't scale up to target maxReplicaCount. See `allowIdleConsumers` below to disable this default behavior. +> - This is so because if there are more number of consumers than the number of partitions in a topic, then extra consumer will have to sit idle. + +### Trigger Specification + +This specification describes the `kafka` trigger for an Apache Kafka topic. + +```yaml +triggers: +- type: kafka + metadata: + # brokerList: kafka.svc:9092 - deprecated + bootstrapServers: kafka.svc:9092 + consumerGroup: my-group + topic: test-topic + lagThreshold: '5' + offsetResetPolicy: latest + allowIdleConsumers: false +``` + +**Parameter list:** + +- `bootstrapServers` - Comma separated list of Kafka brokers "hostname:port" to connect to for bootstrap. +- `consumerGroup` - Name of the consumer group used for checking the offset on the topic and processing the related lag. +- `topic` - Name of the topic on which processing the offset lag. +- `lagThreshold` - Average target value to trigger scaling actions. (default: 10) +- `offsetResetPolicy` - The offset reset policy for the consumer. Options are `latest` or `earliest` (default: `latest`) +- `allowIdleConsumers` - Optional. When set to `true`, the number of replicas can exceed the number of +partitions on a topic, allowing for idle consumers. (default: `false`) + +### Authentication Parameters + + You can use `TriggerAuthentication` CRD to configure the authenticate by providing `sasl`, `username` and `password`, in case your Kafka cluster has SASL authentication turned on. If TLS is required you should set `tls` to `enable`. If required for your Kafka configuration, you may also provide a `ca`, `cert` and `key`. `cert` and `key` must be specified together. + +**Credential based authentication:** + +**SASL:** + +- `sasl`: Kafka SASL auth mode. Optional. If not set, SASL for Kafka is not used. If set, it must be one of `plaintext`, `scram_sha256` or `scram_sha512`. +- `username`: Optional. If `sasl` is set, this is required. +- `password`: Optional. If `sasl` is set, this is required. + +**TLS:** + +- `tls`: Optional. To enable SSL auth for Kafka, set this to `enable`. If not set, TLS for Kafka is not used. +- `ca`: Certificate authority file for TLS client authentication. Optional. +- `cert`: Certificate for client authentication. Optional. Required if `key` is specified. +- `key`: Key for client authentication. Optional. Required if `cert` is specified. + +### New Consumers and Offset Reset Policy + +When a new Kafka consumer is created, it must determine its consumer group initial position, i.e. the offset it will start to read from. The position is decided in Kafka consumers via a parameter `auto.offset.reset` and the possible values to set are `latest` (Kafka default), and `earliest`. This parameter in Keda should be set accordingly. In this initial status, no offset has been committed to Kafka for the consumer group and any request for offset metadata will return an `INVALID_OFFSET`; so Keda has to manage the consumer pod's autoscaling in relation to the offset reset policy that has been specified in the parameters: + +- If the policy is set to `earliest` (a new consumer wants to replay everything in the topic from its beginning) and no offset is committed, the scaler will return a lag value equal to the last offset in the topic (in the case of a new topic, 0), so it will scale the deployment to 0 replicas. If a new message is produced to the topic, Keda will return the new value of the offset (1), and will scale the deployments to consume the message. +- If the policy is set to `latest` (so the new consumer will only consume new messages) and no offset is committed, the scaler will return a negative lag value, and will also tell the HPA to remain `active`, hence the deployment should have the minimum number of replicas running. This is to allow the consumer to read any new message on the topic, and commit its offset. + +### Example + +Your kafka cluster no SASL/TLS auth: + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: kafka-scaledobject + namespace: default +spec: + scaleTargetRef: + name: azure-functions-deployment + pollingInterval: 30 + triggers: + - type: kafka + metadata: + bootstrapServers: localhost:9092 + consumerGroup: my-group # Make sure that this consumer group name is the same one as the one that is consuming topics + topic: test-topic + # Optional + lagThreshold: "50" + offsetResetPolicy: latest +``` + +Your kafka cluster turn on SASL/TLS auth: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: keda-kafka-secrets + namespace: default +data: + sasl: "plaintext" + username: "admin" + password: "admin" + tls: "enable" + ca: + cert: + key: +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-kafka-credential + namespace: default +spec: + secretTargetRef: + - parameter: sasl + name: keda-kafka-secrets + key: sasl + - parameter: username + name: keda-kafka-secrets + key: username + - parameter: password + name: keda-kafka-secrets + key: password + - parameter: tls + name: keda-kafka-secrets + key: tls + - parameter: ca + name: keda-kafka-secrets + key: ca + - parameter: cert + name: keda-kafka-secrets + key: cert + - parameter: key + name: keda-kafka-secrets + key: key +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: kafka-scaledobject + namespace: default +spec: + scaleTargetRef: + name: azure-functions-deployment + pollingInterval: 30 + triggers: + - type: kafka + metadata: + bootstrapServers: localhost:9092 + consumerGroup: my-group # Make sure that this consumer group name is the same one as the one that is consuming topics + topic: test-topic + # Optional + lagThreshold: "50" + offsetResetPolicy: latest + authenticationRef: + name: keda-trigger-auth-kafka-credential +``` diff --git a/content/docs/2.3/scalers/artemis.md b/content/docs/2.3/scalers/artemis.md new file mode 100644 index 000000000..62e47d7f5 --- /dev/null +++ b/content/docs/2.3/scalers/artemis.md @@ -0,0 +1,95 @@ ++++ +title = "ActiveMQ Artemis" +layout = "scaler" +availability = "v1.5+" +maintainer = "Community" +description = "Scale applications based on ActiveMQ Artemis queues" +go_file = "artemis_scaler" ++++ + +### Trigger Specification + +This specification describes the `artemis-queue` trigger for ActiveMQ Artemis queues. + +```yaml +triggers: +- type: artemis-queue + metadata: + managementEndpoint: "artemis-activemq.artemis:8161" + queueName: "test" + brokerName: "artemis-activemq" + brokerAddress: "test" + queueLength: '10' + username: 'ARTEMIS_USERNAME' + password: 'ARTEMIS_PASSWORD' + restApiTemplate: # Optional. Default : "http://<>/console/jolokia/read/org.apache.activemq.artemis:broker=\"<>\",component=addresses,address=\"<>\",subcomponent=queues,routing-type=\"anycast\",queue=\"<>\"/MessageCount" +``` + +**Parameter list:** + +- `managementEndpoint` - ActiveMQ Artemis management endpoint to connect to in `:` format. +- `queueName` - Name of the queue to check for the number of messages available. +- `brokerName` - Name of the broker as defined in Artemis. +- `brokerAddress` - Address of the broker. +- `queueLength` - Target value for queue length passed to the scaler. Example: if one pod can handle 10 messages, set the queue length target to 10. If the actual number of messages in the queue is 30, the scaler scales to 3 pods. (default: 10) +- `restApiTemplate` - Template to build REST API url to get queue size. + - Default - `"http://<>/console/jolokia/read/org.apache.activemq.artemis:broker=\"<>\",component=addresses,address=\"<>\",subcomponent=queues,routing-type=\"anycast\",queue=\"<>\"/MessageCount"`. In this example, `<>`, `<>`, `<>` and `<>` will be replaced automatically during runtime by values from metadata of YAML definition: `managementEndpoint`, `brokerName`, `brokerAddress`, `queueName`. + +### Authentication Parameters + + You can use `TriggerAuthentication` CRD to configure the `username` and `password` to connect to the management endpoint. + +**Username and Password based authentication:** + +- `username` Required. The username to use to connect to the broker's management endpoint. +- `password` Required. The password to use to connect to the broker's management endpoint. + +### Example + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: kedartemis + namespace: kedartemis + labels: + app: kedartemis +type: Opaque +data: + artemis-password: "YXJ0ZW1pcw==" + artemis-username: "YXJ0ZW1pcw==" +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: trigger-auth-kedartemis + namespace: kedartemis +spec: + secretTargetRef: + - parameter: username + name: kedartemis + key: artemis-username + - parameter: password + name: kedartemis + key: artemis-password +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: kedartemis-consumer-scaled-object + namespace: kedartemis +spec: + scaleTargetRef: + name: kedartemis-consumer + triggers: + - type: artemis-queue + metadata: + managementEndpoint: "artemis-activemq.artemis:8161" + queueName: "test" + queueLength: "50" + brokerName: "artemis-activemq" + brokerAddress: "test" + restApiTemplate: # Optional. Default: "http://<>/console/jolokia/read/org.apache.activemq.artemis:broker=\"<>\",component=addresses,address=\"<>\",subcomponent=queues,routing-type=\"anycast\",queue=\"<>\"/MessageCount" + authenticationRef: + name: trigger-auth-kedartemis +``` \ No newline at end of file diff --git a/content/docs/2.3/scalers/aws-cloudwatch.md b/content/docs/2.3/scalers/aws-cloudwatch.md new file mode 100644 index 000000000..4fa5b1f5c --- /dev/null +++ b/content/docs/2.3/scalers/aws-cloudwatch.md @@ -0,0 +1,121 @@ ++++ +title = "AWS CloudWatch" +layout = "scaler" +availability = "v1.0+" +maintainer = "Community" +description = "Scale applications based on AWS CloudWatch." +go_file = "aws_cloudwatch_scaler" ++++ + +### Trigger Specification + +This specification describes the `aws-cloudwatch` trigger that scales based on a AWS CloudWatch. + +```yaml +triggers: +- type: aws-cloudwatch + metadata: + # Required: namespace + namespace: AWS/SQS + # Required: Dimension Name - Supports specifying multiple dimension names by using ";" as a seperator i.e. dimensionName: QueueName;QueueName + dimensionName: QueueName + # Required: Dimension Value - Supports specifying multiple dimension values by using ";" as a seperator i.e. dimensionValue: queue1;queue2 + dimensionValue: keda + metricName: ApproximateNumberOfMessagesVisible + targetMetricValue: "2" + minMetricValue: "0" + # Required: region + awsRegion: "eu-west-1" + # Optional: AWS Access Key ID, can use TriggerAuthentication as well + awsAccessKeyIDFromEnv: AWS_ACCESS_KEY_ID # default AWS_ACCESS_KEY_ID + # Optional: AWS Secret Access Key, can use TriggerAuthentication as well + awsSecretAccessKeyFromEnv: AWS_SECRET_ACCESS_KEY # default AWS_SECRET_ACCESS_KEY + identityOwner: pod | operator # Optional. Default: pod + # Optional: Default Metrict Collection Time + defaultMetricCollectionTime: 300 # default 300 + # Optional: Default Metric Statistic + defaultMetricStat: "Average" # default "Average" + # Optional: Default Metric Statistic Period + defaultMetricStatPeriod: 300 # default 300 +``` + +**Parameter list:** + +- `identityOwner` - Receive permissions on the CloudWatch via Pod Identity or from the KEDA operator itself (see below). + +> When `identityOwner` set to `operator` - the only requirement is that the Keda operator has the correct IAM permissions on the CloudWatch. Additional Authentication Parameters are not required. + +- `defaultMetricCollectionTime` - How long in the past (seconds) should the scaler check AWS Cloudwatch. Used to define **StartTime** ([official documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html)) + +- `defaultMetricStat` - Which statistics metric is going to be used by the query. Used to define **Stat** ([official documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html)) + +- `defaultMetricStatPeriod` - Which frequency is going to be used by the related query. Used to define **Period** ([official documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html)) + + +### Authentication Parameters + +> These parameters are relevant only when `identityOwner` is set to `pod`. + +You can use `TriggerAuthentication` CRD to configure the authenticate by providing either a role ARN or a set of IAM credentials. + +**Pod identity based authentication:** + +- `podIdentity.provider` needs to be set to either `aws-kiam` or `aws-eks` on the `TriggerAuthentication` and the pod/service account must be configured correctly for your pod identity provider. + +**Role based authentication:** + +- `awsRoleArn` - Amazon Resource Names (ARNs) uniquely identify AWS resource + +**Credential based authentication:** + +- `awsAccessKeyID` - Id of the user +- `awsSecretAccessKey` - Access key for the user to authenticate with + +The user will need access to read data from AWS CloudWatch. + +### Example + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: test-secrets +data: + AWS_ACCESS_KEY_ID: + AWS_SECRET_ACCESS_KEY: +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-aws-credentials + namespace: keda-test +spec: + secretTargetRef: + - parameter: awsAccessKeyID # Required. + name: keda-aws-secrets # Required. + key: AWS_ACCESS_KEY_ID # Required. + - parameter: awsSecretAccessKey # Required. + name: keda-aws-secrets # Required. + key: AWS_SECRET_ACCESS_KEY # Required. +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: aws-cloudwatch-queue-scaledobject + namespace: keda-test +spec: + scaleTargetRef: + name: nginx-deployment + triggers: + - type: aws-cloudwatch + metadata: + namespace: AWS/SQS + dimensionName: QueueName + dimensionValue: keda + metricName: ApproximateNumberOfMessagesVisible + targetMetricValue: "2" + minMetricValue: "0" + awsRegion: "eu-west-1" + authenticationRef: + name: keda-trigger-auth-aws-credentials +``` diff --git a/content/docs/2.3/scalers/aws-kinesis.md b/content/docs/2.3/scalers/aws-kinesis.md new file mode 100644 index 000000000..202d79194 --- /dev/null +++ b/content/docs/2.3/scalers/aws-kinesis.md @@ -0,0 +1,100 @@ ++++ +title = "AWS Kinesis Stream" +layout = "scaler" +availability = "v1.1+" +maintainer = "Community" +description = "Scale applications based on AWS Kinesis Stream." +go_file = "aws_kinesis_stream_scaler" ++++ + +### Trigger Specification + +This specification describes the `aws-kinesis-stream` trigger that scales based on the shard count of AWS Kinesis Stream. + +```yaml +triggers: +- type: aws-kinesis-stream + metadata: + # Required + streamName: myKinesisStream + # Required + awsRegion: "eu-west-1" + # Optional: Default: 2 + shardCount: "2" + identityOwner: pod | operator # Optional. Default: pod +``` + +**Parameter list:** + +- `streamName` - Name of AWS Kinesis Stream +- `shardCount` - The target value that a Kinesis data streams consumer can handle. +- `awsRegion` - AWS Region for the Kinesis Stream +- `identityOwner` - Receive permissions on the Kinesis Stream via Pod Identity or from the KEDA operator itself (see below). + +> When `identityOwner` set to `operator` - the only requirement is that the Keda operator has the correct IAM permissions on the Kinesis Stream. Additional Authentication Parameters are not required. + +### Authentication Parameters + +> These parameters are relevant only when `identityOwner` is set to `pod`. + +You can use `TriggerAuthentication` CRD to configure the authenticate by providing either a role ARN or a set of IAM credentials, or use other [KEDA supported authentication methods](https://keda.sh/concepts/authentication). + +#### Delegate auth with TriggerAuthentication + +**Role based authentication:** + +- `awsRoleArn` - Amazon Resource Names (ARNs) uniquely identify AWS resource + +**Credential based authentication:** + +- `awsAccessKeyID` - Id of the user +- `awsSecretAccessKey` - Access key for the user to authenticate with + +The user will need `DescribeStreamSummary` IAM permission policy to read data from AWS Kinesis Streams. + +### Example + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: test-secrets + namespace: keda-test +data: + AWS_ACCESS_KEY_ID: + AWS_SECRET_ACCESS_KEY: +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-aws-credentials + namespace: keda-test +spec: + secretTargetRef: + - parameter: awsAccessKeyID # Required. + name: test-secrets # Required. + key: AWS_ACCESS_KEY_ID # Required. + - parameter: awsSecretAccessKey # Required. + name: test-secrets # Required. + key: AWS_SECRET_ACCESS_KEY # Required. +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: aws-kinesis-stream-scaledobject + namespace: keda-test +spec: + scaleTargetRef: + name: nginx-deployment + triggers: + - type: aws-kinesis-stream + authenticationRef: + name: keda-trigger-auth-aws-credential + metadata: + # Required + streamName: myKinesisStream + # Required + awsRegion: "eu-west-1" + # Optional: Default: 2 + shardCount: "2" +``` diff --git a/content/docs/2.3/scalers/aws-sqs.md b/content/docs/2.3/scalers/aws-sqs.md new file mode 100644 index 000000000..b52f77ead --- /dev/null +++ b/content/docs/2.3/scalers/aws-sqs.md @@ -0,0 +1,100 @@ ++++ +title = "AWS SQS Queue" +layout = "scaler" +availability = "v1.0+" +maintainer = "Community" +description = "Scale applications based on AWS SQS Queue." +go_file = "aws_sqs_queue_scaler" ++++ + +### Trigger Specification + +This specification describes the `aws-sqs-queue` trigger that scales based on an AWS SQS Queue. + +```yaml +triggers: +- type: aws-sqs-queue + metadata: + # Required: queueURL + queueURL: https://sqs.eu-west-1.amazonaws.com/account_id/QueueName + queueLength: "5" # Default: "5" + # Required: awsRegion + awsRegion: "eu-west-1" + identityOwner: pod | operator # Optional. Default: pod +``` + +**Parameter list:** + +- `queueURL` - Full URL for the SQS Queue +- `queueLength` - Target value for queue length passed to the scaler. Example: if one pod can handle 10 messages, set the queue length target to 10. If the actual messages in the SQS Queue is 30, the scaler scales to 3 pods. (default: 5) + +> For the purposes of scaling, "actual messages" is equal to `ApproximateNumberOfMessages` + `ApproximateNumberOfMessagesNotVislble`, since `NotVisible` in SQS terms means the message is still in-flight/processing. + +- `awsRegion` - AWS Region for the SQS Queue +- `identityOwner` - Receive permissions on the SQS Queue via Pod Identity or from the KEDA operator itself (see below). + +> When `identityOwner` set to `operator` - the only requirement is that the Keda operator has the correct IAM permissions on the SQS queue. Additional Authentication Parameters are not required. + +### Authentication Parameters + +> These parameters are relevant only when `identityOwner` is set to `pod`. + +You can use `TriggerAuthentication` CRD to configure the authenticate by providing either a role ARN or a set of IAM credentials. + +**Pod identity based authentication:** + +- `podIdentity.provider` needs to be set to either `aws-kiam` or `aws-eks` on the `TriggerAuthentication` and the pod/service account must be configured correctly for your pod identity provider. + +**Role based authentication:** + +- `awsRoleArn` - Amazon Resource Names (ARNs) uniquely identify AWS resource + +**Credential based authentication:** + +- `awsAccessKeyID` - Id of the user +- `awsSecretAccessKey` - Access key for the user to authenticate with + +The user will need access to read properties from the specified AWS SQS queue. + +### Example + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: test-secrets +data: + AWS_ACCESS_KEY_ID: + AWS_SECRET_ACCESS_KEY: +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-aws-credentials + namespace: keda-test +spec: + secretTargetRef: + - parameter: awsAccessKeyID # Required. + name: test-secrets # Required. + key: AWS_ACCESS_KEY_ID # Required. + - parameter: awsSecretAccessKey # Required. + name: test-secrets # Required. + key: AWS_SECRET_ACCESS_KEY # Required. +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: aws-sqs-queue-scaledobject + namespace: keda-test +spec: + scaleTargetRef: + name: nginx-deployment + triggers: + - type: aws-sqs-queue + authenticationRef: + name: keda-trigger-auth-aws-credentials + metadata: + queueURL: myQueue + queueLength: "5" + awsRegion: "eu-west-1" +``` diff --git a/content/docs/2.3/scalers/azure-event-hub.md b/content/docs/2.3/scalers/azure-event-hub.md new file mode 100644 index 000000000..0192e1394 --- /dev/null +++ b/content/docs/2.3/scalers/azure-event-hub.md @@ -0,0 +1,79 @@ ++++ +title = "Azure Event Hubs" +layout = "scaler" +availability = "v1.0+" +maintainer = "Community" +description = "Scale applications based on Azure Event Hubs." +notice = "As of now, the Event Hub scaler only supports reading checkpoints from Blob Storage, as well as scaling only Event Hub applications written in C#, Java, Python or created with Azure Functions." +go_file = "azure_eventhub_scaler" ++++ + +### Trigger Specification + +This specification describes the `azure-eventhub` trigger for Azure Event Hubs. + +```yaml +triggers: +- type: azure-eventhub + metadata: + connectionFromEnv: EVENTHUB_CONNECTIONSTRING_ENV_NAME + storageConnectionFromEnv: STORAGE_CONNECTIONSTRING_ENV_NAME + consumerGroup: $Default + unprocessedEventThreshold: '64' + blobContainer: 'name_of_container' +``` + +**Parameter list:** + +- `connectionFromEnv` - Name of the environment variable your deployment uses to get the connection string appended with `EntityPath=` +- `storageConnectionFromEnv` - Name of the environment variable that provides connection string for Azure Storage Account to store checkpoint. As of now the Event Hub scaler only reads from Azure Blob Storage. +- `consumerGroup` - Consumer group of event hub consumer. (default: `$default`) +- `unprocessedEventThreshold` - : Average target value to trigger scaling actions. (default: 64) +- `blobContainer` - Container name to store checkpoint. This is needed when a using an Event Hub application written in dotnet or java, and not an Azure function + +### Authentication Parameters + +The common way of authenticating to Azure Event Hub is by using the connection string. However, you can use [Pod Identity](https://azure.github.io/aad-pod-identity/docs/demo/standard_walkthrough/) if you host your cluster in Azure AKS, and if have configured it to support Pod Identity. + +To use Pod Identity, you have to add a [TriggerAuthentication](https://keda.sh/docs/2.0/concepts/authentication/#re-use-credentials-and-delegate-auth-with-triggerauthentication) and configure it to use Pod Identity like so: + +``` +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: nameOfTriggerAuth + namespace: default +spec: + podIdentity: + provider: Azure +``` + +When you do so, the Event Hub scaler will depend on the existence of two configurations you have to provide: `eventHubNamespace` and `eventHubName`. + + +### Example + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: azure-eventhub-scaledobject + namespace: default +spec: + scaleTargetRef: + name: azureeventhub-function + triggers: + - type: azure-eventhub + metadata: + # Required + storageConnectionFromEnv: AzureWebJobsStorage + # Required if not using Pod Identity + connectionFromEnv: EventHub + # Required if using Pod Identity + eventHubNamespace: AzureEventHubNameSpace + eventHubName: NameOfTheEventHub +# Optional + consumerGroup: $Default # default: $Default + unprocessedEventThreshold: '64' # default 64 events. + blobContainer: ehcontainer +``` diff --git a/content/docs/2.3/scalers/azure-log-analytics.md b/content/docs/2.3/scalers/azure-log-analytics.md new file mode 100644 index 000000000..bc844327f --- /dev/null +++ b/content/docs/2.3/scalers/azure-log-analytics.md @@ -0,0 +1,362 @@ ++++ +title = "Azure Log Analytics" +layout = "scaler" +availability = "v2.0+" +maintainer = "Community" +description = "Scale applications based on Azure Log Analytics query result" +go_file = "azure_log_analytics_scaler" ++++ + +### Trigger Specification + +This specification describes the `azure-log-analytics` trigger for Azure Log Analytics query result. Here is an example of providing values in metadata: + +```yaml +triggers: + - type: azure-log-analytics + metadata: + tenantId: "AZURE_AD_TENANT_ID" + clientId: "SERVICE_PRINCIPAL_CLIENT_ID" + clientSecret: "SERVICE_PRINCIPAL_PASSWORD" + workspaceId: "LOG_ANALYTICS_WORKSPACE_ID" + query: | + let AppName = "web"; + let ClusterName = "demo-cluster"; + let AvgDuration = ago(10m); + let ThresholdCoefficient = 0.8; + Perf + | where InstanceName contains AppName + | where InstanceName contains ClusterName + | where CounterName == "cpuUsageNanoCores" + | where TimeGenerated > AvgDuration + | extend AppName = substring(InstanceName, indexof((InstanceName), "/", 0, -1, 10) + 1) + | summarize MetricValue=round(avg(CounterValue)) by CounterName, AppName + | join (Perf + | where InstanceName contains AppName + | where InstanceName contains ClusterName + | where CounterName == "cpuLimitNanoCores" + | where TimeGenerated > AvgDuration + | extend AppName = substring(InstanceName, indexof((InstanceName), "/", 0, -1, 10) + 1) + | summarize arg_max(TimeGenerated, *) by AppName, CounterName + | project Limit = CounterValue, TimeGenerated, CounterPath, AppName) + on AppName + | project MetricValue, Threshold = Limit * ThresholdCoefficient + threshold: "1900000000" + # Alternatively, you can use existing environment variables to read configuration from: + # See details in "Parameter list" section + workspaceIdFromEnv: LOG_ANALYTICS_WORKSPACE_ID_ENV_NAME # Optional. You can use this instead of `workspaceId` parameter. + clientIdFromEnv: SERVICE_PRINCIPAL_CLIENT_ID_ENV_NAME # Optional. You can use this instead of `clientId` parameter. + tenantIdFromEnv: AZURE_AD_TENANT_ID_ENV_NAME # Optional. You can use this instead of `tenantId` parameter. + clientSecretFromEnv: SERVICE_PRINCIPAL_PASSWORD_ENV_NAME # Optional. You can use this instead of `clientSecret` parameter. +``` + +**Parameter list:** + +- `tenantId` - Id of the Azure Active Directory tenant. Follow [this](https://docs.microsoft.com/en-us/cli/azure/account?view=azure-cli-latest#az-account-show) link to retrieve your tenant id. +- `clientId` - Id of the application from your Azure AD Application/service principal. Follow [this](https://docs.microsoft.com/en-us/cli/azure/ad/sp?view=azure-cli-latest) link to create your service principal. +- `clientSecret` - Password from your Azure AD Application/service principal. +- `workspaceId` - Id of Log Analytics workspace. Follow [this](https://docs.microsoft.com/en-us/cli/azure/monitor/log-analytics/workspace?view=azure-cli-latest#az-monitor-log-analytics-workspace-list) link to get your Log Analytics workspace id. +- `query` - Log Analytics [kusto](https://docs.microsoft.com/en-us/azure/azure-monitor/log-query/get-started-queries) query, JSON escaped. You can use [this](https://www.freeformatter.com/json-escape.html) tool to convert your query from Log Analytics query editor to JSON escaped string, and then review YAML specific escapes. +- `threshold` - Value that is used as a threshold to calculate # of pods for scale target. + +The authentication parameters could be provided using environmental variables, instead of setting them directly in metadata. Here is a list of parameters you can use to retrieve values from environment variables: + +- `tenantIdFromEnv` optional: An environmental variable name, that stores Azure Active Directory tenant id. Follow [this](https://docs.microsoft.com/en-us/cli/azure/account?view=azure-cli-latest#az-account-show) link to retrieve your tenant id. +- `clientIdFromEnv` optional: An environmental variable name, that stores Application id from your Azure AD Application/service principal. Follow [this](https://docs.microsoft.com/en-us/cli/azure/ad/sp?view=azure-cli-latest) link to create your service principal. +- `clientSecretFromEnv` optional: An environmental variable name, that stores password from your Azure AD Application/service principal. +- `workspaceIdFromEnv` optional: An environmental variable name, that stores your Log Analytics workspace id. Follow [this](https://docs.microsoft.com/en-us/cli/azure/monitor/log-analytics/workspace?view=azure-cli-latest#az-monitor-log-analytics-workspace-list) link to get your Log Analytics workspace id. + +### Query Guidance + +It is important to design your query to return 1 table with 1 row. A good practice is to add "| limit 1" at the end of your query. + +Scaler will take value from: + +- 1st cell as Metrics Value. +- 2d cell as Threshold (optional). + +You can define threshold in trigger metadata, it will be used if your query results only 1 cell, that will be interpreted as metric value. Be aware, even if you have defined threshold in metadata, it can be overwritten by your query. + +Data types of your query result should be: real, int or long. Other data types are not supported. Later, during runtime, your data will be converted to int64. + +Be careful with setting up "pollingInterval" and long running queries. Test your query before. + +Example query to get `MetricValue` and `Threshold` based on CPU usage and limits, defined for the pod. + +```kusto +let AppName = "web"; +let ClusterName = "demo-cluster"; +let AvgDuration = ago(10m); +let ThresholdCoefficient = 0.8; +Perf +| where InstanceName contains AppName +| where InstanceName contains ClusterName +| where CounterName == "cpuUsageNanoCores" +| where TimeGenerated > AvgDuration +| extend AppName = substring(InstanceName, indexof((InstanceName), "/", 0, -1, 10) + 1) +| summarize MetricValue=round(avg(CounterValue)) by CounterName, AppName +| join (Perf + | where InstanceName contains AppName + | where InstanceName contains ClusterName + | where CounterName == "cpuLimitNanoCores" + | where TimeGenerated > AvgDuration + | extend AppName = substring(InstanceName, indexof((InstanceName), "/", 0, -1, 10) + 1) + | summarize arg_max(TimeGenerated, *) by AppName, CounterName + | project Limit = CounterValue, TimeGenerated, CounterPath, AppName) + on AppName +| project MetricValue, Threshold = Limit * ThresholdCoefficient +``` + +Example result: + +![Azure Log Analytics query example](/img/azure-log-analytics-scaler-query-example.png) + +### Scaler Limitations + +- As it was mentioned before, you can define a threshold using query (2d cell of query result will be interpret as threshold). Be aware! Threshold from query result will be set only once, during scaler creation. So, if your query will return different threshold values during runtime, they will not be propagated to Horizontal Pod Autoscaler target. + +### Authentication Parameters + + You can use `TriggerAuthentication` CRD to configure the authentication by providing a set of Azure Active Directory credentials and resource identifiers. + +**Service Principal based authentication:** + +- `tenantId`: Azure Active Directory tenant id. Follow [this](https://docs.microsoft.com/en-us/cli/azure/account?view=azure-cli-latest#az-account-show) link to retrieve your tenant id. +- `clientId`: Application id from your Azure AD Application/service principal. Follow [this](https://docs.microsoft.com/en-us/cli/azure/ad/sp?view=azure-cli-latest) link to create your service principal. +- `clientSecret`: Password from your Azure AD Application/service principal. +- `workspaceId`: Your Log Analytics workspace id. Follow [this](https://docs.microsoft.com/en-us/cli/azure/monitor/log-analytics/workspace?view=azure-cli-latest#az-monitor-log-analytics-workspace-list) link to get your Log Analytics workspace id. + +**Managed identity based authentication:** + +You can use managed identity to request access token for Log Analytics API. The advantage of this approach is that there is no need to store secrets in Kubernetes. Read [more](https://docs.microsoft.com/en-us/azure/aks/use-managed-identity) about managed identities in Azure Kubernetes Service. + +### Example + +#### Service Principal based authentication + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: kedaloganalytics + namespace: kedaloganalytics + labels: + app: kedaloganalytics +type: Opaque +data: + tenantId: "QVpVUkVfQURfVEVOQU5UX0lE" #Base64 encoded Azure Active Directory tenant id + clientId: "U0VSVklDRV9QUklOQ0lQQUxfQ0xJRU5UX0lE" #Base64 encoded Application id from your Azure AD Application/service principal + clientSecret: "U0VSVklDRV9QUklOQ0lQQUxfUEFTU1dPUkQ=" #Base64 encoded Password from your Azure AD Application/service principal + workspaceId: "TE9HX0FOQUxZVElDU19XT1JLU1BBQ0VfSUQ=" #Base64 encoded Log Analytics workspace id +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: trigger-auth-kedaloganalytics + namespace: kedaloganalytics +spec: + secretTargetRef: + - parameter: tenantId + name: kedaloganalytics + key: tenantId + - parameter: clientId + name: kedaloganalytics + key: clientId + - parameter: clientSecret + name: kedaloganalytics + key: clientSecret + - parameter: workspaceId + name: kedaloganalytics + key: workspaceId +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: kedaloganalytics-consumer-scaled-object + namespace: kedaloganalytics + labels: + deploymentName: kedaloganalytics-consumer +spec: + scaleTargetRef: + kind: #Optional: Default: Deployment, Available Options: ReplicaSet, Deployment, DaemonSet, StatefulSet + name: kedaloganalytics-consumer + pollingInterval: 30 + cooldownPeriod: 30 + minReplicaCount: 1 + maxReplicaCount: 10 + triggers: + - type: azure-log-analytics + metadata: + query: | + let AppName = "web"; + let ClusterName = "demo-cluster"; + let AvgDuration = ago(10m); + let ThresholdCoefficient = 0.8; + Perf + | where InstanceName contains AppName + | where InstanceName contains ClusterName + | where CounterName == "cpuUsageNanoCores" + | where TimeGenerated > AvgDuration + | extend AppName = substring(InstanceName, indexof((InstanceName), "/", 0, -1, 10) + 1) + | summarize MetricValue=round(avg(CounterValue)) by CounterName, AppName + | join (Perf + | where InstanceName contains AppName + | where InstanceName contains ClusterName + | where CounterName == "cpuLimitNanoCores" + | where TimeGenerated > AvgDuration + | extend AppName = substring(InstanceName, indexof((InstanceName), "/", 0, -1, 10) + 1) + | summarize arg_max(TimeGenerated, *) by AppName, CounterName + | project Limit = CounterValue, TimeGenerated, CounterPath, AppName) + on AppName + | project MetricValue, Threshold = Limit * ThresholdCoefficient + threshold: "1900000000" + authenticationRef: + name: trigger-auth-kedaloganalytics +``` + +#### Managed identity based authentication + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: trigger-auth-kedaloganalytics + namespace: kedaloganalytics +spec: + podIdentity: + provider: azure +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: kedaloganalytics-consumer-scaled-object + namespace: kedaloganalytics + labels: + deploymentName: kedaloganalytics-consumer +spec: + scaleTargetRef: + kind: #Optional: Default: Deployment, Available Options: ReplicaSet, Deployment, DaemonSet, StatefulSet + name: kedaloganalytics-consumer + pollingInterval: 30 + cooldownPeriod: 30 + minReplicaCount: 1 + maxReplicaCount: 10 + triggers: + - type: azure-log-analytics + metadata: + workspaceId: "81963c40-af2e-47cd-8e72-3002e08aa2af" + query: | + let AppName = "web"; + let ClusterName = "demo-cluster"; + let AvgDuration = ago(10m); + let ThresholdCoefficient = 0.8; + Perf + | where InstanceName contains AppName + | where InstanceName contains ClusterName + | where CounterName == "cpuUsageNanoCores" + | where TimeGenerated > AvgDuration + | extend AppName = substring(InstanceName, indexof((InstanceName), "/", 0, -1, 10) + 1) + | summarize MetricValue=round(avg(CounterValue)) by CounterName, AppName + | join (Perf + | where InstanceName contains AppName + | where InstanceName contains ClusterName + | where CounterName == "cpuLimitNanoCores" + | where TimeGenerated > AvgDuration + | extend AppName = substring(InstanceName, indexof((InstanceName), "/", 0, -1, 10) + 1) + | summarize arg_max(TimeGenerated, *) by AppName, CounterName + | project Limit = CounterValue, TimeGenerated, CounterPath, AppName) + on AppName + | project MetricValue, Threshold = Limit * ThresholdCoefficient + threshold: "1900000000" + authenticationRef: + name: trigger-auth-kedaloganalytics +``` + +### Guides + +#### Enabling managed identity authentication for Log Analytics scaler + +Use the following commands to create user defined identity, role assignment to Azure Log Analytics and deploy\update Keda: + +```sh +export SUBSCRIPTION_ID="" +export RESOURCE_GROUP="" +export CLUSTER_NAME="" +export CLUSTER_LOCATION="" # "westeurope", "northeurope"... +export IDENTITY_NAME="" #Any name +export LOG_ANALYTICS_RESOURCE_ID="" + +# Login to Azure, set subscription, get AKS credentials +az login +az account set -s "${SUBSCRIPTION_ID}" +az aks get-credentials -n ${CLUSTER_NAME} -g ${RESOURCE_GROUP} + +# ------- Cluster preparation. Run this block only once for fresh cluster. +# Clone repo and run initial role assignment +git clone https://github.com/Azure/aad-pod-identity.git +./aad-pod-identity/hack/role-assignment.sh + +#Deploy aad-pod-identity using Helm 3 +helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts +helm repo update +helm install aad-pod-identity aad-pod-identity/aad-pod-identity --namespace=kube-system +# ------------------------------------------------------------------------------------------- + +#Create identity +az identity create -g ${RESOURCE_GROUP} -n ${IDENTITY_NAME} +export IDENTITY_CLIENT_ID="$(az identity show -g ${RESOURCE_GROUP} -n ${IDENTITY_NAME} --query clientId -otsv)" +export IDENTITY_RESOURCE_ID="$(az identity show -g ${RESOURCE_GROUP} -n ${IDENTITY_NAME} --query id -otsv)" + +#Assign reader permissions for your identity to Log Analytics workspace +#WARNING: It can take some time while identity will be provisioned. +#If you see an error: "Principal SOME_ID does not exist in the directory SOME_ID", just wait couple of minutes and then retry. +az role assignment create --role "Log Analytics Reader" --assignee ${IDENTITY_CLIENT_ID} --scope ${LOG_ANALYTICS_RESOURCE_ID} + +# Allow cluster to control identity created earlier. +ID="$(az aks show -g ${RESOURCE_GROUP} -n ${CLUSTER_NAME} --query servicePrincipalProfile.clientId -otsv)" +if [[ "${ID:-}" == "msi" ]]; then + ID="$(az aks show -g ${RESOURCE_GROUP} -n ${CLUSTER_NAME} --query identityProfile.kubeletidentity.clientId -otsv)" +fi +az role assignment create --role "Managed Identity Operator" --assignee "${ID}" --scope "${IDENTITY_RESOURCE_ID}" + +# Create AzureIdentity and AzureIdentityBinding +cat < # Optional, can use TriggerAuthentication as well + activeDirectoryClientIdFromEnv: CLIENT_ID_ENV_NAME # Optional, can use TriggerAuthentication as well + activeDirectoryClientPasswordFromEnv: CLIENT_PASSWORD_ENV_NAME # Optional, can use TriggerAuthentication as well +``` + +**Parameter list:** + +- `resourceURI` - Shortened URI to the Azure resource with format `"//"`. +- `tenantId` - Id of the tenant that contains the Azure resource. This is used for authentication. +- `subscriptionId` - Id of Azure subscription that contains the Azure resource. This is used for determining the full resource URI. +- `resourceGroupName` - Name of the resource group for the Azure resource. +- `metricName` - Name of the metric which can be found in the [official documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported). +- `targetValue` - Target value to trigger scaling actions. +- `metricAggregationType` - Aggregation method of the Azure Monitor metric. Optionsinclude `Average`, `Total`, `Maximum` with a full list in the [official documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported). +- `metricFilter` - Name of the filter to be more specific by using dimensions listed in the [official documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported). (optional) +- `metricAggregationInterval` - Collection time of the metric in format `"hh:mm:ss"` (default: `"0:5:0"` which is 5 minutes) +- `activeDirectoryClientId` - Id of the Active Directory application which requires at least `Monitoring Reader` permissions. Optional. Required when `TriggerAuthentication` is not provided. +- `activeDirectoryClientPasswordFromEnv` - Name of the environment variable that contains the active directory client password.Optional. Required when `TriggerAuthentication` is not provided. + +Some parameters could be provided using environmental variables, instead of setting them directly in metadata. Here is a list of parameters you can use to retrieve values from environment variables: + +- `activeDirectoryClientIdFromEnv` - Id of the Active Directory application which requires at least `Monitoring Reader` permissions, similar to `activeDirectoryClientId`, but reads it from an environment variable on the scale target. Optional. Required when `TriggerAuthentication` is not provided. + +### Authentication Parameters + +You can use `TriggerAuthentication` CRD to configure the authentication by providing a set of Azure Active Directory credentials or by using pod identity. + +**Credential based authentication:** + +- `activeDirectoryClientId` - Id of the Active Directory application which requires at least `Monitoring Reader` permissions +- `activeDirectoryClientPassword` - Password of the Active Directory application + +The user will need access to read data from the Azure resource. + +### Example + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: azure-monitor-secrets +data: + activeDirectoryClientId: + activeDirectoryClientPassword: +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: azure-monitor-trigger-auth +spec: + secretTargetRef: + - parameter: activeDirectoryClientId + name: azure-monitor-secrets + key: activeDirectoryClientId + - parameter: activeDirectoryClientPassword + name: azure-monitor-secrets + key: activeDirectoryClientPassword + # or Pod Identity, kind: Secret is not required in case of pod Identity + podIdentity: + provider: azure +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: azure-monitor-scaler +spec: + scaleTargetRef: + name: azure-monitor-example + minReplicaCount: 1 + maxReplicaCount: 10 + triggers: + - type: azure-monitor + metadata: + resourceURI: Microsoft.ContainerService/managedClusters/azureMonitorCluster + tenantId: xxx-xxx-xxx-xxx-xxx + subscriptionId: yyy-yyy-yyy-yyy-yyy + resourceGroupName: azureMonitor + metricName: kube_pod_status_ready + metricFilter: namespace eq 'default' + metricAggregationInterval: "0:1:0" + metricAggregationType: Average + targetValue: "1" + authenticationRef: + name: azure-monitor-trigger-auth +``` diff --git a/content/docs/2.3/scalers/azure-service-bus.md b/content/docs/2.3/scalers/azure-service-bus.md new file mode 100644 index 000000000..874d5ab2b --- /dev/null +++ b/content/docs/2.3/scalers/azure-service-bus.md @@ -0,0 +1,85 @@ ++++ +title = "Azure Service Bus" +layout = "scaler" +maintainer = "Microsoft" +description = "Scale applications based on Azure Service Bus Queues or Topics." +availability = "v1.0+" +go_file = "azure_servicebus_scaler" ++++ + +### Trigger Specification + +This specification describes the `azure-servicebus` trigger for Azure Service Bus Queue or Topic. + +```yaml +triggers: +- type: azure-servicebus + metadata: + # Required: queueName OR topicName and subscriptionName + queueName: functions-sbqueue + # or + topicName: functions-sbtopic + subscriptionName: sbtopic-sub1 + # Optional, required when pod identity is used + namespace: service-bus-namespace + # Optional, can use TriggerAuthentication as well + connectionFromEnv: SERVICEBUS_CONNECTIONSTRING_ENV_NAME # This must be a connection string for a queue itself, and not a namespace level (e.g. RootAccessPolicy) connection string [#215](https://github.com/kedacore/keda/issues/215) + # Optional + messageCount: "5" # Optional. Count of messages to trigger scaling on. Default: 5 messages +``` + +**Parameter list:** + +- `messageCount` - Amount of active messages in your Azure Service Bus queue or topic to scale on. +- `queueName` - Name of the Azure Service Bus queue to scale on. (optional) +- `topicName` - Name of the Azure Service Bus topic to scale on. (optional) +- `subscriptionName` - Name of the Azure Service Bus queue to scale on. (optional, required when `topicName` is specified) +- `namespace` - Name of the Azure Service Bus namespace that contains your queue or topic. (optional, required when pod identity is used) +- `connectionFromEnv` - Name of the environment variable your deployment uses to get the connection string of the Azure Service Bus namespace. (optional, can use TriggerAuthentication as well) + +> πŸ’‘ **NOTE:** Service Bus Shared Access Policy needs to be of type `Manage`. Manage access is required for KEDA to be able to get metrics from Service Bus. + +### Authentication Parameters + +You can authenticate by using pod identity or connection string authentication. + +**Connection String Authentication:** + +- `connection` - Connection string for Azure Service Bus Namespace + +### Example + +Here is an example of how to use managed identity: + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: azure-servicebus-auth +spec: + podIdentity: + provider: azure +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: azure-servicebus-queue-scaledobject + namespace: default +spec: + scaleTargetRef: + name: azure-servicebus-queue-function + triggers: + - type: azure-servicebus + metadata: + # Required: queueName OR topicName and subscriptionName + queueName: functions-sbqueue + # or + topicName: functions-sbtopic + subscriptionName: sbtopic-sub1 + # Required: Define what Azure Service Bus to authenticate to with Managed Identity + namespace: service-bus-namespace + # Optional + messageCount: "5" # default 5 + authenticationRef: + name: azure-servicebus-auth # authenticationRef would need either podIdentity or define a connection parameter +``` diff --git a/content/docs/2.3/scalers/azure-storage-blob.md b/content/docs/2.3/scalers/azure-storage-blob.md new file mode 100644 index 000000000..273210436 --- /dev/null +++ b/content/docs/2.3/scalers/azure-storage-blob.md @@ -0,0 +1,72 @@ ++++ +title = "Azure Blob Storage" +layout = "scaler" +availability = "v1.1+" +maintainer = "Community" +description = "Scale applications based on the count of blobs in a given Azure Blob Storage container." +notice = "As of now, this Azure Blob Storage scaler scales based on the count of the blobs in a container as opposed to the Azure Functions behavior where code is only triggered on new blobs." +go_file = "azure_blob_scaler" ++++ + +### Trigger Specification + +This specification describes the `azure-blob` trigger for Azure Blob Storage. It scales based on the count of blobs in a given blob storage container and assumes the worker is responsible for clearing the container by deleting/moving the blobs once the blob processing completed. + +```yaml +triggers: +- type: azure-blob + metadata: + blobContainerName: functions-blob + blobCount: '5' + connectionFromEnv: STORAGE_CONNECTIONSTRING_ENV_NAME + blobPrefix: myprefix + blobDelimiter: /example +``` + +**Parameter list:** + +- `blobContainerName` - Name of container in an Azure Storage account +- `blobCount` - Average target value to trigger scaling actions. (default: 5) +- `connectionFromEnv` - Name of the environment variable your deployment uses to get the connection string. +- `blobPrefix` - Prefix for the Blob. Use this to specifiy sub path for the blobs if required. (default: `""`) +- `blobDelimiter` - Delimiter for identifying the blob prefix. (default: `/`) + +### Authentication Parameters + +You can authenticate by using pod identity or connection string authentication. + +**Connection String Authentication:** + +- `connection` - Connection string for Azure Storage Account + +### Example + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: azure-blob-auth +spec: + podIdentity: + provider: azure +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: azure-blob-scaledobject + namespace: default +spec: + scaleTargetRef: + name: azureblob-function + triggers: + - type: azure-blob + metadata: + # Required + blobContainerName: functionsblob + # Optional + blobCount: "5" # default 5 + blobPrefix: blobsubpath # Default : "" + blobDelimiter: "/" # Default: "/" + authenticationRef: + name: azure-blob-auth # authenticationRef would need either podIdentity or define a connection parameter +``` diff --git a/content/docs/2.3/scalers/azure-storage-queue.md b/content/docs/2.3/scalers/azure-storage-queue.md new file mode 100644 index 000000000..eb0d7ff5d --- /dev/null +++ b/content/docs/2.3/scalers/azure-storage-queue.md @@ -0,0 +1,69 @@ ++++ +title = "Azure Storage Queue" +layout = "scaler" +availability = "v1.0+" +maintainer = "Microsoft" +description = "Scale applications based on Azure Storage Queues." +go_file = "azure_queue_scaler" ++++ + +### Trigger Specification + +This specification describes the `azure-queue` trigger for Azure Storage Queue. + +```yaml +triggers: +- type: azure-queue + metadata: + queueName: orders + queueLength: '5' + connectionFromEnv: STORAGE_CONNECTIONSTRING_ENV_NAME +``` + +**Parameter list:** + +- `queueName` - Name of the queue. +- `queueLength` - Target value for queue length passed to the scaler. Example: if one pod can handle 10 messages, set the queue length target to 10. If the actual number of messages in the queue is 30, the scaler scales to 3 pods. (default: 5) +- `connectionFromEnv` - Name of the environment variable your deployment uses to get the connection string. + +### Authentication Parameters + +You can authenticate by using pod identity or connection string authentication. + +**Connection String Authentication:** + +- `connection` - Connection string for Azure Storage Account + +### Example + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: azure-queue-auth +spec: + podIdentity: + provider: azure +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: azure-queue-scaledobject + namespace: default +spec: + scaleTargetRef: + name: azurequeue-function + triggers: + - type: azure-queue + metadata: + # Required + queueName: functionsqueue + # Required: connection OR authenticationRef that defines connection + connectionFromEnv: STORAGE_CONNECTIONSTRING_ENV_NAME # Default: AzureWebJobsStorage. Reference to a connection string in deployment + # or authenticationRef as defined below + # + # Optional + queueLength: "5" # default 5 + authenticationRef: + name: azure-queue-auth # authenticationRef would need either podIdentity or define a connection parameter +``` diff --git a/content/docs/2.3/scalers/cpu.md b/content/docs/2.3/scalers/cpu.md new file mode 100644 index 000000000..98c0fd36c --- /dev/null +++ b/content/docs/2.3/scalers/cpu.md @@ -0,0 +1,51 @@ ++++ +title = "CPU" +layout = "scaler" +availability = "v2.0+" +maintainer = "Community" +description = "Scale applications based on cpu metrics." +go_file = "cpu_memory_scaler" ++++ + +> **Notice:** +> - This scaler will never scale to 0 and even when user defines multiple scaler types (eg. Kafka + cpu/memory, or Prometheus + cpu/memory), the deployment will never scale to 0 +> - This scaler only applies to ScaledObject, not to Scaling Jobs. + +### Trigger Specification + +This specification describes the `cpu` trigger that scales based on cpu metrics. + +```yaml +triggers: +- type: cpu + metadata: + # Required + type: Value/ Utilization/ AverageValue + value: "60" +``` + +**Parameter list:** + +- `type` - Type of metric to use. Options are `Utilization`, `Value`, or `AverageValue` +- `value` - Value to trigger scaling actions for + - When using `Value`, the target value is the value of the metric itself (as a quantity) + - When using `Utilization`, the target value is the average of the resource metric across all relevant pods, represented as a percentage of the requested value of the resource for the pods. + - When using `AverageValue`, the target value is the target value of the average of the metric across all relevant pods (quantity). + +### Example + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: cpu-scaledobject + namespace: default +spec: + scaleTargetRef: + name: my-deployment + triggers: + - type: cpu + metadata: + type: Utilization + value: "50" +``` diff --git a/content/docs/2.3/scalers/cron.md b/content/docs/2.3/scalers/cron.md new file mode 100644 index 000000000..ddb8484dd --- /dev/null +++ b/content/docs/2.3/scalers/cron.md @@ -0,0 +1,50 @@ ++++ +title = "Cron" +layout = "scaler" +availability = "v1.5+" +maintainer = "Community" +description = "Scale applications based on a cron schedule." +go_file = "cron_scaler" ++++ + +### Trigger Specification + +This specification describes the `cron` trigger that scales based on a Cron Schedule. + +```yaml +triggers: +- type: cron + metadata: + # Required + timezone: Asia/Kolkata # The acceptable values would be a value from the IANA Time Zone Database. + start: 30 * * * * # Every hour on the 30th minute + end: 45 * * * * # Every hour on the 45th minute + desiredReplicas: "10" +``` + +**Parameter list:** + +- `timezone` - One of the acceptable values from the IANA Time Zone Database. The list of timezones can be found [here](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) +- `start` - Cron expression indicating the start of the cron schedule. +- `end` - Cron expression indicating the end of the cron schedule. +- `desiredReplicas` - Number of replicas to which the resource has to be scaled between the start and end of the cron schedule. + +### Example + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: cron-scaledobject + namespace: default +spec: + scaleTargetRef: + name: my-deployment + triggers: + - type: cron + metadata: + timezone: Asia/Kolkata + start: 30 * * * * + end: 45 * * * * + desiredReplicas: "10" +``` diff --git a/content/docs/2.3/scalers/external-push.md b/content/docs/2.3/scalers/external-push.md new file mode 100644 index 000000000..eec1d0369 --- /dev/null +++ b/content/docs/2.3/scalers/external-push.md @@ -0,0 +1,51 @@ ++++ +title = "External Push" +layout = "scaler" +availability = "v2.0+" +maintainer = "Microsoft" +description = "Scale applications based on an external push scaler." +go_file = "external_scaler" ++++ + +### Trigger Specification + +This specification describes the `external-push` trigger for an external push scaler. + +```yaml +triggers: +- type: external-push + metadata: + scalerAddress: external-scaler-service:8080 + tlsCertFile: /path/to/tls/cert.pem # optional +``` + +**Parameter list:** + +- `scalerAddress`: Address of the external push scaler implementing `ExternalScaler.StreamIsActive` in externalscaler.proto. Format must be `host:port`. +- `tlsCertFile`: Location of a certificate to use for the GRPC connection to authenticate with. (optional) + +The entire metadata object is passed to the external scaler in `ScaledObjectRef.scalerMetadata` + +> For implementing an external scaler, refer to [External Scalers Concept](../concepts/external-scalers.md) + +### Authentication Parameters + +Not supported. + +### Example + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: name + namespace: namespace +spec: + scaleTargetRef: + name: keda-node + triggers: + - type: external-push + metadata: + scalerAddress: external-scaler-service:8080 + tlsCertFile: /path/to/tls/cert.pem # optional +``` diff --git a/content/docs/2.3/scalers/external.md b/content/docs/2.3/scalers/external.md new file mode 100644 index 000000000..4a68b81ec --- /dev/null +++ b/content/docs/2.3/scalers/external.md @@ -0,0 +1,52 @@ ++++ +title = "External" +layout = "scaler" +availability = "v1.0+" +maintainer = "Microsoft" +description = "Scale applications based on an external scaler." +go_file = "external_scaler" ++++ + +### Trigger Specification + +This specification describes the `external` trigger for an external scaler. + +```yaml +triggers: +- type: external + metadata: + scalerAddress: external-scaler-service:8080 + tlsCertFile : /path/to/tls/cert.pem +``` + +**Parameter list:** + +- `scalerAddress` - Address of the external scaler. Format must be `host:port`. +- `tlsCertFile` - Location of a certificate to use for the GRPC connection to authenticate with. (optional) + +> For implementing an external scaler, refer to [External Scalers Concept](../concepts/external-scalers.md) + +### Authentication Parameters + +Not supported yet. + +### Example + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: redis-scaledobject + namespace: keda-redis-test +spec: + scaleTargetRef: + name: keda-redis-node + triggers: + - type: external + metadata: + scalerAddress: redis-external-scaler-service:8080 + address: REDIS_HOST + password: REDIS_PASSWORD + listName: mylist + listLength: "5" +``` diff --git a/content/docs/2.3/scalers/gcp-pub-sub.md b/content/docs/2.3/scalers/gcp-pub-sub.md new file mode 100644 index 000000000..fef6c2607 --- /dev/null +++ b/content/docs/2.3/scalers/gcp-pub-sub.md @@ -0,0 +1,84 @@ ++++ +title = "Google Cloud Platformβ€Ž Pub/Sub" +layout = "scaler" +availability = "v1.0+" +maintainer = "Community" +description = "Scale applications based on Google Cloud Platformβ€Ž Pub/Sub." +go_file = "gcp_pub_sub_scaler" ++++ + +### Trigger Specification + +This specification describes the `gcp-pubsub` trigger for Google Cloud Platformβ€Ž Pub/Sub. + +```yaml +triggers: +- type: gcp-pubsub + metadata: + subscriptionSize: "5" # Optional - Default is 5 + subscriptionName: "mysubscription" # Required + credentialsFromEnv: GOOGLE_APPLICATION_CREDENTIALS_JSON # Required +``` + +The Google Cloud Platformβ€Ž (GCP) Pub/Sub trigger allows you to scale based on the number of messages in your Pub/Sub subscription. + +The `credentialsFromEnv` property maps to the name of an environment variable in the scale target (`scaleTargetRef`) that contains the service account credentials (JSON). KEDA will use those to connect to Google Cloud Platform and collect the required stack driver metrics in order to read the number of messages in the Pub/Sub subscription. + +`subscriptionName` defines the subscription that should be monitored. The `subscriptionSize` determines the target average which the deployment will be scaled on. The default `subscriptionSize` is 5. + +Here's an [example](https://github.com/kedacore/sample-go-gcppubsub). + +### Authentication Parameters +You can use `TriggerAuthentication` CRD to configure the authenticate by providing the service account credentials in JSON. + + +**Credential based authentication:** + +- `GoogleApplicationCredentials` - Service account credentials in JSON + +### Example + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: pubsub-scaledobject + namespace: keda-pubsub-test +spec: + scaleTargetRef: + name: keda-pubsub-go + triggers: + - type: gcp-pubsub + metadata: + subscriptionSize: "5" + subscriptionName: "mysubscription" # Required + credentialsFromEnv: GOOGLE_APPLICATION_CREDENTIALS_JSON # Required +``` + +### Example using TriggerAuthentication + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-gcp-credentials +spec: + secretTargetRef: + - parameter: GoogleApplicationCredentials + name: pubsub-secret # Required. Refers to the name of the secret + key: GOOGLE_APPLICATION_CREDENTIALS_JSON # Required. +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: pubsub-scaledobject +spec: + scaleTargetRef: + name: keda-pubsub-go + triggers: + - type: gcp-pubsub + authenticationRef: + name: keda-trigger-auth-gcp-credentials + metadata: + subscriptionName: "input" # Required +``` diff --git a/content/docs/2.3/scalers/huawei-cloudeye.md b/content/docs/2.3/scalers/huawei-cloudeye.md new file mode 100644 index 000000000..9b4782a8f --- /dev/null +++ b/content/docs/2.3/scalers/huawei-cloudeye.md @@ -0,0 +1,122 @@ ++++ +title = "Huawei Cloudeye" +layout = "scaler" +availability = "v1.1+" +maintainer = "Community" +description = "Scale applications based on a Huawei Cloudeye." +go_file = "huawei_cloudeye_scaler" ++++ + +### Trigger Specification + +This specification describes the `huawei-cloudeye` trigger that scales based on a Huawei Cloudeye. + +```yaml +triggers: +- type: huawei-cloudeye + metadata: + namespace: SYS.ELB + metricName: mb_l7_qps + dimensionName: lbaas_instance_id + dimensionValue: 5e052238-0346-xxb0-86ea-92d9f33e29d2 + targetMetricValue: "100" + minMetricValue: "1" +``` + +**Parameter list:** + +- `namespace` - Namespace of the metric.The format is service.item; service and item must be strings, must start with a letter, can only contain 0-9/a-z/A-Z/_, the total length of service.item is 3, the maximum is 32. +- `metricName` - Name of the metric. +- `dimensionName` - Name of the metric dimension +- `dimensionValue` - Value of the metric dimension. +- `targetMetricValue` - Target value for your metric. +- `minMetricValue` - Minimum value for your metric. If the actual value of the metric you get from cloudeye is less than the minimum value, then the scaler is not active. +- `metricCollectionTime` - Collection time of the metric in seconds. Equivalent to the earliest start time of the end time. (default: 300) +- `metricFilter` - Aggregation method of the metric. Options are `average`, `max`, `min`, `sum` and `variance`. (default: average) +- `metricPeriod` - Granularity of the metric in seconds. (default: 300) + +### Authentication Parameters + +You can use `TriggerAuthentication` CRD to configure the authenticate by providing a set of IAM credentials. + +**Credential based authentication:** + +- `IdentityEndpoint` - Identity Endpoint +- `ProjectID` - Project ID +- `DomainID` - Id of domain +- `Domain` - Domain +- `Region` - Region +- `Cloud` - Cloud name. The fefault value is myhuaweicloud.com +- `AccessKey` - Id of the user +- `SecretKey` - Access key for the user to authenticate with + +The user will need access to read data from Huawei Cloudeye. + +### Example + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: keda-huawei-secrets + namespace: keda-test +data: + IdentityEndpoint: + ProjectID: + DomainID: + Region: + Domain: + AccessKey: + SecretKey: +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-huawei-credential + namespace: keda-test +spec: + secretTargetRef: + - parameter: IdentityEndpoint # Required. + name: keda-huawei-secrets # Required. + key: IdentityEndpoint # Required. + - parameter: ProjectID # Required. + name: keda-huawei-secrets # Required. + key: ProjectID # Required. + - parameter: DomainID # Required. + name: keda-huawei-secrets # Required. + key: DomainID # Required. + - parameter: Region # Required. + name: keda-huawei-secrets # Required. + key: Region # Required. + - parameter: Domain # Required. + name: keda-huawei-secrets # Required. + key: Domain # Required. + - parameter: AccessKey # Required. + name: keda-huawei-secrets # Required. + key: AccessKey # Required. + - parameter: SecretKey # Required. + name: keda-huawei-secrets # Required. + key: SecretKey # Required. +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: huawei-cloudeye-scaledobject + namespace: keda-test +spec: + scaleTargetRef: + name: nginx-deployment + maxReplicaCount: 5 + minReplicaCount: 2 + triggers: + - type: huawei-cloudeye + metadata: + namespace: SYS.ELB + dimensionName: lbaas_instance_id + dimensionValue: 5e052238-0346-47b0-xxea-92d9f33e29d2 + metricName: mb_l7_qps + targetMetricValue: "100" + minMetricValue: "1" + authenticationRef: + name: keda-trigger-auth-huawei-credential +``` diff --git a/content/docs/2.3/scalers/ibm-mq.md b/content/docs/2.3/scalers/ibm-mq.md new file mode 100644 index 000000000..eac6a7de0 --- /dev/null +++ b/content/docs/2.3/scalers/ibm-mq.md @@ -0,0 +1,98 @@ ++++ +title = "IBM MQ" +layout = "scaler" +availability = "v2.0+" +maintainer = "Community" +description = "Scale applications based on IBM MQ Queue" +go_file = "ibmmq_scaler" ++++ + +### Trigger Specification + +This specification describes the `ibmmq` trigger for IBM MQ Queue. + +```yaml +triggers: +- type: ibmmq + metadata: + host: # REQUIRED - IBM MQ Queue Manager Admin REST Endpoint + queueManager: # REQUIRED - Your queue manager + queueName: # REQUIRED - Your queue name + tlsDisabled: # OPTIONAL - Set 'true' to disable TLS. Default: false + queueDepth: # OPTIONAL - Queue depth target for HPA. Default: 5 messages + usernameFromEnv: # Optional: Provide admin username from env instead of as a secret + passwordFromEnv: # Optional: Provide admin password from env instead of as a secret +``` + +**Parameter list:** + +- `host`: REQUIRED - IBM MQ Queue Manager Admin REST Endpoint. Example URI endpoint structure on IBM cloud `https://example.mq.appdomain.cloud/ibmmq/rest/v2/admin/action/qmgr/QM/mqsc` +- `queueManager`: REQUIRED - Name of the queue manager from which messages will be consumed +- `queueName`: REQUIRED - Name of the Queue within the Queue Manager defined from which messages will be consumed +- `tlsDisabled`: OPTIONAL - A boolean: Can be set to 'true' to disable TLS. False by default. +- `queueDepth`: OPTIONAL - Queue depth Target for HPA. Will be set to Default Value of 5 if not Provided. +- `usernameFromEnv`: OPTIONAL: Provide admin username from env instead of as a secret +- `passwordFromEnv`: OPTIONAL: Provide admin password from env instead of as a secret + +### Authentication Parameters + +TriggerAuthentication CRD is used to connect and authenticate to IBM MQ: + +**Authentication Parameters** + +- `ADMIN_USER`: REQUIRED - The admin REST endpoint username for your MQ Queue Manager +- `ADMIN_PASSWORD`: REQUIRED - The admin REST endpoint API key for your MQ Queue Manager +- `usernameFromEnv`: OPTIONAL: Provide admin username from env instead of as a secret +- `passwordFromEnv`: OPTIONAL: Provide admin password from env instead of as a secret + +### Example + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: keda-ibmmq-secret +data: + ADMIN_USER: # REQUIRED - Admin Username + ADMIN_PASSWORD: # REQUIRED - Admin Password +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: ibmmq-scaledobject + namespace: default + labels: + deploymentName: ibmmq-deployment +spec: + scaleTargetRef: + name: ibmmq-deployment + pollingInterval: 5 # OPTIONAL - Default: 30 seconds + cooldownPeriod: 30 # OPTIONAL - Default: 300 seconds + maxReplicaCount: 18 # OPTIONAL - Default: 100 + triggers: + - type: ibmmq + metadata: + host: # REQUIRED - IBM MQ Queue Manager Admin REST Endpoint + queueManager: # REQUIRED - Your queue manager + queueName: # REQUIRED - Your queue name + tlsDisabled: # OPTIONAL - Set 'true' to disable TLS. Default: false + queueDepth: # OPTIONAL - Queue depth target for HPA. Default: 5 messages + usernameFromEnv: # Optional: Provide admin username from env instead of as a secret + passwordFromEnv: # Optional: Provide admin password from env instead of as a secret + authenticationRef: + name: keda-ibmmq-trigger-auth +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-ibmmq-trigger-auth + namespace: default +spec: + secretTargetRef: + - parameter: username + name: keda-ibmmq-secret + key: ADMIN_USER + - parameter: password + name: keda-ibmmq-secret + key: ADMIN_PASSWORD +``` diff --git a/content/docs/2.3/scalers/influxdb.md b/content/docs/2.3/scalers/influxdb.md new file mode 100644 index 000000000..813e7f58d --- /dev/null +++ b/content/docs/2.3/scalers/influxdb.md @@ -0,0 +1,74 @@ ++++ +title = "InfluxDB" +layout = "scaler" +availability = "v2.1+" +maintainer = "Community" +description = "Scale applications based on InfluxDB queries" +go_file = "influxdb_scaler" ++++ + +### Trigger Specification + +This specification describes the `influxdb` trigger that scales based on the results of a InfluxDB query. + +```yaml +triggers: + - type: influxdb + metadata: + serverURL: http://influxdb:8086 + organizationName: influx-org + organizationNameFromEnv: INFLUXDB_ORG_NAME # Optional: You can use this instead of `organizationName` parameter. See details in "Parameter List" section + thresholdValue: '4' + query: | + from(bucket: "bucket-of-interest") + |> range(start: -12h) + |> filter(fn: (r) => r._measurement == "stat") + authToken: some-auth-token + authTokenFromEnv: INFLUXDB_AUTH_TOKEN # Optional: You can use this instead of `authToken` parameter. See details in "Parameter List" section + metricName: influx-metric # Optional: This value will default to a masked version of the url and organization name if not set by the user (metrics name value would be then `influxdb-https---xxx-influx_org`) +``` + +**Parameter list:** + +- `authToken` field holds the authentication token needed for the InfluxDB client to communicate with an associated server. +- `authTokenFromEnv` defines the authorization token, similar to `authToken`, but reads it from an environment variable on the scale target. +- `organizationName` is the organization name needed for the client to locate all information contained in that [organization](https://docs.influxdata.com/influxdb/v2.0/organizations/) such as buckets, tasks, etc. +- `organizationNameFromEnv` defines the organization name, similar to `organizationName`, but reads it from an environment variable on the scale target. +- `serverURL` holds the url value of the InfluxDB server. +- `thresholdValue` is provided by the user. This value can vary from use case to use case depending on the data of interest, and is needed to trigger the scaling in/out depending on what value comes back from the query. +- `query` is the flux query that will yield the value for the scaler to compare the `thresholdValue` against. +- `metricName` an optional name to assign to the metric. If not set KEDA will generate a name based on masked version of the server url and organization name. If using more than one trigger it is required that all `metricName`(s) be unique. + +### Authentication Parameters + +You can authenticate by using an authorization token. + +**Authorization Token Authentication:** + +- `authToken` - Authorization token for InfluxDB server. + +### Example + +Below is an example of how to deploy a scaled object with the `InfluxDB` scale trigger + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: influxdb-scaledobject + namespace: my-project +spec: + scaleTargetRef: + name: nginx-worker + triggers: + - type: influxdb + metadata: + serverURL: http://influxdb:8086 + organizationNameFromEnv: INFLUXDB_ORG_NAME + thresholdValue: '4' + query: | + from(bucket: "bucket-of-interest") + |> range(start: -12h) + |> filter(fn: (r) => r._measurement == "stat") + authTokenFromEnv: INFLUXDB_AUTH_TOKEN +``` diff --git a/content/docs/2.3/scalers/liiklus-topic.md b/content/docs/2.3/scalers/liiklus-topic.md new file mode 100644 index 000000000..cd5392e85 --- /dev/null +++ b/content/docs/2.3/scalers/liiklus-topic.md @@ -0,0 +1,60 @@ ++++ +title = "Liiklus Topic" +layout = "scaler" +availability = "v1.0+" +maintainer = "Community" +description = "Scale applications based on Liiklus Topic." +go_file = "liiklus_scaler" ++++ + +### Trigger Specification + +This specification describes the `liiklus` trigger for Liiklus Topic. + +```yaml +triggers: +- type: liiklus + metadata: + # Required + address: localhost:6565 + group: my-group + topic: test-topic + # Optional + lagThreshold: "50" + groupVersion: 1 +``` + +**Parameter list:** + +- `address` - Address of the gRPC liiklus API endpoint +- `group` - Name of consumer group +- `topic` - Topic to monitor and scale based on `lagThreshold` +- `lagThreshold` - Value to trigger scaling actions for (default: 10) +- `groupVersion` - Version of the group to consider when looking at messages. See [docs](https://github.com/bsideup/liiklus/blob/22efb7049ebcdd0dcf6f7f5735cdb5af1ae014de/app/src/test/java/com/github/bsideup/liiklus/GroupVersionTest.java) (default: 0) + +### Authentication Parameters + +Not supported yet. + +### Example + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: liiklus-scaledobject + namespace: default +spec: + scaleTargetRef: + name: function-deployment + pollingInterval: 30 + triggers: + - type: liiklus + metadata: + # Required + address: localhost:6565 + group: my-group # Make sure that this consumer group name is the same one as the one that is consuming topics + topic: test-topic + # Optional + lagThreshold: "50" # Default value is set to 10 +``` diff --git a/content/docs/2.3/scalers/memory.md b/content/docs/2.3/scalers/memory.md new file mode 100644 index 000000000..2d110f158 --- /dev/null +++ b/content/docs/2.3/scalers/memory.md @@ -0,0 +1,51 @@ ++++ +title = "Memory" +layout = "scaler" +availability = "v2.0+" +maintainer = "Community" +description = "Scale applications based on memory metrics." +go_file = "cpu_memory_scaler" ++++ + +> **Notice:** +> - This scaler will never scale to 0 and even when user defines multiple scaler types (eg. Kafka + cpu/memory, or Prometheus + cpu/memory), the deployment will never scale to 0 +> - This scaler only applies to ScaledObject, not to Scaling Jobs. + +### Trigger Specification + +This specification describes the `memory` trigger that scales based on memory metrics. + +```yaml +triggers: +- type: memory + metadata: + # Required + type: Value/ Utilization/ AverageValue + value: "60" +``` + +**Parameter list:** + +- `type` - Type of metric to use. Options are `Utilization`, `Value`, or `AverageValue` +- `value` - Value to trigger scaling actions for + - When using `Value`, the target value is the value of the metric itself (as a quantity) + - When using `Utilization`, the target value is the average of the resource metric across all relevant pods, represented as a percentage of the requested value of the resource for the pods. + - When using `AverageValue`, the target value is the target value of the average of the metric across all relevant pods (quantity). + +### Example + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: memory-scaledobject + namespace: default +spec: + scaleTargetRef: + name: my-deployment + triggers: + - type: memory + metadata: + type: Utilization + value: "50" +``` diff --git a/content/docs/2.3/scalers/metrics-api.md b/content/docs/2.3/scalers/metrics-api.md new file mode 100644 index 000000000..848db1db6 --- /dev/null +++ b/content/docs/2.3/scalers/metrics-api.md @@ -0,0 +1,253 @@ ++++ +title = "Metrics API" +layout = "scaler" +availability = "v2.0+" +maintainer = "Community" +description = "Scale applications based on a metric provided by an API" +go_file = "metrics_api_scaler" ++++ + +### Trigger Specification + +This specification describes the `metrics-api` trigger that scales based on a metric value provided by an API. + +This scaler allows users to utilize **any existing APIs** as a metric provider. + +Here is an example of trigger configuration using metrics-api scaler: + +```yaml +triggers: +- type: metrics-api + metadata: + targetValue: "8" + url: "http://api:3232/api/v1/stats" + valueLocation: "components.worker.tasks" +``` + +**Parameter list:** + +- `url`: Full URL of the API operation to call to get the metric value (eg. `http://app:1317/api/v1/stats`). +- `valueLocation`: [GJSON path notation](https://github.com/tidwall/gjson#path-syntax) to refer to the field in + the payload containing the metric value +- `targetValue`: Target value to scale on. When the metric provided by the API is equal or higher to this value, + KEDA will start scaling out. When the metric is 0 or less, KEDA will scale down to 0. + +### Authentication Parameters + +Metrics Scaler API supported three types of authentication - API Key based authentication, basic authentication and TLS +authentication. + +You can use `TriggerAuthentication` CRD to configure the authentication. Specify `authMode` and other trigger parameters + along with secret credentials in `TriggerAuthentication` as mentioned below: + +**API Key based authentication:** +- `authMode`: It must be set to `apiKey` in case of API key Authentication. Specify this in trigger configuration. +- `method`: This specifies the possible methods API Key based authentication supports. +Possible values are `header` and `query`. `header` is the default method. Specify this in trigger configuration. +- `keyParamName`: This is either header key or query param used for passing apikey. +Default header is `X-API-KEY` and default query param is `api_key`. Specify this in trigger configuration. +If your implementation has different key, please specify it here. +- `apiKey`: API Key needed for authentication. + +**Basic authentication:** +- `authMode`: It must be set to `basic` in case of Basic Authentication. Specify this in trigger configuration. +- `username`: This is a required field. Provide the username to be used for basic authentication. +- `password`: Provide the password to be used for authentication. For convenience, this has been marked optional, +because many applications implement basic auth with a username as apikey and password as empty. + +**TLS authentication:** +- `authMode`: It must be set to `tls` in case of TLS Authentication. Specify this in trigger configuration. +- `ca`: Certificate authority file for TLS client authentication. This is a required field. +- `cert`: Certificate for client authentication. This is a required field. +- `key`: Key for client authentication. Optional. This is a required field. + +### Example + +Here is a full example of scaled object definition using Metric API trigger: + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: http-scaledobject + namespace: keda + labels: + deploymentName: dummy +spec: + maxReplicaCount: 12 + scaleTargetRef: + name: dummy + triggers: + - type: metrics-api + metadata: + targetValue: "7" + url: "http://api:3232/components/stats" + valueLocation: 'components.worker.tasks' +``` + +When checking current metric Metrics API scaler sends GET request to provided `url` and then uses `valueLocation` +to access the value in response's payload. + +The above example expects that the API endpoint will return response similar to this one: +```json +{ + "components": { + "worker": { + "tasks": 12, + ... + }, + ... + }, + ... +} +``` +Assuming such response, Metrics API trigger will figure out that current metric value is 12. + +> πŸ’‘ **NOTE:**The value of the metric can either be an integral unquoted json number type (e.g. 123). Or a quantity (e.g. "123", "1.23", "10Mi"). + +Here is an example of a metric scaler with API Key based authentication, + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: keda-metric-api-secret + namespace: default +data: + apiKey: "APIKEY" +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-metric-api-creds + namespace: default +spec: + secretTargetRef: + - parameter: apiKey + name: keda-metric-api-secret + key: apiKey +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: http-scaledobject + namespace: keda + labels: + deploymentName: dummy +spec: + maxReplicaCount: 12 + scaleTargetRef: + name: dummy + triggers: + - type: metrics-api + metadata: + targetValue: "7" + url: "http://api:3232/components/stats" + valueLocation: 'components.worker.tasks' + authMode: "apiKey" + method: "query" + keyParamName: "QUERY_KEY" + authenticationRef: + name: keda-metric-api-creds +``` + +Here is an example of a metric scaler with Basic Authentication, define the `Secret` and `TriggerAuthentication` as follows + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: keda-metric-api-secret + namespace: default +data: + username: "username" + password: "password" +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-metric-api-creds + namespace: default +spec: + secretTargetRef: + - parameter: username + name: keda-metric-api-secret + key: username + - parameter: password + name: keda-metric-api-secret + key: password +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: http-scaledobject + namespace: keda + labels: + deploymentName: dummy +spec: + maxReplicaCount: 12 + scaleTargetRef: + name: dummy + triggers: + - type: metrics-api + metadata: + targetValue: "7" + url: "http://api:3232/components/stats" + valueLocation: 'components.worker.tasks' + authMode: "basic" + authenticationRef: + name: keda-metric-api-creds +``` + + +Here is an example of a metric scaler with TLS Authentication, define the `Secret` and `TriggerAuthentication` as follows + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: keda-metric-api-secret + namespace: default +data: + cert: "cert" + key: "key" + ca: "ca" +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-metric-api-creds + namespace: default +spec: + secretTargetRef: + - parameter: cert + name: keda-metric-api-secret + key: cert + - parameter: key + name: keda-metric-api-secret + key: key + - parameter: ca + name: keda-metric-api-secret + key: ca +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: http-scaledobject + namespace: keda + labels: + deploymentName: dummy +spec: + maxReplicaCount: 12 + scaleTargetRef: + name: dummy + triggers: + - type: metrics-api + metadata: + targetValue: "7" + url: "http://api:3232/components/stats" + valueLocation: 'components.worker.tasks' + authMode: "tls" + authenticationRef: + name: keda-metric-api-creds +``` diff --git a/content/docs/2.3/scalers/mongodb.md b/content/docs/2.3/scalers/mongodb.md new file mode 100644 index 000000000..e5b2b325c --- /dev/null +++ b/content/docs/2.3/scalers/mongodb.md @@ -0,0 +1,116 @@ ++++ +title = "MongoDB" +layout = "scaler" +maintainer = "Community" +description = "Scale applications based on MongoDB queries." +availability = "v2.1+" +go_file = "mongo_scaler" ++++ + +### Trigger Specification + +This specification describes the `mongodb` trigger that scales based on result of MongoDB query. + +```yaml +triggers: + - type: mongodb + metadata: + # Required: database name + dbName: "test" + # Required: collection name + collection: "test_collection" + # Required: query expr, used by filter data + query: '{"region":"eu-1","state":"running","plan":"planA"}' + # Required: according to the number of query result, to scale job + queryValue: "1" + # Optional: This value will default to a masked version of the host and collection name if not set by the user (metrics name value would be then `mongodb-https---xxx-test_collection`) + metricName: "global-metric" +``` + +**Parameter list:** + +- `dbName` - Name of the database +- `collection` - Name of the collection +- `query` - A MongoDB query that should return single numeric value +- `queryValue` - A threshold that will define when scaling should occur +- `metricName` - An optional name to assign to the metric. If not set KEDA will generate a name based on masked version of the server hostname and collection name. If using more than one trigger it is required that all `metricName`(s) be unique. + +To provide information about how to connect to MongoDB you can provide + +- `connectionStringFromEnv` - A MongoDB connection string that should point to environment variable with valid value + +Or provide more detailed information: + +- `host` - The host of the MongoDB server +- `port` - The port of the MongoDB server +- `username` - Username to authenticate with to MongoDB database +- `passwordFromEnv` - Password for the given user + +### Authentication Parameters + +You can authenticate by using connection string or password authentication. + +**Connection String Authentication:** + +- `connectionString` - Connection string for MongoDB database + +**Password Authentication:** + +- `password` - Password for the configured user to login to MongoDB database variables. + +### Example + +Here is an example of how to deploy a scaled Job with the `MongoDB` scale trigger which uses `TriggerAuthentication`. + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledJob +metadata: + name: mongodb-job +spec: + jobTargetRef: + template: + spec: + containers: + - name: mongodb-update + image: 1314520999/mongodb-update:latest + args: + - --connectStr=mongodb://test_user:test_password@mongoDB-svc.mongoDB.svc.cluster.local:27017/test + - --dataBase=test + - --collection=test_collection + imagePullPolicy: IfNotPresent + restartPolicy: Never + backoffLimit: 1 + pollingInterval: 30 # Optional. Default: 30 seconds + maxReplicaCount: 30 # Optional. Default: 100 + successfulJobsHistoryLimit: 0 # Optional. Default: 100. How many completed jobs should be kept. + failedJobsHistoryLimit: 10 # Optional. Default: 100. How many failed jobs should be kept. + triggers: + - type: mongodb + metadata: + dbName: "test" + collection: "test_collection" + query: '{"region":"eu-1","state":"running","plan":"planA"}' + queryValue: "1" + metricName: "global-metric" + authenticationRef: + name: mongodb-trigger +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: mongodb-trigger +spec: + secretTargetRef: + - parameter: connectionString + name: mongodb-secret + key: connect +--- +apiVersion: v1 +kind: Secret +metadata: + name: mongodb-secret +type: Opaque +data: + connect: bW9uZ29kYjovL3Rlc3RfdXNlcjp0ZXN0X3Bhc3N3b3JkQG1vbmdvZGItc3ZjLm1vbmdvREIuc3ZjLmNsdXN0ZXIubG9jYWw6MjcwMTcvdGVzdA== +``` diff --git a/content/docs/2.3/scalers/mssql.md b/content/docs/2.3/scalers/mssql.md new file mode 100644 index 000000000..438b8cc30 --- /dev/null +++ b/content/docs/2.3/scalers/mssql.md @@ -0,0 +1,122 @@ ++++ +title = "MSSQL" +layout = "scaler" +availability = "v2.2+" +maintainer = "Microsoft" +description = "Scale applications based on Microsoft SQL Server (MSSQL) query results." +go_file = "mssql_scaler" ++++ + +### Trigger Specification + +This specification describes the `mssql` trigger that scales based on the results of a [Microsoft SQL Server](https://www.microsoft.com/sql-server/) (MSSQL) query result. This trigger supports local [MSSQL containers](https://hub.docker.com/_/microsoft-mssql-server) as well as SQL Server endpoints hosted in the cloud, such as [Azure SQL Database](https://azure.microsoft.com/services/sql-database/). + +```yaml +triggers: +- type: mssql + metadata: + connectionStringFromEnv: MSSQL_CONNECTION_STRING + query: "SELECT COUNT(*) FROM backlog WHERE state='running' OR state='queued'" + targetValue: 1 + metricName: backlog_process_count # optional - the generated value would be `mssql-{sha256hash}` +``` + +> πŸ’‘ **NOTE:** The connection string format supported by this scaler has some incompatibilities with connection string formats supported by other platforms, like .NET. For example, the MSSQL instance's port number must be separated into its own `Port` property instead of adding it to the `Server` property. You can learn more about all the supported connection string formats for this mssql scaler [here](https://github.com/denisenkom/go-mssqldb#the-connection-string-can-be-specified-in-one-of-three-formats). + +Alternatively, you configure connection parameters explicitly instead of providing a connection string: + +```yaml +triggers: +- type: mssql + metadata: + username: "kedaUser" + passwordFromEnv: MSSQL_PASSWORD + host: mssqlinst.namespace.svc.cluster.local + port: "1433" # optional + database: test_db_name + query: "SELECT COUNT(*) FROM backlog WHERE state='running' OR state='queued'" + targetValue: 1 + metricName: backlog_process_count # optional - the generated value would be `mssql-test_db_name` +``` + +The `mssql` trigger always requires the following information: + +- `query` - a [T-SQL](https://docs.microsoft.com/sql/t-sql/language-reference) query that returns a single numeric value. This can be a regular query or the name of a stored procedure. +- `targetValue` - a threshold that is used as `targetAverageValue` in the Horizontal Pod Autoscaler (HPA). + +To connect to the MSSQL instance, you can provide either: + +- `connectionStringFromEnv` - The name of an environment variable containing a valid MSSQL connection string. + +Or provide more detailed connection parameters explicitly (a connection string will be generated for you at runtime): + +- `host` - The hostname of the MSSQL instance endpoint. +- `port` - The port number of the MSSQL instance endpoint. The default for MSSQL is 1433. +- `database` - The name of the database to query. +- `username` - The username credential for connecting to the MSSQL instance. +- `passwordFromEnv` - The name of an environment variable containing the password credential for connecting to the MSSQL instance. + +When configuring with a connection string, you can use either a URL format (note the URL encoding of special characters): + +``` +sqlserver://user1:Password%231@example.database.windows.net:1433?database=AdventureWorks +``` + +Or the more traditional ADO format: + +``` +Server=example.database.windows.net;Port=1433;Database=AdventureWorks;Persist Security Info=False;User ID=user1;Password=Password#1;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30; +``` + +You can also optionally assign a name to the metric using the `metricName` value. If not specified, the `metricName` will be generated automatically based on the `database` value (if specified), or the `host` value, or will be in the form `mssql-{sha256hash}` where `{sha256hash}` is a SHA-256 hash of the connection string. + +### Authentication parameters + +As an alternative to using environment variables, you can authenticate with the MSSQL instance using connection string or password authentication via `TriggerAuthentication` configuration. + +**Connection string authentication:** + +- `connectionString` - The connection string for the MSSQL instance. + +**Password authentication:** + +- `password` - The password credential for connecting to the MSSQL instance. + +### Example + +The following is an example of how to deploy a scaled object with the `mssql` scale trigger that uses `TriggerAuthentication` and a connection string. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: mssql-secrets +type: Opaque +data: + mssql-connection-string: "Server=example.database.windows.net;port=1433;Database=AdventureWorks;Persist Security Info=False;User ID=user1;Password=Password#1;Encrypt=True;TrustServerCertificate=False;" +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-mssql-secret +spec: + secretTargetRef: + - parameter: connectionString + name: mssql-secrets + key: mssql-connection-string +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: mssql-scaledobject +spec: + scaleTargetRef: + name: consumer # e.g. the name of the resource to scale + triggers: + - type: mssql + metadata: + targetValue: 1 + query: "SELECT COUNT(*) FROM backlog WHERE state='running' OR state='queued'" + authenticationRef: + name: keda-trigger-auth-mssql-secret +``` diff --git a/content/docs/2.3/scalers/mysql.md b/content/docs/2.3/scalers/mysql.md new file mode 100644 index 000000000..2b4488ff9 --- /dev/null +++ b/content/docs/2.3/scalers/mysql.md @@ -0,0 +1,90 @@ ++++ +title = "MySQL" +layout = "scaler" +availability = "v1.2+" +maintainer = "Community" +description = "Scale applications based on MySQL query result." +go_file = "mysql_scaler" ++++ + +### Trigger Specification + +This specification describes the `mysql` trigger that scales based on result of MySQL query. + +The trigger always requires the following information: + +- `query` - a MySQL query that should return single numeric value +- `queryValue` - a threshold that is used as `targetAverageValue` in HPA. + +To provide information about how to connect to MySQL you can provide + +- `connectionStringFromEnv` MySQL connection string that should point to environment variable with valid value + +Or provide more detailed information: + +- `host` - The host of the MySQL server +- `port` - The port of the MySQL server +- `dbName` - Name of the database +- `username` - Username to authenticate with to MySQL database +- `passwordFromEnv` - Password for the given user, this should be blank (no password) or point to an environment + variable with the password + +Some parameters could be provided using environmental variables, instead of setting them directly in metadata. Here is a list of parameters you can use to retrieve values from environment variables: + +- `hostFromEnv` - The host of the MySQL server, similar to `host`, but reads it from an environment variable on the scale target. +- `portFromEnv` - The port of the MySQL server, similar to `port`, but reads it from an environment variable on the scale target. + +### Authentication Parameters + +You can authenticate by using connection string or password authentication. + +**Connection String Authentication:** + +- `connectionString` - Connection string for MySQL database + +**Password Authentication:** + +- `password` - Password for configured user to login to MySQL database +variables. + +### Example + +Here is an example of how to deploy a scaled object with the `mysql` scale trigger which uses `TriggerAuthentication`. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: mysql-secrets + namespace: my-project +type: Opaque +data: + mysql_conn_str: dXNlckB0Y3AobXlzcWw6MzMwNikvc3RhdHNfZGI= # base64 encoded value of mysql connectionString of format user@tcp(mysql:3306)/stats_db +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-mysql-secret + namespace: my-project +spec: + secretTargetRef: + - parameter: connectionString + name: mysql-secrets + key: mysql_conn_str +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: mysql-scaledobject + namespace: my-project +spec: + scaleTargetRef: + name: worker + triggers: + - type: mysql + metadata: + queryValue: "4" + query: "SELECT CEIL(COUNT(*) / 6) FROM task_instance WHERE state='running' OR state='queued'" + authenticationRef: + name: keda-trigger-auth-mysql-secret +``` diff --git a/content/docs/2.3/scalers/nats-streaming.md b/content/docs/2.3/scalers/nats-streaming.md new file mode 100644 index 000000000..f4e725916 --- /dev/null +++ b/content/docs/2.3/scalers/nats-streaming.md @@ -0,0 +1,60 @@ ++++ +title = "NATS Streaming" +layout = "scaler" +availability = "v1.0+" +maintainer = "Community" +description = "Scale applications based on NATS Streaming." +go_file = "stan_scaler" ++++ + +### Trigger Specification + +This specification describes the `stan` trigger for NATS Streaming. + +```yaml +triggers: +- type: stan + metadata: + natsServerMonitoringEndpoint: "stan-nats-ss.stan.svc.cluster.local:8222" + queueGroup: "grp1" + durableName: "ImDurable" + subject: "Test" + lagThreshold: "10" +``` + +**Parameter list:** + +- `natsServerMonitoringEndpoint` - Location of the Nats Streaming monitoring endpoint. +- `queueGroup` - Name of queue group of the subscribers +- `durableName` - Name of durability used by subscribers +- `subject` - Name of the channel +- `lagThreshold` - Average target value to trigger scaling actions. + +### Authentication Parameters + +Not supported yet. + +### Example + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: stan-scaledobject + namespace: gonuts +spec: + pollingInterval: 10 # Optional. Default: 30 seconds + cooldownPeriod: 30 # Optional. Default: 300 seconds + minReplicaCount: 0 # Optional. Default: 0 + maxReplicaCount: 30 # Optional. Default: 100 + scaleTargetRef: + name: gonuts-sub + triggers: + - type: stan + metadata: + natsServerMonitoringEndpoint: "stan-nats-ss.stan.svc.cluster.local:8222" + queueGroup: "grp1" + durableName: "ImDurable" + subject: "Test" + lagThreshold: "10" +``` diff --git a/content/docs/2.3/scalers/openstack-metric.md b/content/docs/2.3/scalers/openstack-metric.md new file mode 100644 index 000000000..7781fc044 --- /dev/null +++ b/content/docs/2.3/scalers/openstack-metric.md @@ -0,0 +1,218 @@ ++++ +title = "OpenStack Metric" +layout = "scaler" +availability = "v2.3+" +maintainer = "Community" +<<<<<<< HEAD +<<<<<<< HEAD +<<<<<<< HEAD +description = "Scale applications based on a threshold reached by a specific measure from Openstack Metric API." +======= +description = "Scale applications based on the metrics collected from an specific resource." +>>>>>>> Create Openstack Metric Scaler docs +======= +description = "Scale applications based on a threshold reached by a specific measure from Openstack Metric API." +>>>>>>> improve openstack metric scaler initial description +======= +description = "Scale applications based on a threshold reached by a specific measure from OpenStack Metric API." +>>>>>>> Update content/docs/2.2/scalers/openstack-metric.md +go_file = "openstack_metric_scaler" ++++ + +### Trigger Specification +<<<<<<< HEAD +<<<<<<< HEAD +This specification describes the `openstack-metric` trigger for OpenStack metrics. + +> The Openstack metric API follows [gnocchi](https://gnocchi.xyz/) API. Attempt to the fact that gnocchi API is an open-source time series database embedded in openstack system and every parameter on Openstack Metric API follows its patterns but you don't need to reference anything from gnocchi. It scales based on a specific measure from a given resource metric. +======= +This specification describes the `openstack-metric` trigger for OpenStack metrics. The Openstack metric API follows [gnocchi](https://gnocchi.xyz/) API. Attempt to the fact that gnocchi api is an open-source time series database embedded in openstack system and every parameter on Openstack Metric API follows it's patterns but you don't need to reference anything from gnocchi. It scales based on an spcific measeure from a given resource metric. +>>>>>>> Create Openstack Metric Scaler docs +======= +This specification describes the `openstack-metric` trigger for OpenStack metrics. + +<<<<<<< HEAD +> The Openstack metric API follows [gnocchi](https://gnocchi.xyz/) API. Attempt to the fact that gnocchi API is an open-source time series database embedded in openstack system and every parameter on Openstack Metric API follows its patterns but you don't need to reference anything from gnocchi. It scales based on a specific measure from a given resource metric. +<<<<<<< HEAD +>>>>>>> Update content/docs/2.2/scalers/openstack-metric.md +It's highly recommended to check [gnocchi](https://gnocchi.xyz/) docs. +======= +======= +> The OpenStack metric API follows [Gnocchi](https://gnocchi.xyz/) API. Attempt to the fact that Gnocchi API is an open-source time series database embedded in OpenStack system and every parameter on OpenStack Metric API follows its patterns but you don't need to reference anything from Gnocchi. It scales based on a specific measure from a given resource metric. +>>>>>>> Update content/docs/2.2/scalers/openstack-metric.md +It's highly recommended to check [Gnocchi](https://gnocchi.xyz/) docs. +>>>>>>> Update content/docs/2.2/scalers/openstack-metric.md + +```yaml +triggers: +- type: openstack-metric + metadata: + metricsURL: http://localhost:8041/v1/metric #required + metricID: 003bb589-166d-439d-8c31-cbf098d863de #required + aggregationMethod: "mean" #required + granularity: 300 #required (seconds) + threshold: 1250 #required + timeout: 30 #optional +``` + +<<<<<<< HEAD +<<<<<<< HEAD +> Protocol (http or https) should always be provided when specifying URLs +======= +> Please, always provide the protocol (http or https) when specifying URLs. This is needed due to Go URL parsing issues :sweat_smile: +>>>>>>> Create Openstack Metric Scaler docs +======= +> Protocol (http or https) should always be provided when specifying URLs +>>>>>>> Update content/docs/2.2/scalers/openstack-metric.md + +**Parameter list:** +- `metricsURL` - The URL to check for the metrics API, based. It must contain the hostname, the metric port, the API version, and the resource ID. The pattern is: `http://:///metric` +- `metricID` - The Id of the intendend metric +- `aggregationMethod` - The aggregation method that will be used to calculate metrics, it must follows the configured possible metrics derived from gnocchi API like: `mean`, `min`, `max`, `std`, `sum`, `count`, the complete aggregation methods list can be found [here](https://gnocchi.xyz/rest.html#archive-policy). +- `granularity` - The configured granularity from metric collection in seconds. it must follow the same value configured in OpenStack, but it must be coutned in seconds. Sample: If you have a 5 minutes time window granularity defined, so you must input a value of 300 seconds (5*60). +- `threshold` - The target value that, when reached, will scale the application. +- `timeout` - The timeout, in seconds, for the HTTP client requests that will query the Metric API. If not specified, the default value is `30` seconds. This must be an integer value. If `0` or negative value is provided it defaults to `300` milliseconds. + +### Authentication Parameters + +To authenticate, this scaler uses tokens. Tokens are automatically retrieved by the scaler from [Keystone](https://docs.openstack.org/keystone/latest/), the official OpenStack Identity Provider. You can provide your credentials using Secrets either by using the "password" method or the "application credentials" method. Both cases use `TriggerAuthentication`. + +#### Password + +- `authURL` - The Keystone authentication URL. The pattern is: `http://://` +- `userID` - The OpenStack project user ID +- `password` - The password for the provided user +- `projectID` - The OpenStack project ID + +#### Application Credentials + +- `authURL` - The Keystone authentication URL. The pattern is: `http://://` +- `appCredentialID` - The Application Credential ID +- `appCredentialSecret` - The Application Credential secret + +### Example + +#### Password method + +Here is an example of how to deploy a scaled object with the `openstack-metric` scale trigger which uses `TriggerAuthentication` and the Password method from above. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: openstack-secret-password + namespace: default +type: Opaque +data: + authURL: aHR0cDovL2xvY2FsaG9zdDo1MDAwL3YzLw== + userID: MWYwYzI3ODFiNDExNGQxM2E0NGI4ODk4Zjg1MzQwYmU= + password: YWRtaW4= + projectID: YjE2MWRjNTE4Y2QyNGJkYTg0ZDk0ZDlhMGU3M2ZjODc= +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: metrics-password-trigger-authentication + namespace: default +spec: + secretTargetRef: + - parameter: authURL + name: openstack-secret-password + key: authURL + - parameter: userID + name: openstack-secret-password + key: userID + - parameter: password + name: openstack-secret-password + key: password + - parameter: projectID + name: openstack-secret-password + key: projectID +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: openstack-metric-scaledobject + namespace: default +spec: + scaleTargetRef: + name: my-deployment + pollingInterval: 10 + cooldownPeriod: 10 + minReplicaCount: 0 + triggers: + - type: openstack-metric + metadata: + metricsURL: http://localhost:8041/v1/metric + metricID: faf01aa5-da88-4929-905d-b83fbab46771 + aggregationMethod: "mean" + granularity: 300 + threshold: 1250 + timeout: 30 + authenticationRef: + name: openstack-metric-password-trigger-authentication +``` + +#### Application Credentials method + +You can also use the Application Credentials method. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: openstack-secret-appcredentials + namespace: default +type: Opaque +data: + authURL: aHR0cDovL2xvY2FsaG9zdDo1MDAwL3YzLw== + appCredentialID: OWYyY2UyYWRlYmFkNGQxNzg0NTgwZjE5ZTljMTExZTQ= + appCredentialSecret: LVdSbFJBZW9sMm91Z3VmZzNEVlBqcll6aU9za1pkZ3c4Y180XzRFU1pZREloT0RmajJkOHg0dU5yb3NudVIzWmxDVTZNLTVDT3R5NDFJX3M5R1N5Wnc= +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: openstack-metric-appcredentials-trigger-authentication + namespace: default +spec: + secretTargetRef: + - parameter: authURL + name: openstack-secret-appcredentials + key: authURL + - parameter: appCredentialID + name: openstack-secret-appcredentials + key: appCredentialID + - parameter: appCredentialSecret + name: openstack-secret-appcredentials + key: appCredentialSecret +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: openstack-metric-scaledobject + namespace: default +spec: + scaleTargetRef: + name: my-deployment + pollingInterval: 10 + cooldownPeriod: 10 + minReplicaCount: 0 + triggers: + - type: openstack-metric + metadata: + metricsURL: http://localhost:8041/v1/metric + metricID: faf01aa5-da88-4929-905d-b83fbab46771 + aggregationMethod: "mean" + granularity: 300 + threshold: 1250 + authenticationRef: + name: openstack-metric-appcredentials-trigger-authentication +<<<<<<< HEAD +<<<<<<< HEAD +``` +======= +``` +>>>>>>> Create Openstack Metric Scaler docs +======= +``` +>>>>>>> Update content/docs/2.2/scalers/openstack-metric.md diff --git a/content/docs/2.3/scalers/openstack-swift.md b/content/docs/2.3/scalers/openstack-swift.md new file mode 100644 index 000000000..459c23595 --- /dev/null +++ b/content/docs/2.3/scalers/openstack-swift.md @@ -0,0 +1,178 @@ ++++ +title = "OpenStack Swift" +layout = "scaler" +availability = "v2.1+" +maintainer = "Community" +description = "Scale applications based on the count of objects in a given OpenStack Swift container." +go_file = "openstack_swift_scaler" ++++ + +### Trigger Specification + +This specification describes the `openstack-swift` trigger for OpenStack Swift containers. It scales based on the count of objects in a given container. + +```yaml +triggers: +- type: openstack-swift + metadata: + containerName: my-container # Required + swiftURL: http://localhost:8080/v1/b161dc815cd24bda84d94d9a0e73cf78 # Optional + objectCount: "2" # Optional + objectPrefix: "my-prefix" # Optional + objectDelimiter: "/" # Optional + objectLimit: "10" # Optional + onlyFiles: "true" # Optional + timeout: "2" # Optional +``` + +> Please, always provide the protocol (http or https) when specifying URLs. + +**Parameter list:** + +- `swiftURL` - The URL to query the Swift API. If not provided, the scaler will try to find the Swift public URL for a certain region, using the OpenStack catalog, which is returned when requesting an authentication token. +- `containerName` - Name of Swift container in an OpenStack account +- `objectCount` - Average target value to trigger scaling actions. If not specified, the default value is `2` +- `objectPrefix` - Prefix for the object. Only objects with this prefix will be returned. Use this prefix to specify sub-paths for the objects, if required. If not specified, the default value is `""`. +- `objectDelimiter` - Delimiter for identifying the object prefix. It is the character used to split object names. The default value is `""`. +- `objectLimit` - The maximum number of objects returned by the API. By default, the Swift API only returns up to 10000 names. The default value is `""`, which will use the Swift API standard limit. Negative values default to the Swift API standard limit. +- `onlyFiles` - Boolean value that specifies if the scaler should be triggered only by the number of files, without considering folders. Inside a container, one can have files and folders. Folders (empty or not) are counted as objects, just as files are. If one wants to scale based on only files, this parameter must be set to `true`. The default value is `false`. +- `timeout` - The timeout, in seconds, for the HTTP client requests that will query the Swift API. If not specified, the default value is `30` seconds. This must be an integer value. If `0` or negative value is provided it defaults to `300` milliseconds. + +For more information about `prefix`, `delimiter`, and `limit`, please, refer to the [Object Store API](https://docs.openstack.org/api-ref/object-store/). + +### Authentication Parameters + +To authenticate, this scaler uses tokens. Tokens are automatically retrieved by the scaler from [Keystone](https://docs.openstack.org/keystone/latest/), the official OpenStack Identity Provider. You can provide your credentials using Secrets either by using the "password" method or the "application credentials" method. Both cases use `TriggerAuthentication`. + +#### Password + +- `authURL` - The Keystone authentication URL. The scaler supports only Keystone v3. You shouldn't include the `/v3` parameter in your URL path. This is done automatically by the scaler. +- `userID` - The OpenStack project user ID +- `password` - The password for the provided user +- `projectID` - The OpenStack project ID +- `regionName` - The OpenStack region name where the Swift service is available. This parameter is not required and is used only when the `swiftURL` is not provided to the scaler. If the region name is not provided, it will look for the first Swift public URL available in the OpenStack catalog. + +#### Application Credentials + +- `authURL` - The Keystone authentication URL. The scaler supports only Keystone v3. You shouldn't include the `/v3` parameter in your URL path. This is done automatically by the scaler. +- `appCredentialID` - The Application Credential ID +- `appCredentialSecret` - The Application Credential secret +- `regionName` - The OpenStack region name where the Swift service is available. This parameter is not required and is used only when the `swiftURL` is not provided to the scaler. If the region name is not provided, it will look for the first Swift public URL available in the OpenStack catalog. + +### Example + +#### Password method + +Here is an example of how to deploy a scaled object with the `openstack-swift` scale trigger which uses `TriggerAuthentication` and the Password method from above. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: openstack-secret-password + namespace: default +type: Opaque +data: + authURL: aHR0cDovL2xvY2FsaG9zdDo1MDAwLw== + userID: MWYwYzI3ODFiNDExNGQxM2E0NGI4ODk4Zjg1MzQwYmU= + password: YWRtaW4= + projectID: YjE2MWRjNTE4Y2QyNGJkYTg0ZDk0ZDlhMGU3M2ZjODc= + regionName: Y2FsaWZvcm5pYS0x +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: swift-password-trigger-authentication + namespace: default +spec: + secretTargetRef: + - parameter: authURL + name: openstack-secret-password + key: authURL + - parameter: userID + name: openstack-secret-password + key: userID + - parameter: password + name: openstack-secret-password + key: password + - parameter: projectID + name: openstack-secret-password + key: projectID + - parameter: regionName + name: openstack-secret-password + key: regionName +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: swift-scaledobject + namespace: default +spec: + scaleTargetRef: + name: my-deployment + pollingInterval: 10 + cooldownPeriod: 10 + minReplicaCount: 0 + triggers: + - type: openstack-swift + metadata: + containerName: my-container + objectCount: "1" + onlyFiles: "true" + authenticationRef: + name: swift-password-trigger-authentication +``` + +#### Application Credentials method + +You can also use the Application Credentials method. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: openstack-secret-appcredentials + namespace: default +type: Opaque +data: + authURL: aHR0cDovL2xvY2FsaG9zdDo1MDAwLw== + appCredentialID: OWYyY2UyYWRlYmFkNGQxNzg0NTgwZjE5ZTljMTExZTQ= + appCredentialSecret: LVdSbFJBZW9sMm91Z3VmZzNEVlBqcll6aU9za1pkZ3c4Y180XzRFU1pZREloT0RmajJkOHg0dU5yb3NudVIzWmxDVTZNLTVDT3R5NDFJX3M5R1N5Wnc= +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: swift-appcredentials-trigger-authentication + namespace: default +spec: + secretTargetRef: + - parameter: authURL + name: openstack-secret-appcredentials + key: authURL + - parameter: appCredentialID + name: openstack-secret-appcredentials + key: appCredentialID + - parameter: appCredentialSecret + name: openstack-secret-appcredentials + key: appCredentialSecret +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: swift-scaledobject + namespace: default +spec: + scaleTargetRef: + name: my-deployment + pollingInterval: 10 + cooldownPeriod: 10 + minReplicaCount: 0 + triggers: + - type: openstack-swift + metadata: + swiftURL: http://localhost:8080/v1/AUTH_b161dc518cd24bda84d94d9a0e73fc87 + containerName: my-container + objectCount: "1" + authenticationRef: + name: swift-appcredentials-trigger-authentication +``` \ No newline at end of file diff --git a/content/docs/2.3/scalers/postgresql.md b/content/docs/2.3/scalers/postgresql.md new file mode 100644 index 000000000..f5af5e011 --- /dev/null +++ b/content/docs/2.3/scalers/postgresql.md @@ -0,0 +1,100 @@ ++++ +title = "PostgreSQL" +layout = "scaler" +availability = "v1.2+" +maintainer = "Community" +description = "Scale applications based on a PostgreSQL query." +go_file = "postgresql_scaler" ++++ + +### Trigger Specification + +This specification describes the `postgresql` trigger that scales based on a postgresql query + +The Postgresql scaler allows for two connection options: + +A user can offer a full connection string +(often in the form of an environment variable secret) + +- `connectionFromEnv` postgreSQL connection string that should point to environment variable with valid value + +Alternatively, a user can specify individual +arguments (host, userName, password, etc.), and the scaler will form a connection string +internally. + +- `host:` Service URL to postgresql. Note that you should use a full svc URL as KEDA will need to contact postgresql from a different namespace +- `userName:` Username for postgresql user +- `passwordFromEnv` Password for postgresql user +- `port:` Postgresql port +- `dbName:` Postgresql Database name +- `sslmode:` SSL policy for communicating with database + +Finally, a user inserts a query that returns the desired value + +- `query:` What query to poll postgresql with. Query must return an integer. +- `targetQueryValue` - a threshold that is used as `targetAverageValue` in HPA. +- `metricName` - an optional name to assign to the metric. If not set KEDA will generate a name based on either the connection string if set or the db name. If using more than one trigger it is required that all metricNames be unique. + +This is an example of using a full connection string with `AIRFLOW_CONN_AIRFLOW_DB` set as `postgresql://test@localhost`: + +```yaml +triggers: +- type: postgresql + metadata: + connectionFromEnv: AIRFLOW_CONN_AIRFLOW_DB + query: "SELECT ceil(COUNT(*)::decimal / 16) FROM task_instance WHERE state='running' OR state='queued'" + targetQueryValue: 1 + metricName: backlog_process_count #optional. Generated value would be `postgresql-postgresql---test@localhost` +``` + +While this is an example of specifying each parameter: + +```yaml +triggers: +- type: postgresql + metadata: + userName: "kedaUser" + passwordFromEnv: PG_PASSWORD + host: postgres-svc.namespace.cluster.local #use the cluster-wide namespace as KEDA + #lives in a different namespace from your postgres + port: "5432" + dbName: test_db_name + sslmode: disable + query: "SELECT ceil(COUNT(*)::decimal / 16) FROM task_instance WHERE state='running' OR state='queued'" + targetQueryValue: 1 + metricName: backlog_process_count #optional. Generated value would be `postgresql-test_db_name` +``` + +### Authentication Parameters + +You can authenticate by using a password or store the password within the connectionString. + +**Connection String Authentication:** + +- `connection` - Connection string for postgreSQL database + +**Password Authentication:** + +- `password` - Password for configured user to login to postgreSQL database +variables. + +### Example + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: airflow-worker +spec: + scaleTargetRef: + name: airflow-worker + pollingInterval: 10 # Optional. Default: 30 seconds + cooldownPeriod: 30 # Optional. Default: 300 seconds + maxReplicaCount: 10 # Optional. Default: 100 + triggers: + - type: postgresql + metadata: + connectionFromEnv: AIRFLOW_CONN_AIRFLOW_DB + query: "SELECT ceil(COUNT(*)::decimal / 16) FROM task_instance WHERE state='running' OR state='queued'" + targetQueryValue: 1 +``` diff --git a/content/docs/2.3/scalers/prometheus.md b/content/docs/2.3/scalers/prometheus.md new file mode 100644 index 000000000..e7820cc29 --- /dev/null +++ b/content/docs/2.3/scalers/prometheus.md @@ -0,0 +1,287 @@ ++++ +title = "Prometheus" +layout = "scaler" +availability = "v1.0+" +maintainer = "Community" +description = "Scale applications based on Prometheus." +go_file = "prometheus_scaler" ++++ + +### Trigger Specification + +This specification describes the `prometheus` trigger that scales based on a Prometheus. + +```yaml +triggers: +- type: prometheus + metadata: + # Required + serverAddress: http://:9090 + metricName: http_requests_total + query: sum(rate(http_requests_total{deployment="my-deployment"}[2m])) # Note: query must return a vector/scalar single element response + threshold: '100' +``` + +**Parameter list:** + +- `serverAddress` - Address of Prometheus +- `metricName` - Metric name to use +- `query` - Query to run +- `threshold` - Value to start scaling for + +### Authentication Parameters + +Prometheus Scaler supports three types of authentication - bearer authentication, basic authentication and TLS authentication. + +You can use `TriggerAuthentication` CRD to configure the authentication. It is possible to specify multiple authentication types i.e. `authModes: "tls,basic"` Specify `authModes` and other trigger parameters along with secret credentials in `TriggerAuthentication` as mentioned below: + +**Bearer authentication:** +- `authModes`: It must contain `bearer` in case of Bearer Authentication. Specify this in trigger configuration. +- `bearerToken`: The token needed for authentication. This is a required field. + +**Basic authentication:** +- `authMode`: It must contain `basic` in case of Basic Authentication. Specify this in trigger configuration. +- `username`: This is a required field. Provide the username to be used for basic authentication. +- `password`: Provide the password to be used for authentication. For convenience, this has been marked optional, because many applications implement basic auth with a username as apikey and password as empty. + +**TLS authentication:** +- `authMode`: It must contain `tls` in case of TLS Authentication. Specify this in trigger configuration. +- `ca`: Certificate authority file for TLS client authentication. +- `cert`: Certificate for client authentication. This is a required field. +- `key`: Key for client authentication. Optional. This is a required field. + +> πŸ’‘ **NOTE:**It's also possible to set the CA certificate regardless of the selected `authMode` (also without any authentication). This might be usefull if you are using an enterprise CA. + +### Example + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: prometheus-scaledobject + namespace: default +spec: + scaleTargetRef: + name: my-deployment + triggers: + - type: prometheus + metadata: + serverAddress: http://:9090 + metricName: http_requests_total + threshold: '100' + query: sum(rate(http_requests_total{deployment="my-deployment"}[2m])) +``` + +Here is an example of a prometheus scaler with bearer authentication, + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: keda-prom-secret + namespace: default +data: + bearerToken: "BEARER_TOKEN" + ca: "CUSTOM_CA_CERT" +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-prom-creds + namespace: default +spec: + secretTargetRef: + - parameter: bearerToken + name: keda-prom-secret + key: bearerToken + # might be required if you're using a custom CA + - parameter: ca + name: keda-prom-secret + key: ca +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: prometheus-scaledobject + namespace: keda + labels: + deploymentName: dummy +spec: + maxReplicaCount: 12 + scaleTargetRef: + name: dummy + triggers: + - type: prometheus + metadata: + serverAddress: http://:9090 + metricName: http_requests_total + threshold: '100' + query: sum(rate(http_requests_total{deployment="my-deployment"}[2m])) + authModes: "bearer" + authenticationRef: + name: keda-prom-creds +``` + +Here is an example of a prometheus scaler with Basic Authentication, define the `Secret` and `TriggerAuthentication` as follows + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: keda-prom-secret + namespace: default +data: + username: "username" + password: "password" +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-prom-creds + namespace: default +spec: + secretTargetRef: + - parameter: username + name: keda-prom-secret + key: username + - parameter: password + name: keda-prom-secret + key: password +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: prometheus-scaledobject + namespace: keda + labels: + deploymentName: dummy +spec: + maxReplicaCount: 12 + scaleTargetRef: + name: dummy + triggers: + - type: metrics-api + metadata: + serverAddress: http://:9090 + metricName: http_requests_total + threshold: '100' + query: sum(rate(http_requests_total{deployment="my-deployment"}[2m])) + authModes: "basic" + authenticationRef: + name: keda-prom-creds +``` + + +Here is an example of a prometheus scaler with TLS Authentication, define the `Secret` and `TriggerAuthentication` as follows + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: keda-prom-secret + namespace: default +data: + cert: "cert" + key: "key" + ca: "ca" +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-prom-creds + namespace: default +spec: + secretTargetRef: + - parameter: cert + name: keda-prom-secret + key: cert + - parameter: key + name: keda-prom-secret + key: key + - parameter: ca + name: keda-prom-secret + key: ca +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: prometheus-scaledobject + namespace: keda + labels: + deploymentName: dummy +spec: + maxReplicaCount: 12 + scaleTargetRef: + name: dummy + triggers: + - type: metrics-api + metadata: + serverAddress: http://:9090 + metricName: http_requests_total + threshold: '100' + query: sum(rate(http_requests_total{deployment="my-deployment"}[2m])) + authModes: "tls" + authenticationRef: + name: keda-prom-creds +``` + +Here is an example of a prometheus scaler with TLS and Basic Authentication, define the `Secret` and `TriggerAuthentication` as follows + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: keda-prom-secret + namespace: default +data: + cert: "cert" + key: "key" + ca: "ca" + username: "username" + password: "password" +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-prom-creds + namespace: default +spec: + secretTargetRef: + - parameter: cert + name: keda-prom-secret + key: cert + - parameter: key + name: keda-prom-secret + key: key + - parameter: ca + name: keda-prom-secret + key: ca + - parameter: username + name: keda-prom-secret + key: username + - parameter: password + name: keda-prom-secret + key: password +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: prometheus-scaledobject + namespace: keda + labels: + deploymentName: dummy +spec: + maxReplicaCount: 12 + scaleTargetRef: + name: dummy + triggers: + - type: metrics-api + metadata: + serverAddress: http://:9090 + metricName: http_requests_total + threshold: '100' + query: sum(rate(http_requests_total{deployment="my-deployment"}[2m])) + authModes: "tls,basic" + authenticationRef: + name: keda-prom-creds +``` \ No newline at end of file diff --git a/content/docs/2.3/scalers/rabbitmq-queue.md b/content/docs/2.3/scalers/rabbitmq-queue.md new file mode 100644 index 000000000..92ce1b554 --- /dev/null +++ b/content/docs/2.3/scalers/rabbitmq-queue.md @@ -0,0 +1,190 @@ ++++ +title = "RabbitMQ Queue" +layout = "scaler" +availability = "v1.0+" +maintainer = "Microsoft" +description = "Scale applications based on RabbitMQ Queue." +go_file = "rabbitmq_scaler" ++++ + +### Trigger Specification + +This specification describes the `rabbitmq` trigger for RabbitMQ Queue. + +```yaml +triggers: +- type: rabbitmq + metadata: + host: amqp://localhost:5672/vhost # Optional. If not specified, it must be done by using TriggerAuthentication. + protocol: auto # Optional. Specifies protocol to use, either amqp or http, or auto to autodetect based on the `host` value. Default value is auto. + mode: QueueLength # QueueLength or MessageRate + value: 100 # message backlog or publish/sec. target per instance + queueName: testqueue + vhostName: / # Optional. If not specified, use the vhost in the `host` connection string. + # Alternatively, you can use existing environment variables to read configuration from: + # See details in "Parameter list" section + hostFromEnv: RABBITMQ_HOST # Optional. You can use this instead of `host` parameter +``` + +**Parameter list:** + +- `host`: Host of RabbitMQ with format `://:/vhost`. The resolved host should follow a format like `amqp://guest:password@localhost:5672/vhost` or + `http://guest:password@localhost:15672/vhost`. When using a username/password consider using `hostFromEnv` or a TriggerAuthentication. + +- `queueName`: Name of the queue to read message from. Required. +- `mode`: QueueLength to trigger on number of messages in the queue. MessageRate to trigger on the publish rate into the queue. Required. +- `value`: Message backlog or Publish/sec. rate to trigger on +- `protocol`: Protocol to be used for communication. Either `auto`, `http`, or `amqp`. It should correspond with the `host` value. Optional, will autodetect based on the `host` URL if possible. +- `vhostName`: Vhost to use for the connection, overrides any vhost set in the connection string from `host`/`hostFromEnv`. +- `queueLength`: DEPRECATED! Use `mode: QueueLength` and `value: ##` instead. Target value for queue length passed to the scaler. Example: if one pod can handle 10 messages, set the queue length target to 10. If the actual number of messages in the queue is 30, the scaler scales to 3 pods. Default is 20 unless `publishRate` is specified, in which case `queueLength` is disabled for this trigger. + +Some parameters could be provided using environmental variables, instead of setting them directly in metadata. Here is a list of parameters you can use to retrieve values from environment variables: + +- `hostFromEnv`: The host and port of the Redis server, similar to `host`, but reads it from an environment variable on the scale target. + +> πŸ’‘ **Note:** `host`/`hostFromEnv` has an optional vhost name after the host slash which will be used to scope API request. + +> πŸ’‘ **Note:** `mode: MessageRate` requires protocol `http`. + +> ⚠ **Important:** if you have unacknowledged messages and want to have these counted for the scaling to happen, make sure to utilize the `http` REST API interface which allows for these to be counted. + +> ⚠ **Important:** If scaling against both is desired then the `ScaledObject` should have two triggers, one for `mode: QueueLength` and the other for `mode: MessageRate`. HPA will scale based on the largest result considering each of the two triggers independently. + +### Authentication Parameters + +TriggerAuthentication CRD is used to connect and authenticate to RabbitMQ: + +- For AMQP, the URI should look similar to `amqp://guest:password@localhost:5672/vhost` +- For HTTP, the URI should look similar to `http://guest:password@localhost:15672/vhost` + +> See the [RabbitMQ Ports](https://www.rabbitmq.com/networking.html#ports) section for more details on how to configure the ports. + +### Example + +#### AMQP protocol: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: keda-rabbitmq-secret +data: + host: # base64 encoded value of format amqp://guest:password@localhost:5672/vhost +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-rabbitmq-conn + namespace: default +spec: + secretTargetRef: + - parameter: host + name: keda-rabbitmq-secret + key: host +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: rabbitmq-scaledobject + namespace: default +spec: + scaleTargetRef: + name: rabbitmq-deployment + triggers: + - type: rabbitmq + metadata: + protocol: amqp + queueName: testqueue + mode: QueueLength + value: "20" + authenticationRef: + name: keda-trigger-auth-rabbitmq-conn +``` + +#### HTTP protocol (`QueueLength`): + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: keda-rabbitmq-secret +data: + host: # base64 encoded value of format http://guest:password@localhost:15672/vhost +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-rabbitmq-conn + namespace: default +spec: + secretTargetRef: + - parameter: host + name: keda-rabbitmq-secret + key: host +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: rabbitmq-scaledobject + namespace: default +spec: + scaleTargetRef: + name: rabbitmq-deployment + triggers: + - type: rabbitmq + metadata: + protocol: http + queueName: testqueue + mode: QueueLength + value: "20" + authenticationRef: + name: keda-trigger-auth-rabbitmq-conn +``` + +#### HTTP protocol (`MessageRate` and `QueueLength`): + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: keda-rabbitmq-secret +data: + host: # base64 encoded value of format http://guest:password@localhost:15672/vhost +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-rabbitmq-conn + namespace: default +spec: + secretTargetRef: + - parameter: host + name: keda-rabbitmq-secret + key: host +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: rabbitmq-scaledobject + namespace: default +spec: + scaleTargetRef: + name: rabbitmq-deployment + triggers: + - type: rabbitmq + metadata: + protocol: http + queueName: testqueue + mode: QueueLength + value: "20" + authenticationRef: + name: keda-trigger-auth-rabbitmq-conn + - type: rabbitmq + metadata: + protocol: http + queueName: testqueue + mode: MessageRate + value: "100" + authenticationRef: + name: keda-trigger-auth-rabbitmq-conn +``` diff --git a/content/docs/2.3/scalers/redis-cluster-lists.md b/content/docs/2.3/scalers/redis-cluster-lists.md new file mode 100644 index 000000000..6fd17a92b --- /dev/null +++ b/content/docs/2.3/scalers/redis-cluster-lists.md @@ -0,0 +1,102 @@ ++++ +title = "Redis Lists (supports Redis Cluster)" +layout = "scaler" +availability = "v2.1+" +maintainer = "Community" +description = "Redis Lists scaler with support for Redis Cluster topology" +go_file = "redis_scaler" ++++ + +### Trigger Specification + +This specification describes the `redis-cluster` trigger that scales based on the length of a list in a Redis Cluster. + +```yaml +triggers: +- type: redis-cluster + metadata: + addresses: localhost:6379 # Comma separated list of the format host:port + passwordFromEnv: REDIS_PASSWORD + listName: mylist # Required + listLength: "5" # Required + enableTLS: "false" # optional + # Alternatively, you can use existing environment variables to read configuration from: + # See details in "Parameter list" section + addressesFromEnv: REDIS_ADDRESSES # Optional. You can use this instead of `addresses` parameter +``` + +**Parameter list:** + +- `addresses` - Comma separated list of hosts and ports of the Redis Cluster nodes. +- `hosts` - Comma separated list of hosts of the Redis Cluster nodes. Alternative to `addresses` and requires `ports` to be configured as well. +- `ports` - Comma separated list of corresponding ports for the hosts of the Redis Cluster nodes. Alternative to `addresses` and requires `hosts` to be configured as well. +- `passwordFromEnv` - Environment variable to read the authentication password from to authenticate with the Redis server. + - Both the hostname and password fields need to be set to the names of the environment variables in the target deployment that contain the host name and password respectively. +- `listName` - Name of the Redis List that you want to monitor +- `listLength` - Average target value to trigger scaling actions +- `enableTLS` - If set to `true` allow a connection to a redis queue using tls, the default value for this parameter is false. + +Some parameters could be provided using environmental variables, instead of setting them directly in metadata. Here is a list of parameters you can use to retrieve values from environment variables: + +- `addressesFromEnv` - The hosts and their respective ports of the Redis Cluster nodes, similar to `addresses`, but reads it from an environment variable on the scale target. +- `hostsFromEnv` - The hosts of the Redis Cluster nodes, similar to `hosts`, but reads it from an environment variable on the scale target. +- `portsFromEnv` - The corresponding ports for the hosts of the Redis Cluster nodes, similar to `ports`, but reads it from an environment variable on the scale target. + +### Authentication Parameters + +You can authenticate by using a password. + +**Connection Authentication:** + +- `addresses` - Comma separated list of host:port format. +- `hosts` - Comma separated list of hostname of the Redis Cluster nodes. If specified, the `ports` should also be specified +- `ports` - Comma separated list of ports of the Redis Cluster nodes. If specified, the `hosts` should also be specified + +**Password Authentication:** + +- `password` - Redis password to authenticate with + +### Example + +Here is an example of how to deploy a scaled object with the `redis-cluster` scale trigger which uses `TriggerAuthentication`. + +You can also provide the `passwordFromEnv` on the `ScaledObject` directly. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: votes-db-secret + namespace: my-project +type: Opaque +data: + redis_password: YWRtaW4= +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-redis-secret + namespace: my-project +spec: + secretTargetRef: + - parameter: password + name: votes-db-secret + key: redis_password +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: redis-scaledobject + namespace: my-project +spec: + scaleTargetRef: + name: votes + triggers: + - type: redis-cluster + metadata: + addresses: node1:6379, node2:6379, node3:6379 + listName: mylist + listLength: "10" + authenticationRef: + name: keda-trigger-auth-redis-secret +``` diff --git a/content/docs/2.3/scalers/redis-cluster-streams.md b/content/docs/2.3/scalers/redis-cluster-streams.md new file mode 100644 index 000000000..81fec2541 --- /dev/null +++ b/content/docs/2.3/scalers/redis-cluster-streams.md @@ -0,0 +1,145 @@ ++++ +title = "Redis Streams (supports Redis Cluster)" +layout = "scaler" +availability = "v2.1+" +maintainer = "Community" +description = "Redis Streams scaler with support for Redis Cluster topology" +go_file = "redis_streams_scaler" ++++ + +### Trigger Specification + +Redis 5.0 introduced [Redis Streams](https://redis.io/topics/streams-intro) which is an append-only log data structure. + +One of its features includes [`Consumer Groups`](https://redis.io/topics/streams-intro#consumer-groups), that allows a group of clients to co-operate consuming a different portion of the same stream of messages. + +This specification describes the `redis-cluster-streams` trigger that scales based on the *Pending Entries List* (see [`XPENDING`](https://redis.io/commands/xpending)) for a specific Consumer Group of a Redis Stream and supports Redis Cluster topology. + + +```yaml +triggers: +- type: redis-cluster-streams + metadata: + addresses: localhost:6379 # Required if hosts and ports are not provided. Format - comma separated list of host:port + hosts: localhost # Comma separated lists of hosts. Required if address is not provided + ports: "6379" # Comma separated lists of ports. Required if addresses are not provided and hosts has been provided. + passwordFromEnv: REDIS_PASSWORD # optional (can also use authenticationRef) + stream: my-stream # Required - name of the Redis Stream + consumerGroup: my-consumer-group # Required - name of consumer group associated with Redis Stream + pendingEntriesCount: "10" # Required - number of entries in the Pending Entries List for the specified consumer group in the Redis Stream + enableTLS: "false" # optional + # Alternatively, you can use existing environment variables to read configuration from: + # See details in "Parameter list" section + addressesFromEnv: REDIS_ADDRESSES # Optional. You can use this instead of `addresses` parameter + hostsFromEnv: REDIS_HOSTS # Optional. You can use this instead of `hosts` parameter + portsFromEnv: REDIS_PORTS # Optional. You can use this instead of `ports` parameter +``` + +**Parameter list:** + +- `addresses`: Comma separated list of hosts and ports of Redis Cluster nodes in the format `host:port` for example `node1:6379, node2:6379, node3:6379` + +> As an alternative to the `addresses` field, the user can specify `hosts` and `ports` parameters. + +- `hosts`: Comma separated list of hosts of Redis Cluster nodes. + +> It is not required if `addresses` has been provided + +- `ports`: Comma separated list of ports for corresponding hosts of Redis Cluster nodes. + +> It is only to be used along with the `hosts`/`hostsFromEnv` attribute and not required if `addresses` has been provided + +- `passwordFromEnv` (optional): Name of the environment variable your deployment uses to get the Redis password. + +- `stream`: Name of the Redis Stream +- `consumerGroup`: Name of the Consumer group associated with Redis Stream +- `pendingEntriesCount`: Threshold for the number of `Pending Entries List`. This is the average target value to scale the workload. Defaults to `5` +- `enableTLS`: Set this to `true` if TLS connection to Redis is required. Defaults to `false` + +Some parameters could be provided using environmental variables, instead of setting them directly in metadata. Here is a list of parameters you can use to retrieve values from environment variables: + +- `addressesFromEnv`: The hosts and corresponding ports of Redis Cluster nodes, similar to `addresses`, but reads it from an environment variable on the scale target. +Name of the environment variable your deployment uses to get the URLs of Redis Cluster nodes. The resolved hosts should follow a format like `node1:6379, node2:6379, node3:6379 ...`. +- `hostsFromEnv`: The hosts of the Redis Cluster nodes, similar to `hosts`, but reads it from an environment variable on the scale target. +- `portsFromEnv`: The corresponding ports for the hosts of Redis Cluster nodes, similar to `ports`, but reads it from an environment variable on the scale target. + +### Authentication Parameters + +The scaler supports two modes of authentication: + +#### Using password authentication + +Use the `password` field in the `metadata` to specify the name of an environment variable that your deployment uses to get the Redis password. + +This is usually resolved from a `Secret V1` or a `ConfigMap V1` collections. `env` and `envFrom` are both supported. + +Here is an example: + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: redis-streams-scaledobject + namespace: default +spec: + scaleTargetRef: + name: redis-streams-consumer + pollingInterval: 15 + cooldownPeriod: 200 + maxReplicaCount: 25 + minReplicaCount: 1 + triggers: + - type: redis-cluster-streams + metadata: + addressesFromEnv: REDIS_ADDRESSES + passwordFromEnv: REDIS_PASSWORD # name of the environment variable in the Deployment + stream: my-stream + consumerGroup: consumer-group-1 + pendingEntriesCount: "10" +``` + +#### Using `TriggerAuthentication` + +You can use `TriggerAuthentication` CRD to configure the authentication. For example: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: redis-streams-password +type: Opaque +data: + redis_password: +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-redis-stream-triggerauth +spec: + secretTargetRef: + - parameter: password + name: redis-streams-password # name of the Secret + key: redis_password # name of the key in the Secret +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: redis-streams-scaledobject + namespace: default +spec: + scaleTargetRef: + name: redis-streams-consumer + pollingInterval: 15 + cooldownPeriod: 200 + maxReplicaCount: 25 + minReplicaCount: 1 + triggers: + - type: redis-cluster-streams + metadata: + address: node1:6379, node2:6379, node3:6379 + stream: my-stream + consumerGroup: consumer-group-1 + pendingEntriesCount: "10" + authenticationRef: + name: keda-redis-stream-triggerauth # name of the TriggerAuthentication resource +``` diff --git a/content/docs/2.3/scalers/redis-lists.md b/content/docs/2.3/scalers/redis-lists.md new file mode 100644 index 000000000..2ff44d6d1 --- /dev/null +++ b/content/docs/2.3/scalers/redis-lists.md @@ -0,0 +1,104 @@ ++++ +title = "Redis Lists" +layout = "scaler" +availability = "v1.0+" +maintainer = "Community" +description = "Scale applications based on Redis Lists." +go_file = "redis_scaler" ++++ + +### Trigger Specification + +This specification describes the `redis` trigger that scales based on the length of a list in Redis. + +```yaml +triggers: +- type: redis + metadata: + address: localhost:6379 # Format must be host:port + passwordFromEnv: REDIS_PASSWORD + listName: mylist # Required + listLength: "5" # Required + enableTLS: "false" # optional + databaseIndex: "0" # optional + # Alternatively, you can use existing environment variables to read configuration from: + # See details in "Parameter list" section + addressFromEnv: REDIS_HOST # Optional. You can use this instead of `address` parameter +``` + +**Parameter list:** + +- `address` - The host and port of the Redis server. +- `host` - The host of the Redis server. Alternative to `address` and requires `port` to be configured as well. +- `port` - The port of the Redis server. Alternative to `address` and requires `host` to be configured as well. +- `passwordFromEnv` - Environment variable to read the authentication password from to authenticate with the Redis server. + - Both the hostname and password fields need to be set to the names of the environment variables in the target deployment that contain the host name and password respectively. +- `listName` - Name of the Redis List that you want to monitor +- `listLength` - Average target value to trigger scaling actions +- `enableTLS` - If set to `true` allow a connection to a redis queue using tls, the default value for this parameter is false. +- `databaseIndex` - Index of Redis database to use. If not specified, the default value is 0 + +Some parameters could be provided using environmental variables, instead of setting them directly in metadata. Here is a list of parameters you can use to retrieve values from environment variables: + +- `addressFromEnv` - The host and port of the Redis server, similar to `address`, but reads it from an environment variable on the scale target. +- `hostFromEnv` - The host of the Redis server, similar to `host`, but reads it from an environment variable on the scale target. +- `portFromEnv` - The port of the Redis server, similar to `port`, but reads it from an environment variable on the scale target. + +### Authentication Parameters + +You can authenticate by using a password. + +**Connection Authentication:** + +- `address` - The hostname and port for the Redis server (host:port format). +- `host` - The hostname of the Redis server. If specified, the `port` should also be specified +- `port` - The port of the Redis server. If specified, the `host` should also be specified + +**Password Authentication:** + +- `password` - Redis password to authenticate with + +### Example + +Here is an example of how to deploy a scaled object with the `redis` scale trigger which uses `TriggerAuthentication`. + +You can also provide the `passwordFromEnv` on the `ScaledObject` directly. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: votes-db-secret + namespace: my-project +type: Opaque +data: + redis_password: YWRtaW4= +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-trigger-auth-redis-secret + namespace: my-project +spec: + secretTargetRef: + - parameter: password + name: votes-db-secret + key: redis_password +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: redis-scaledobject + namespace: my-project +spec: + scaleTargetRef: + name: votes + triggers: + - type: redis + metadata: + address: localhost:6379 + listName: mylist + listLength: "10" + authenticationRef: + name: keda-trigger-auth-redis-secret +``` diff --git a/content/docs/2.3/scalers/redis-streams.md b/content/docs/2.3/scalers/redis-streams.md new file mode 100644 index 000000000..3e8e0add9 --- /dev/null +++ b/content/docs/2.3/scalers/redis-streams.md @@ -0,0 +1,147 @@ ++++ +title = "Redis Streams" +layout = "scaler" +availability = "v1.5+" +maintainer = "Community" +description = "Scale applications based on Redis Streams." +go_file = "redis_streams_scaler" ++++ + +### Trigger Specification + +Redis 5.0 introduced [Redis Streams](https://redis.io/topics/streams-intro) which is an append-only log data structure. + +One of its features includes [`Consumer Groups`](https://redis.io/topics/streams-intro#consumer-groups), that allows a group of clients to co-operate consuming a different portion of the same stream of messages. + +This specification describes the `redis-streams` trigger that scales based on the *Pending Entries List* (see [`XPENDING`](https://redis.io/commands/xpending)) for a specific Consumer Group of a Redis Stream. + + +```yaml +triggers: +- type: redis-streams + metadata: + address: localhost:6379 # Required if host and port are not provided. Format - host:port + host: localhost # Required if address is not provided + port: "6379" # Required if address is not provided and host has been provided. + passwordFromEnv: REDIS_PASSWORD # optional (can also use authenticationRef) + stream: my-stream # Required - name of the Redis Stream + consumerGroup: my-consumer-group # Required - name of consumer group associated with Redis Stream + pendingEntriesCount: "10" # Required - number of entries in the Pending Entries List for the specified consumer group in the Redis Stream + enableTLS: "false" # optional + databaseIndex: "0" # optional + # Alternatively, you can use existing environment variables to read configuration from: + # See details in "Parameter list" section + addressFromEnv: REDIS_ADDRESS # Optional. You can use this instead of `address` parameter + hostFromEnv: REDIS_HOST # Optional. You can use this instead of `host` parameter + portFromEnv: REDIS_PORT # Optional. You can use this instead of `port` parameter +``` + +**Parameter list:** + +- `address`: The host and port of the Redis server in the format `host:port`, for example `my-redis:6379` + +> As an alternative to the `address` field, the user can specify `host` and `port` parameters. + +- `host`: The host of the Redis server + +> It is not required if `address` has been provided + +- `port`: The port of the Redis server + +> It is only to be used along with the `host`/`hostFromEnv` attribute and not required if `address` has been provided + +- `passwordFromEnv` (optional): Name of the environment variable your deployment uses to get the Redis password. + +- `stream`: Name of the Redis Stream +- `consumerGroup`: Name of the Consumer group associated with Redis Stream +- `pendingEntriesCount`: Threshold for the number of `Pending Entries List`. This is the average target value to scale the workload. Defaults to `5` +- `databaseIndex`: The Redis database index. Defaults to `0` if not specified +- `enableTLS`: Set this to `true` if TLS connection to Redis is required. Defaults to `false` + +Some parameters could be provided using environmental variables, instead of setting them directly in metadata. Here is a list of parameters you can use to retrieve values from environment variables: + +- `addressFromEnv`: The host and port of the Redis server, similar to `address`, but reads it from an environment variable on the scale target. +Name of the environment variable your deployment uses to get the Redis server URL. The resolved host should follow a format like `my-redis:6379`. +- `hostFromEnv`: The host of the Redis server, similar to `host`, but reads it from an environment variable on the scale target. +- `portFromEnv`: The port of the Redis server, similar to `port`, but reads it from an environment variable on the scale target. + +### Authentication Parameters + +The scaler supports two modes of authentication: + +#### Using password authentication + +Use the `password` field in the `metadata` to specify the name of an environment variable that your deployment uses to get the Redis password. + +This is usually resolved from a `Secret V1` or a `ConfigMap V1` collections. `env` and `envFrom` are both supported. + +Here is an example: + +```yaml +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: redis-streams-scaledobject + namespace: default +spec: + scaleTargetRef: + name: redis-streams-consumer + pollingInterval: 15 + cooldownPeriod: 200 + maxReplicaCount: 25 + minReplicaCount: 1 + triggers: + - type: redis-streams + metadata: + addressFromEnv: REDIS_HOST + passwordFromEnv: REDIS_PASSWORD # name of the environment variable in the Deployment + stream: my-stream + consumerGroup: consumer-group-1 + pendingEntriesCount: "10" +``` + +#### Using `TriggerAuthentication` + +You can use `TriggerAuthentication` CRD to configure the authentication. For example: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: redis-streams-password +type: Opaque +data: + redis_password: +--- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: keda-redis-stream-triggerauth +spec: + secretTargetRef: + - parameter: password + name: redis-streams-password # name of the Secret + key: redis_password # name of the key in the Secret +--- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: redis-streams-scaledobject + namespace: default +spec: + scaleTargetRef: + name: redis-streams-consumer + pollingInterval: 15 + cooldownPeriod: 200 + maxReplicaCount: 25 + minReplicaCount: 1 + triggers: + - type: redis-streams + metadata: + address: localhost:6379 + stream: my-stream + consumerGroup: consumer-group-1 + pendingEntriesCount: "10" + authenticationRef: + name: keda-redis-stream-triggerauth # name of the TriggerAuthentication resource +``` diff --git a/content/docs/2.3/troubleshooting.md b/content/docs/2.3/troubleshooting.md new file mode 100644 index 000000000..9bc0b05b1 --- /dev/null +++ b/content/docs/2.3/troubleshooting.md @@ -0,0 +1,6 @@ ++++ +title = "Troubleshooting" +description = "How to address commonly encountered KEDA issues" ++++ + +{{< troubleshooting >}}