Client throughput quota utilization (#592)

Co-authored-by: Paulo Borges <paulohtb@hotmail.com>
Co-authored-by: Jake Cahill <45230295+JakeSCahill@users.noreply.github.com>
Co-authored-by: Michele Cyran <michele@redpanda.com>
Co-authored-by: Joyce Fee <102751339+Feediver1@users.noreply.github.com>
Co-authored-by: Mike Boquard <michael@redpanda.com>
Co-authored-by: tris0laris <57298792+tris0laris@users.noreply.github.com>
Co-authored-by: Dave Voutila <voutilad@gmail.com>
Co-authored-by: Angela Simms <102690377+asimms41@users.noreply.github.com>
Co-authored-by: Andrew Hsu <xuzuan@gmail.com>
Co-authored-by: Oren Leiman <mumblemumble777@gmail.com>
Co-authored-by: Gellért Peresztegi-Nagy <pereszteginagy.gellert@gmail.com>
Co-authored-by: Rogger Vasquez <59714880+r-vasquez@users.noreply.github.com>

modules/develop/pages/kafka-clients.adoc

@@ -47,6 +47,6 @@ Redpanda does not currently support the following Apache Kafka features:
 
 * Managing SASL users with Kafka APIs: DescribeUserSCRAMCredentialsRequest and AlterUserSCRAMCredentialsRequest APIs.
 * HTTP Proxy (pandaproxy): Unlike other REST proxy implementations in the Kafka ecosystem, Redpanda HTTP Proxy does not support topic and ACLs CRUD through the HTTP Proxy. HTTP Proxy is designed for clients producing and consuming data that do not perform administrative functions.
-* Quotas per user for bandwidth and API request rates (though xref:manage:cluster-maintenance/manage-throughput.adoc#client-throughput-limits[quotas per client and per client group (per CPU core)] are supported).
+* Quotas per user for bandwidth and API request rates. However, xref:manage:cluster-maintenance/manage-throughput.adoc#client-throughput-limits[quotas per client and per client group] using AlterClientQuotas and DescribeClientQuotas APIs are supported.
 
 If you have any issues while working with a Kafka tool, you can https://github.com/redpanda-data/redpanda/issues/new[file an issue^].

modules/get-started/pages/whats-new.adoc

@@ -25,6 +25,13 @@ You can now deploy data transform functions that xref:develop:data-transforms/de
 
 The docs now also include an xref:develop:data-transforms/index.adoc[expanded guide] designed to help you master the creation, deployment, and management of data transforms in Redpanda.
 
+
+== Client throughput management
+
+Redpanda now includes `rpk` and Redpanda Console support for managing xref:manage:cluster-maintenance/manage-throughput.adoc#client-throughput-limits[client throughput limits]. You can set client throughput quotas using `rpk` for an individual client based on client ID, as well as for a group of clients based on ID prefix matching. New xref:manage:cluster-maintenance/manage-throughput.adoc#monitor-client-throughput[metrics] are available for insights on client throughput utilization.
+
+Client throughput quotas, previously applied on a per-shard basis, now apply on a per-broker basis. Cluster configuration properties for managing client quotas are xref:upgrade:deprecated/index.adoc[deprecated], including `target_quota_byte_rate` which is disabled by default with the value `0`.
+
 == Next steps
 
 * xref:install-beta.adoc[]

modules/reference/pages/internal-metrics-reference.adoc

@@ -87,9 +87,35 @@ Current effective quota for the quota balancer, in bytes per second.
 
 ---
 
+=== vectorized_kafka_quotas_client_quota_throttle_time
+
+Client quota throttling delay, in seconds, per rule and quota type based on xref:manage:cluster-maintenance/manage-throughput.adoc#client-throughput-limits[client throughput limits].
+
+*Type*: histogram
+
+*Labels*:
+
+* `quota_rule=("not_applicable" | "kafka_client_default" | "cluster_client_default" | "kafka_client_prefix" | "cluster_client_prefix" | "kafka_client_id")`
+* `quota_type=("produce_quota" | "fetch_quota" | "partition_mutation_quota")`
+
+---
+
+=== vectorized_kafka_quotas_client_quota_throughput
+
+Client quota throughput per rule and quota type.
+
+*Type*: histogram
+
+*Labels*:
+
+* `quota_rule=("not_applicable" | "kafka_client_default" | "cluster_client_default" | "kafka_client_prefix" | "cluster_client_prefix" | "kafka_client_id")`
+* `quota_type=("produce_quota" | "fetch_quota" | "partition_mutation_quota")`
+
+---
+
 === vectorized_kafka_quotas_throttle_time
 
-Histogram of throttle times, in seconds.
+Histogram of throttle times, in seconds, based on xref:manage:cluster-maintenance/manage-throughput.adoc#broker-wide-throughput-limits[broker-wide throughput limits].
 
 *Type*: histogram
 

modules/reference/pages/properties/cluster-properties.adoc

@@ -1674,30 +1674,6 @@ Time between cluster join retries in milliseconds.
 
 ---
 
-=== kafka_admin_topic_api_rate
-
-Target quota rate for partition mutations per <<default_window_sec,`default_window_sec`>>. If `null`, the property is disabled, and no quota rate is applied.
-
-*Unit*: partition mutations per `default_window_second`
-
-*Related properties*:
-
-* <<default_window_sec,`default_window_sec`>>
-
-*Requires restart:* No
-
-*Optional:* No
-
-*Visibility:* `user`
-
-*Type:* integer
-
-*Accepted values:* [`0`, `4294967295`]
-
-*Default:* `null`
-
----
-
 === kafka_batch_max_bytes
 
 Maximum size of a batch processed by the server. If the batch is compressed, the limit applies to the compressed batch size.
@@ -1716,58 +1692,6 @@ Maximum size of a batch processed by the server. If the batch is compressed, the
 
 ---
 
-=== kafka_client_group_byte_rate_quota
-
-A map specifying the produce-rate quota per client group. The configurable fields:
-
-Example: `([{'group_name': 'first_group','clients_prefix': 'group_1','quota': 10240}])`
-
-*Related topics*:
-
-* xref:manage:cluster-maintenance/manage-throughput.adoc#client-group-throughput-limits[Client group throughput limits]
-
-*Requires restart:* No
-
-*Optional:* Yes
-
-*Visibility:* `user`
-
-*Accepted Values:*
-
-* `group_name`: Name of a client group
-* `clients_prefix`: Prefix to prepend to the name of each client belonging to the group specified by `group_name`
-* `quota`: Produce-rate quota of each client in bytes per second
-
-*Default:* `null`
-
----
-
-=== kafka_client_group_fetch_byte_rate_quota
-
-A map specifying the fetch-rate quota per client group. The configurable fields:
-
-Example: `([{'group_name': 'first_group','clients_prefix': 'group_1','quota': 10240}])`
-
-*Related topics*:
-
-* xref:manage:cluster-maintenance/manage-throughput.adoc#client-group-throughput-limits[Client group throughput limits]
-
-*Requires restart:* No
-
-*Optional:* Yes
-
-*Visibility:* `user`
-
-*Accepted Values:*
-
-* `group_name`: Name of a client group
-* `clients_prefix`: Prefix to prepend to the name of each client belonging to the group specified by `group_name`
-* `quota`: Produce-rate quota of each client in bytes per second
-
-*Default:* `null`
-
----
-
 === kafka_connection_rate_limit
 
 Maximum connections per second for one core. If `null` (the default), then the number of connections per second is unlimited.
@@ -5236,54 +5160,6 @@ List of superuser usernames.
 
 ---
 
-=== target_fetch_quota_byte_rate
-
-Target fetch-size quota byte rate. If `null`, the property is disabled, and no quota byte rate is applied.
-
-*Unit*: bytes per second
-
-*Requires restart:* No
-
-*Optional:* No
-
-*Visibility:* `user`
-
-*Type:* integer
-
-*Accepted values:* [`0`, `4294967295`]
-
-*Default:* `null`
-
----
-
-=== target_quota_byte_rate
-
-Target quota byte rate.
-
-The `target_quota_byte_rate` property applies to a producer client that isn't a member of a client group configured by <<kafka_client_group_byte_rate_quota,`kafka_client_group_byte_rate_quota`>>. It sets the maximum throughput quota of a client sending to a Redpanda broker node.
-
-When set to 0, this property is disabled. 
-
-*Unit*: bytes per second
-
-*Requires restart:* No
-
-*Optional:* Yes
-
-*Visibility:* `user`
-
-*Type:* integer
-
-*Accepted values:* [`0`, `4294967295`]
-
-*Default:* `0`
-
-*Related topics*:
-
-* xref:manage:cluster-maintenance/manage-throughput.adoc#client-throughput-limits[Client throughput limits]
-
----
-
 === tm_sync_timeout_ms
 
 Transaction manager's synchronization timeout. Maximum time to wait for internal state machine to catch up before rejecting a request.

modules/reference/pages/public-metrics-reference.adoc

@@ -497,6 +497,32 @@ Latency of produce/consume requests per broker. This duration measures from when
 
 ---
 
+=== redpanda_kafka_quotas_client_quota_throttle_time
+
+Client quota throttling delay, in seconds, per rule and quota type.
+
+*Type*: histogram
+
+*Labels*:
+
+* `quota_rule=("not_applicable" | "kafka_client_default" | "cluster_client_default" | "kafka_client_prefix" | "cluster_client_prefix" | "kafka_client_id")``
+* `quota_type=("produce_quota" | "fetch_quota" | "partition_mutation_quota")`
+
+---
+
+=== redpanda_kafka_quotas_client_quota_throughput
+
+Client quota throughput per rule and quota type.
+
+*Type*: histogram
+
+*Labels*:
+
+* `quota_rule=("not_applicable" | "kafka_client_default" | "cluster_client_default" | "kafka_client_prefix" | "cluster_client_prefix" | "kafka_client_id")`
+* `quota_type=("produce_quota" | "fetch_quota" | "partition_mutation_quota")`
+
+---
+
 == Consumer group metrics
 
 === redpanda_kafka_consumer_group_committed_offset

modules/upgrade/pages/deprecated/index.adoc

@@ -9,9 +9,19 @@ This index helps you to identify deprecated features in Redpanda releases and pl
 |===
 | Deprecated in  | Feature | Details
 
+| 24.2
+a| Client throughput quota cluster configuration:
+
+- `kafka_client_group_byte_rate_quota`
+- `kafka_client_group_fetch_byte_rate_quota`
+- `target_quota_byte_rate`
+- `target_fetch_quota_byte_rate`
+- `kafka_admin_topic_api_rate`
+| Use xref:reference:rpk/rpk-cluster/rpk-cluster-quotas.adoc[`rpk cluster quotas`] to manage xref:manage:cluster-maintenance/manage-throughput.adoc#client-throughput-limits[client throughput limits] based on the Kafka API.
+
 | 23.3.2
 | `space_management_enable_override`
-| This property was a workaround in older version of Redpanda. It is no longer required and the property is ignored.
+| This property was a workaround in an older version of Redpanda. It is no longer required and the property is ignored.
 
 | 23.3.1
 | xref:reference:cluster-properties.adoc#delete_retention_ms[`delete_retention_ms`]
@@ -189,6 +199,12 @@ This is an exhaustive list of all the deprecated properties.
 
 - id_allocator_replication
 
+- kafka_admin_topic_api_rate
+
+- kafka_client_group_byte_rate_quota
+
+- kafka_client_group_fetch_byte_rate_quota
+
 - max_version
 
 - min_version
@@ -203,6 +219,10 @@ This is an exhaustive list of all the deprecated properties.
 
 - seq_table_min_size
 
+- target_fetch_quota_byte_rate
+
+- target_quota_byte_rate
+
 - tm_violation_recovery_policy
 
 - transaction_coordinator_replication

modules/upgrade/pages/rolling-upgrade.adoc

@@ -21,6 +21,8 @@ include::partial$rolling-upgrades/upgrade-limitations.adoc[]
 
 === Review incompatible changes
 
+Starting in version 24.2, xref:manage:cluster-maintenance/manage-throughput.adoc#client-throughput-limits[client throughput limits] are compatible with the https://cwiki.apache.org/confluence/display/KAFKA/KIP-546%3A+Add+Client+Quota+APIs+to+the+Admin+Client[AlterClientQuotas and DescribeClientQuotas^] Kafka APIs. Redpanda determines client throughput limits on a per-broker basis. In earlier versions, client throughput quotas were applied from cluster configurations on a per-shard basis. 
+
 Patch releases in https://github.com/redpanda-data/redpanda/discussions/9522[22.3.14^] and https://github.com/redpanda-data/redpanda/discussions/9523[23.1.2^] changed the behavior when remote read is disabled and the requested Raft term falls below the local log's beginning. In earlier versions, Redpanda returned an offset -1. With the patch, when you request a value older than the lowest offset, Redpanda returns the lowest offset, not -1.
 
 == Find a new version