Skip to content

Commit

Permalink
cluster: consider shard0 reserve in check_cluster_limits
Browse files Browse the repository at this point in the history
Improve the user error feedback when the
`topic_partitions_reserve_shard0` cluster config is used and a user
tried to allocate a topic that is above the partition limits.

Previously this check was only considered as part of the
`max_final_capacity` hard constraint, which meant that the kafka error
message was more vague (No nodes are available to perform allocation
after hard constraints were solved) and there were no clear broker logs
to indicate this.

Now this is also considered inside `check_cluster_limits` which leads to
more specific error messages on both the kafka api (unable to create
topic with 20 partitions due to hardware constraints) and in broker
logs:

```
WARN  2024-11-29 13:18:13,907 [shard 0:main] cluster - partition_allocator.cc:183 - Refusing to create 20 partitions as total partition count 20 would exceed the core-based limit 18 (per-shard limit: 20, shard0 reservation: 2)
```

(cherry picked from commit b632190)
  • Loading branch information
pgellert authored and vbotbuildovich committed Dec 5, 2024
1 parent 98620aa commit 7a70496
Showing 1 changed file with 7 additions and 3 deletions.
10 changes: 7 additions & 3 deletions src/v/cluster/scheduling/partition_allocator.cc
Original file line number Diff line number Diff line change
Expand Up @@ -169,15 +169,19 @@ std::error_code partition_allocator::check_cluster_limits(

// Refuse to create a partition count that would violate the per-core
// limit.
const uint64_t core_limit = (effective_cpu_count * _partitions_per_shard());
const uint64_t core_limit = (effective_cpu_count * _partitions_per_shard())
- (broker_count * _partitions_reserve_shard0());
if (proposed_total_partitions > core_limit) {
vlog(
clusterlog.warn,
"Refusing to create {} partitions as total partition count {} would "
"exceed core limit {}",
"exceed the core-based limit {} (per-shard limit: {}, shard0 "
"reservation: {})",
new_partitions_replicas_requested,
proposed_total_partitions,
effective_cpu_count * _partitions_per_shard());
core_limit,
_partitions_per_shard(),
_partitions_reserve_shard0());
return errc::topic_invalid_partitions_core_limit;
}

Expand Down

0 comments on commit 7a70496

Please sign in to comment.