Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] unbalanced assignment caused by hash collision in ConsistentHashAllocateStrategy.java #113

Open
humkum opened this issue Mar 22, 2024 · 1 comment
Assignees

Comments

@humkum
Copy link
Contributor

humkum commented Mar 22, 2024

In current consistent hash allocate strategy, we will get unbalanced assignment caused by hash collision.
For example, we have a topic named TopicTest, the topic allocate at 3 brokers: broker0, broker1, broker2, 10 queues per broker, and we have 32 task managers. The allocation result using the ConsistentHashAllocateStrategy as follows:

{7=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=0], 8=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=1], 9=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=2], 10=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=3], 11=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=4], 12=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=5], 13=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=6], 14=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=7], 15=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=8], 16=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=9], 17=MessageQueue [topic=TopicTest, brokerName=broker1, queueId=9], 18=MessageQueue [topic=TopicTest, brokerName=broker0, queueId=9]}
allocation details, format is taskIndex=allocate num:
{7=1, 8=2, 9=3, 10=3, 11=3, 12=3, 13=3, 14=3, 15=3, 16=3, 17=2, 18=1}

We have 32 task manager, but only 12 task manager can work normally, which means we have 20 wasted task manager.

We should resolve the hash collision, I'm glad for your suggestion.

@humkum humkum changed the title [Bug] [Bug] unbalanced assignment caused by hash collision in ConsistentHashAllocateStrategy.java Mar 22, 2024
@lizhimins lizhimins assigned lizhimins and humkum and unassigned lizhimins Mar 27, 2024
@lizhimins
Copy link
Member

可以修改或者增加策略,也可以像 rocketmq 中 (https://github.com/apache/rocketmq) 的 org.apache.rocketmq.client.consumer.rebalance.AllocateMessageQueueAveragely。
如果用了类似 avg 或者 circle 的策略,需要额外处理下分区减少的情况,采用一致性 hash 基本只需要考虑增加分区。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants