Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(consumer): support multiple balance strategies #2339

Merged
merged 1 commit into from
Sep 27, 2022
Merged

feat(consumer): support multiple balance strategies #2339

merged 1 commit into from
Sep 27, 2022

Conversation

napallday
Copy link
Contributor

As shown below, Kafka supports specifying multiple rebalance protocol from the very begining. (source)

v0 supported in 0.9.0.0 and greater
JoinGroupRequest => GroupId SessionTimeout MemberId ProtocolType GroupProtocols
  GroupId => string
  SessionTimeout => int32
  MemberId => string
  ProtocolType => string
  GroupProtocols => [ProtocolName ProtocolMetadata]
    ProtocolName => string
    ProtocolMetadata => bytes
 
v1 supported in 0.10.1.0 and greater
JoinGroupRequest => GroupId SessionTimeout MemberId ProtocolType GroupProtocols
  GroupId => string
  SessionTimeout => int32
  RebalanceTimeout => int32
  MemberId => string
  ProtocolType => string
  GroupProtocols => [ProtocolName ProtocolMetadata]
    ProtocolName => string
    ProtocolMetadata => bytes

However, Sarama only supports one group protocol right now, by specifying Consumer.Group.Rebalance.Strategy. This prevents users from enjoying some benefits in Kafka. For example, if users wanna change the default one range to a more advanced one like sticky, when the new consumers with sticky protocol try to join in the consumer group, they will be rejected by the coordinator with error ErrInconsistentGroupProtocol(kafka server: The provider group protocol type is incompatible with the other members) and thus cannot consume any messages. Only when all the instances are upgraded with sticky protocol can they resume consume message again.

In contrast, the client supporting multiple group protocols enables rolling upgrades without downtime -- The upgraded member includes both the new protocol and the protocol, and the coordinator will choose a single protocol which all members support. Once all members have upgraded, the coordinator will choose whichever protocol is listed first in the GroupProtocols array.

In this PR, I introduce a new config Consumer.Group.Rebalance.GroupStrategies and mark the old one Consumer.Group.Rebalance.Strategy deprecated. To make it backward compatible:

  • The dafault value for Consumer.Group.Rebalance.GroupStrategies is []{BalanceStrategyRange}
  • Consumer.Group.Rebalance.Strategy takes precedence of Consumer.Group.Rebalance.GroupStrategies

Thus,

  • if Consumer.Group.Rebalance.Strategy isn't explicitly configured previously, then the new default value range in Consumer.Group.Rebalance.GroupStrategies will be used.
  • if Consumer.Group.Rebalance.Strategy is configured by users previously, Consumer.Group.Rebalance.Strategy is also chosen as the group protocol.

I believe it's a desirable feature in sarama, since other Go Kafka Libraries, like segmentio/kafka-go, already support it.

@dnwe
Copy link
Collaborator

dnwe commented Sep 27, 2022

@Jacob-bzx thanks for your great contribution!

@dnwe dnwe merged commit 5e2c2ef into IBM:main Sep 27, 2022
remicalixte pushed a commit to remicalixte/sarama that referenced this pull request Dec 1, 2023
…-balance-strategies

feat(consumer): support multiple balance strategies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants