Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI Failure (uneven distribution of group coordinators) in ConsumerGroupBalancingTest.test_coordinator_nodes_balance #9697

Closed
abhijat opened this issue Mar 29, 2023 · 17 comments · Fixed by #16775
Closed

CI Failure (uneven distribution of group coordinators) in ConsumerGroupBalancingTest.test_coordinator_nodes_balance #9697

abhijat opened this issue Mar 29, 2023 · 17 comments · Fixed by #16775
Labels
area/controller ci-failure kind/bug Something isn't working

Comments

@abhijat
Copy link
Contributor

abhijat commented Mar 29, 2023

https://buildkite.com/redpanda/redpanda/builds/25896#018726b5-759c-4f47-945f-75913c8092ad

Module: rptest.tests.consumer_group_balancing_test
Class:  ConsumerGroupBalancingTest
Method: test_coordinator_nodes_balance

The error thrown is related to uneven distribution of group coordinators, a relevant log before the error is:

[ERROR - 2023-03-28 06:13:54,299 - consumer_group_balancing_test - test_coordinator_nodes_balance - lineno:194]: Uneven distribution of group coordinators across nodes. coordinators of groups: {1: 2, 2: 1}, groups: 3, brokers: 3, coord_density: [2, 1], expected_coord_density: [1, 1, 1]

It looks like one node was assigned two groups instead of each node having one group to coordinate.

test_id:    rptest.tests.consumer_group_balancing_test.ConsumerGroupBalancingTest.test_coordinator_nodes_balance
--
  | status:     FAIL
  | run time:   22.018 seconds
  |  
  |  
  | RuntimeError('Uneven distribution of group coordinators across brokers')
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
  | data = self.run_test()
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
  | return self.test_context.function(self.test)
  | File "/root/tests/rptest/services/cluster.py", line 49, in wrapped
  | r = f(self, *args, **kwargs)
  | File "/root/tests/rptest/tests/consumer_group_balancing_test.py", line 200, in test_coordinator_nodes_balance
  | raise RuntimeError(
  | RuntimeError: Uneven distribution of group coordinators across brokers
@abhijat abhijat added kind/bug Something isn't working ci-failure labels Mar 29, 2023
@VladLazar VladLazar mentioned this issue Mar 30, 2023
Merged
7 tasks
@rystsov
Copy link
Contributor

rystsov commented Mar 30, 2023

Another instance: https://buildkite.com/redpanda/redpanda/builds/26073#01873143-ef88-400b-9d1b-da6075250ece

@rystsov rystsov changed the title CI Failure uneven distribution of group coordinators across brokers in ConsumerGroupBalancingTest.test_coordinator_nodes_balance CI Failure (uneven distribution of group coordinators) in ConsumerGroupBalancingTest.test_coordinator_nodes_balance Mar 30, 2023
@andijcr
Copy link
Contributor

andijcr commented Apr 3, 2023

https://buildkite.com/redpanda/redpanda/builds/26242#01873b74-085c-4f52-b5dc-50cbc62c0b8

FAIL test: ConsumerGroupBalancingTest.test_coordinator_nodes_balance (1/49 runs)
259   failure at 2023-04-01T07:34:49.453Z: RuntimeError('Uneven distribution of group coordinators across brokers')
260       on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/26242#01873b74-085c-4f52-b5dc-50cbc62c0b8

@jcsp
Copy link
Contributor

jcsp commented Apr 11, 2023

https://buildkite.com/redpanda/redpanda/builds/26781#01876cf3-396c-42d7-b15d-78673f41c858

@r-vasquez r-vasquez mentioned this issue Apr 12, 2023
Merged
3 tasks
@ztlpn ztlpn mentioned this issue Apr 19, 2023
Merged
7 tasks
@dotnwat
Copy link
Member

dotnwat commented Apr 19, 2023

https://buildkite.com/redpanda/redpanda/builds/27459#01879823-de7d-4a10-bcd2-0e6ff26c4370

This was referenced Apr 20, 2023
Merged
Merged
@dlex dlex mentioned this issue Apr 28, 2023
Merged
7 tasks
@ztlpn
Copy link
Contributor

ztlpn commented May 2, 2023

https://buildkite.com/redpanda/redpanda/builds/28380#0187db16-706b-4b37-9988-609b1e1dd45e

@BenPope
Copy link
Member

BenPope commented May 8, 2023

https://buildkite.com/redpanda/redpanda/builds/28707#0187f2cc-3c1a-44a0-bcfc-108b90a00ae5

@michael-redpanda
Copy link
Contributor

https://buildkite.com/redpanda/vtools/builds/7372#0187e21e-278f-41d2-b856-5ffb127922e2

@abhijat abhijat mentioned this issue May 12, 2023
Merged
7 tasks
@andijcr andijcr mentioned this issue May 18, 2023
Merged
7 tasks
@VladLazar
Copy link
Contributor 

FAIL test: ConsumerGroupBalancingTest.test_coordinator_nodes_balance (1/67 runs)
  failure at 2023-05-22T12:04:16.163Z: RuntimeError('Uneven distribution of group coordinators across brokers')
      on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/29566#01884312-0d99-44f6-91d3-64e17c51bc74

@NyaliaLui
Copy link
Contributor

https://buildkite.com/redpanda/redpanda/builds/29776#01884e0b-2c83-46e6-822d-ff5e703ed97c

@NyaliaLui
Copy link
Contributor 

FAIL test: ConsumerGroupBalancingTest.test_coordinator_nodes_balance (1/31 runs)
  failure at 2023-05-26T07:16:22.076Z: RuntimeError('Uneven distribution of group coordinators across brokers')
      on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/29955#018856ae-e95d-4020-a793-ab515874ccff

@dlex dlex mentioned this issue May 26, 2023
Merged
@dlex dlex mentioned this issue Jun 13, 2023
Merged
@ztlpn ztlpn mentioned this issue Jun 16, 2023
Merged
7 tasks
@travisdowns
Copy link
Member

FAIL test: ConsumerGroupBalancingTest.test_coordinator_nodes_balance (1/13 runs)
failure at 2023-06-20T16:15:26.302Z: RuntimeError('Uneven distribution of group coordinators across brokers')
on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/31650#0188d959-5be6-494e-9304-8f89c7e65e45

@BenPope BenPope mentioned this issue Jun 23, 2023
Merged
7 tasks
@dotnwat
Copy link
Member

dotnwat commented Jun 27, 2023

https://buildkite.com/redpanda/redpanda/builds/31994#0188fa5a-9d70-42ae-9d6c-f9a3fdf20449

@bharathv bharathv mentioned this issue Jun 28, 2023
Merged
7 tasks
@dotnwat
Copy link
Member

dotnwat commented Jun 28, 2023

https://buildkite.com/redpanda/redpanda/builds/32121#01890122-6689-48c1-ae80-341aabbf509d

@andijcr andijcr mentioned this issue Jun 30, 2023
Merged
6 tasks
@andrwng andrwng mentioned this issue Aug 11, 2023
Merged
7 tasks
@BenPope BenPope mentioned this issue Aug 16, 2023
Merged
@rockwotj rockwotj mentioned this issue Sep 6, 2023
Merged
7 tasks
@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/32377
*https://buildkite.com/redpanda/redpanda/builds/32926
*https://buildkite.com/redpanda/redpanda/builds/33082
*https://buildkite.com/redpanda/redpanda/builds/33091
*https://buildkite.com/redpanda/redpanda/builds/33100
*https://buildkite.com/redpanda/redpanda/builds/33206
*https://buildkite.com/redpanda/redpanda/builds/33619
*https://buildkite.com/redpanda/redpanda/builds/33673
*https://buildkite.com/redpanda/redpanda/builds/34284
*https://buildkite.com/redpanda/redpanda/builds/36208
*https://buildkite.com/redpanda/redpanda/builds/36337
*https://buildkite.com/redpanda/redpanda/builds/37262
*https://buildkite.com/redpanda/redpanda/builds/38396
*https://buildkite.com/redpanda/redpanda/builds/38436
*https://buildkite.com/redpanda/redpanda/builds/39034
*https://buildkite.com/redpanda/redpanda/builds/39530

@BenPope BenPope mentioned this issue Nov 13, 2023
Merged
@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/40574
*https://buildkite.com/redpanda/redpanda/builds/41067
*https://buildkite.com/redpanda/redpanda/builds/41371
*https://buildkite.com/redpanda/redpanda/builds/42164

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/43202

@vbotbuildovich
Copy link
Collaborator

*https://buildkite.com/redpanda/redpanda/builds/45098

mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Feb 28, 2024
@mmaslankaprv
tests: refactor consumer group balancing test
83bda49 
Refactor consumer group balancing test to directly validate
`__consumer_offsets` partition replicas placement instead of validating
consumer groups coordinators.

Fixes: redpanda-data#9697

Signed-off-by: Michal Maslanka <michal@redpanda.com>
@mmaslankaprv mmaslankaprv mentioned this issue Feb 28, 2024
Merged
7 tasks
@travisdowns travisdowns mentioned this issue Mar 15, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/controller ci-failure kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

tests: refactor consumer group balancing test mmaslankaprv/redpanda