Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI Failure (Consumer failed to consume up to offsets) in SimpleEndToEndTest.test_consumer_interruption #11459

Closed
michael-redpanda opened this issue Jun 15, 2023 · 1 comment · Fixed by #11484
Assignees
Labels
ci-failure kind/bug Something isn't working

Comments

@michael-redpanda
Copy link
Contributor

https://buildkite.com/redpanda/redpanda/builds/31278#0188ba00-ddd0-4598-8716-3479f8281fb5
https://buildkite.com/redpanda/redpanda/builds/31243#0188b813-7347-4912-8b77-0d8907c94503

Module: rptest.tests.simple_e2e_test
Class:  SimpleEndToEndTest
Method: test_consumer_interruption
test_id:    rptest.tests.simple_e2e_test.SimpleEndToEndTest.test_consumer_interruption
status:     FAIL
run time:   5 minutes 35.610 seconds


    TimeoutError("Consumer failed to consume up to offsets {TopicPartition(topic='topic-qdwowrffwq', partition=0): 77960} after waiting 300s, last committed offsets: {1: {TopicPartition(topic='topic-qdwowrffwq', partition=0): 60836}}.")
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 83, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/simple_e2e_test.py", line 63, in test_consumer_interruption
    self.run_validation(min_records=100000,
  File "/root/tests/rptest/tests/end_to_end.py", line 275, in run_validation
    self.run_consumer_validation(
  File "/root/tests/rptest/tests/end_to_end.py", line 296, in run_consumer_validation
    self.await_consumed_offsets(last_acked_offsets,
  File "/root/tests/rptest/tests/end_to_end.py", line 227, in await_consumed_offsets
    wait_until(
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Consumer failed to consume up to offsets {TopicPartition(topic='topic-qdwowrffwq', partition=0): 77960} after waiting 300s, last committed offsets: {1: {TopicPartition(topic='topic-qdwowrffwq', partition=0): 60836}}.
@mmaslankaprv
Copy link
Member

This is the test setup issue

mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Jun 16, 2023
Previously the producer used high throughput which may lead to the
situation in which consumer continued without a gap in the offset space
even after the topic was deleted.
The producer might have already produced enough records to fill the gap
and allow consumer to continue where it left of.

Using lower producer throughput and waiting for large number of records
to be consumed before deleting topic will prevent this situation from
happening.

Fixes: redpanda-data#11459

Signed-off-by: Michal Maslanka <michal@redpanda.com>
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Jul 24, 2023
Previously the producer used high throughput which may lead to the
situation in which consumer continued without a gap in the offset space
even after the topic was deleted.
The producer might have already produced enough records to fill the gap
and allow consumer to continue where it left of.

Using lower producer throughput and waiting for large number of records
to be consumed before deleting topic will prevent this situation from
happening.

Fixes: redpanda-data#11459

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 7f848d0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-failure kind/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants