Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI Failure (search victim assert) in ControllerEraseTest.test_erase_controller_log #8217

Closed
rystsov opened this issue Jan 13, 2023 · 11 comments · Fixed by #8400, #11350, #11970 or #16495
Closed

CI Failure (search victim assert) in ControllerEraseTest.test_erase_controller_log #8217

rystsov opened this issue Jan 13, 2023 · 11 comments · Fixed by #8400, #11350, #11970 or #16495
Assignees
@mmaslankaprv
Labels
area/controller ci-failure kind/bug Something isn't working sev/low Bugs which are non-functional paper cuts, e.g. typos, issues in log messages

Comments

@rystsov
Copy link
Contributor

rystsov commented Jan 13, 2023

https://buildkite.com/redpanda/redpanda/builds/21131#0185aa14-d78f-4589-83fe-33c79c1b9029

Module: rptest.tests.controller_erase_test
Class:  ControllerEraseTest
Method: test_erase_controller_log
Arguments:
{
  "partial": true
}

test_id:    rptest.tests.controller_erase_test.ControllerEraseTest.test_erase_controller_log.partial=True
status:     FAIL
run time:   32.670 seconds

    AssertionError()
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/controller_erase_test.py", line 99, in test_erase_controller_log
    assert self.redpanda.search_log_node(victim_node,
AssertionError
@rystsov rystsov added kind/bug Something isn't working ci-failure labels Jan 13, 2023
@rystsov rystsov mentioned this issue Jan 13, 2023
Merged
6 tasks
@piyushredpanda
Copy link
Contributor

@VadimPlh : can you please help with this?

@VadimPlh
Copy link
Contributor

Problem

Test delete last segment from disk and after startup node should signal that last_applied_offset does not present on the disk.
But in this test run node only replicated last controller log segment, but not applied it. So last_applied_offset was from another segment on the disk, which test did not delete.

The problem is that the test isn't waiting for a node's last_applied for the controller to advance far enough before restarting it.

Fix

Chatted with @jcsp and @mmaslankaprv
Idea is add new admin_api adding something like /v1/debug/controller_status that reports the controller log HWM and its last_applied. In test we should ask status for all nodes and wait when all of then will apply new offset
In future we can move sync point inside admin_api call. And node will be responsible for barrier

@VadimPlh VadimPlh mentioned this issue Jan 25, 2023
Merged
6 tasks
@dlex
Copy link
Contributor

dlex commented Jun 2, 2023 •
edited
Loading

Again in

@dlex dlex reopened this Jun 2, 2023
@dlex dlex mentioned this issue Jun 2, 2023
Merged
7 tasks
@bharathv bharathv mentioned this issue Jun 3, 2023
Merged
7 tasks
This was referenced Jun 3, 2023
Merged
Merged
@andijcr
Copy link
Contributor

andijcr commented Jun 5, 2023

https://buildkite.com/redpanda/redpanda/builds/30528#0188828a-83c4-4af4-8528-4cbeb9e6913a

@andijcr andijcr mentioned this issue Jun 5, 2023
Merged
7 tasks
@mmaslankaprv mmaslankaprv mentioned this issue Jun 6, 2023
Merged
7 tasks
@andrwng andrwng mentioned this issue Jun 6, 2023
Merged
7 tasks
@andijcr
Copy link
Contributor

andijcr commented Jun 7, 2023

https://buildkite.com/redpanda/redpanda/builds/30727#0188929a-b242-453e-a232-9b8b24f46c21

@rockwotj rockwotj mentioned this issue Jun 8, 2023
Merged
7 tasks
@dlex dlex mentioned this issue Jun 8, 2023
Merged
7 tasks
@VladLazar VladLazar mentioned this issue Jun 9, 2023
Merged
7 tasks
@andijcr
Copy link
Contributor

andijcr commented Jun 9, 2023

https://buildkite.com/redpanda/redpanda/builds/30865#01889b73-2349-4383-a96c-2ec39fe9ffae

@michael-redpanda
Copy link
Contributor

https://buildkite.com/redpanda/redpanda/builds/30962#0188a0ad-1dce-43f5-9250-c17af0d41a62

@michael-redpanda michael-redpanda mentioned this issue Jun 9, 2023
Merged
7 tasks
@twmb
Copy link
Contributor

twmb commented Jun 11, 2023

https://buildkite.com/redpanda/redpanda/builds/30990#0188a260-54c8-4595-8bcd-df3166d0f7ad

@twmb twmb mentioned this issue Jun 11, 2023
Merged
7 tasks
@mmaslankaprv mmaslankaprv added the sev/low Bugs which are non-functional paper cuts, e.g. typos, issues in log messages label Jul 10, 2023
@mmaslankaprv
Copy link
Member

The test selected segment which contained not applied data (dirty) after the segment was deleted there was no inconsistency in the data.

mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Jul 10, 2023
@mmaslankaprv
tests: more strict fix for last_applied_offset check
47a2c05 
Wait for all victim node records to be applied. If a victim node
contains some of the records that were not applied or about to be
truncated the test should wait before selecting segments to trim as in
the case if segment contains only dirty records removing it will not
cause inconsistency.

Fixes: redpanda-data#8217

Signed-off-by: Michal Maslanka <michal@redpanda.com>
@mmaslankaprv mmaslankaprv mentioned this issue Jul 10, 2023
Merged
7 tasks
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Jul 13, 2023
@mmaslankaprv
tests: wait for the victim node to apply the dirty offset
9d241e2 
Controller erasure test is supposed to validate if there is a mismatch
between the last appended entry in kvstore and controller max offset. In
order for the test to work correctly we must wait for all the messages
to be committed as we only delete the last segment that contains a
single message (new replicated configuration). In order to make the test
reliable change the condition to wait for the applied offset on the node
where controller log is going to be removed to be equal to the leader
dirty offset.

Fixes: redpanda-data#8217

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 57fb4c0)
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Jul 13, 2023
@mmaslankaprv
tests: wait for the victim node to apply the dirty offset
52ecf13 
Controller erasure test is supposed to validate if there is a mismatch
between the last appended entry in kvstore and controller max offset. In
order for the test to work correctly we must wait for all the messages
to be committed as we only delete the last segment that contains a
single message (new replicated configuration). In order to make the test
reliable change the condition to wait for the applied offset on the node
where controller log is going to be removed to be equal to the leader
dirty offset.

Fixes: redpanda-data#8217

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 57fb4c0)
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Jul 13, 2023
@mmaslankaprv
tests: wait for the victim node to apply the dirty offset
67666e6 
Controller erasure test is supposed to validate if there is a mismatch
between the last appended entry in kvstore and controller max offset. In
order for the test to work correctly we must wait for all the messages
to be committed as we only delete the last segment that contains a
single message (new replicated configuration). In order to make the test
reliable change the condition to wait for the applied offset on the node
where controller log is going to be removed to be equal to the leader
dirty offset.

Fixes: redpanda-data#8217

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 57fb4c0)
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Jul 20, 2023
@mmaslankaprv
tests: wait for the victim node to apply the dirty offset
a1c4ae5 
Controller erasure test is supposed to validate if there is a mismatch
between the last appended entry in kvstore and controller max offset. In
order for the test to work correctly we must wait for all the messages
to be committed as we only delete the last segment that contains a
single message (new replicated configuration). In order to make the test
reliable change the condition to wait for the applied offset on the node
where controller log is going to be removed to be equal to the leader
dirty offset.

Fixes: redpanda-data#8217

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 57fb4c0)
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Jul 20, 2023
@mmaslankaprv
tests: wait for the victim node to apply the dirty offset
33332c2 
Controller erasure test is supposed to validate if there is a mismatch
between the last appended entry in kvstore and controller max offset. In
order for the test to work correctly we must wait for all the messages
to be committed as we only delete the last segment that contains a
single message (new replicated configuration). In order to make the test
reliable change the condition to wait for the applied offset on the node
where controller log is going to be removed to be equal to the leader
dirty offset.

Fixes: redpanda-data#8217

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 57fb4c0)
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Jul 24, 2023
@mmaslankaprv
tests: more strict fix for last_applied_offset check
38feb81 
Wait for all victim node records to be applied. If a victim node
contains some of the records that were not applied or about to be
truncated the test should wait before selecting segments to trim as in
the case if segment contains only dirty records removing it will not
cause inconsistency.

Fixes: redpanda-data#8217

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit 47a2c05)
@andijcr andijcr mentioned this issue Jan 29, 2024
Merged
@mmaslankaprv mmaslankaprv reopened this Feb 6, 2024
@mmaslankaprv mmaslankaprv reopened this Feb 6, 2024
mmaslankaprv added a commit to mmaslankaprv/redpanda that referenced this issue Feb 6, 2024
@mmaslankaprv
tests: disable snapshots when testing partial controller deletion
f978777 
When testing partial deletion the test selects a segment to remove from
controller log. Before deleting the segment but after it was selected it
may be accounted in the controller snapshot.

Disabled controller snapshot to prevent the test racing with
snapshot creation.

Fixes: redpanda-data#8217

Signed-off-by: Michal Maslanka <michal@redpanda.com>
@mmaslankaprv mmaslankaprv mentioned this issue Feb 6, 2024
Merged
7 tasks
@piyushredpanda piyushredpanda mentioned this issue Feb 7, 2024
Merged
vbotbuildovich pushed a commit to vbotbuildovich/redpanda that referenced this issue Feb 9, 2024
@mmaslankaprv @vbotbuildovich
tests: disable snapshots when testing partial controller deletion
08c5bd9 
When testing partial deletion the test selects a segment to remove from
controller log. Before deleting the segment but after it was selected it
may be accounted in the controller snapshot.

Disabled controller snapshot to prevent the test racing with
snapshot creation.

Fixes: redpanda-data#8217

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit f978777)
@vbotbuildovich vbotbuildovich mentioned this issue Feb 9, 2024
Closed
@michael-redpanda michael-redpanda mentioned this issue Feb 28, 2024
Merged
@piyushredpanda piyushredpanda mentioned this issue Mar 18, 2024
Merged
@michael-redpanda michael-redpanda mentioned this issue Mar 28, 2024
Closed
@piyushredpanda piyushredpanda mentioned this issue Apr 16, 2024
Merged
@WillemKauf WillemKauf mentioned this issue May 13, 2024
Merged
7 tasks
@ztlpn ztlpn mentioned this issue May 16, 2024
Merged
vbotbuildovich pushed a commit to vbotbuildovich/redpanda that referenced this issue May 16, 2024
@mmaslankaprv @vbotbuildovich
tests: disable snapshots when testing partial controller deletion
d3cb9a6 
When testing partial deletion the test selects a segment to remove from
controller log. Before deleting the segment but after it was selected it
may be accounted in the controller snapshot.

Disabled controller snapshot to prevent the test racing with
snapshot creation.

Fixes: redpanda-data#8217

Signed-off-by: Michal Maslanka <michal@redpanda.com>
(cherry picked from commit f978777)
@vbotbuildovich vbotbuildovich mentioned this issue May 16, 2024
Closed
@michael-redpanda michael-redpanda mentioned this issue Jun 12, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmaslankaprv mmaslankaprv

Labels
area/controller ci-failure kind/bug Something isn't working sev/low Bugs which are non-functional paper cuts, e.g. typos, issues in log messages
Projects
None yet
Milestone
No milestone
10 participants
@andijcr @jcsp @twmb @rystsov @dlex @VadimPlh @VladLazar @mmaslankaprv @piyushredpanda @michael-redpanda