Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI Failure (Assert failure) in gtest_raft_rpunit.test_with_relaxed_acks #17731

Closed
ztlpn opened this issue Apr 10, 2024 · 1 comment · Fixed by #17895
Closed

CI Failure (Assert failure) in gtest_raft_rpunit.test_with_relaxed_acks #17731

ztlpn opened this issue Apr 10, 2024 · 1 comment · Fixed by #17895
Labels
area/replication ci-failure ci-ignore Automatic ci analysis tools ignore this issue ci-rca/redpanda CI Root Cause Analysis - Redpanda Issue kind/bug Something isn't working rpunit unit test ci-failure (not ducktape)

Comments

@ztlpn
Copy link
Contributor

ztlpn commented Apr 10, 2024 •
edited by github-actions bot
Loading

https://buildkite.com/redpanda/redpanda/builds/47599#018ec6f7-9571-436c-a0bb-5272c6f6e17d

ESC_bk;t=1712741728546^GERROR 2024-04-10 09:35:23,651 [shard 0:main] assert - Assert failure: (/var/lib/buildkite-agent/builds/buildkite-amd6
4-builders-i-0e0a361e6522fc09b-1/redpanda/redpanda/src/v/raft/tests/basic_raft_fixture_test.cc:360) 'o <= dirty_offset' last visible offset 1
091 on node 0 can not be larger than log end offset 603
ESC_bk;t=1712741728546^GERROR 2024-04-10 09:35:23,651 [shard 0:main] assert - Backtrace below:
ESC_bk;t=1712741728546^G0x7bc916 /vectorized/lib64/libseastar.so+0x20de4cb /vectorized/lib64/libseastar.so+0x20dffb8 /vectorized/lib64/libsea
star.so+0x20e0950 0xb1d79d 0xb1e064 /vectorized/lib64/libseastar.so+0x19bf00c /vectorized/lib64/libseastar.so+0x19c575c /vectorized/lib64/lib
seastar.so+0x19c8505 /vectorized/lib64/libseastar.so+0x19c68c0 /vectorized/lib64/libseastar.so+0x1770dd5 /vectorized/lib64/libseastar.so+0x17
6f16f /vectorized/lib64/libseastar_testing.so+0x6da6d /vectorized/lib64/libseastar.so+0x18b78c9 /lib64/libc.so.6+0x91016 /lib64/libc.so.6+0x1

JIRA Link: CORE-2213
@ztlpn ztlpn added kind/bug Something isn't working ci-failure ci-ignore Automatic ci analysis tools ignore this issue rpunit unit test ci-failure (not ducktape) labels Apr 10, 2024
@ztlpn ztlpn mentioned this issue Apr 10, 2024
Closed
@ztlpn ztlpn added the ci-rca/redpanda CI Root Cause Analysis - Redpanda Issue label Apr 12, 2024
@ztlpn
Copy link
Contributor Author

ztlpn commented Apr 12, 2024

The cause is a truncation event caused by a delayed append_entries request. We need to check that batches in the request are actually different to what the follower has in its log before truncating.

ztlpn added a commit to ztlpn/redpanda that referenced this issue Apr 16, 2024
@ztlpn
raft: in append_entries skip batches that we already have
09b8df7 
This is important for the case when we already have _all_ batches locally
(possible if e.g. the request was delayed/duplicated). In this case we don't
want to truncate, otherwise we might lose already committed data.

Fixes redpanda-data#17731
@ztlpn ztlpn mentioned this issue Apr 16, 2024
Merged
6 tasks
ztlpn added a commit to ztlpn/redpanda that referenced this issue Apr 17, 2024
@ztlpn
raft: in append_entries skip batches that we already have
49d13fd 
This is important for the case when we already have _all_ batches locally
(possible if e.g. the request was delayed/duplicated). In this case we don't
want to truncate, otherwise we might lose already committed data.

Fixes redpanda-data#17731
@ztlpn ztlpn mentioned this issue Apr 17, 2024
Merged
6 tasks
ztlpn added a commit to ztlpn/redpanda that referenced this issue Apr 18, 2024
@ztlpn
raft: in append_entries skip batches that we already have
f0c5772 
This is important for the case when we already have _all_ batches locally
(possible if e.g. the request was delayed/duplicated). In this case we don't
want to truncate, otherwise we might lose already committed data.

Fixes redpanda-data#17731
ztlpn added a commit to ztlpn/redpanda that referenced this issue May 16, 2024
@ztlpn
raft: in append_entries skip batches that we already have
84476c5 
This is important for the case when we already have _all_ batches locally
(possible if e.g. the request was delayed/duplicated). In this case we don't
want to truncate, otherwise we might lose already committed data.

Fixes redpanda-data#17731

(cherry picked from commit f0c5772)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/replication ci-failure ci-ignore Automatic ci analysis tools ignore this issue ci-rca/redpanda CI Root Cause Analysis - Redpanda Issue kind/bug Something isn't working rpunit unit test ci-failure (not ducktape)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

raft: in append_entries skip batches that we already have ztlpn/redpanda
2 participants
@ztlpn @rockwotj