CI Failure (Internal Server Error) in `EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures` #14898

StephanDollberg · 2023-11-10T17:52:58Z

https://buildkite.com/redpanda/redpanda/builds/40810

Module: rptest.tests.e2e_shadow_indexing_test
Class: EndToEndShadowIndexingTestWithDisruptions
Method: test_write_with_node_failures
Arguments: {
    "cloud_storage_type": 1
}

test_id:    EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures
status:     FAIL
run time:   124.586 seconds

HTTPError('500 Server Error: Internal Server Error for url: http://docker-rp-24:9644/v1/cloud_storage/reset_scrubbing_metadata/kafka/__consumer_offsets/1')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 269, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 159, in wrapped
    self.redpanda.maybe_do_internal_scrub()
  File "/root/tests/rptest/services/redpanda.py", line 3880, in maybe_do_internal_scrub
    results = self.wait_for_internal_scrub(cloud_partitions)
  File "/root/tests/rptest/services/redpanda.py", line 3985, in wait_for_internal_scrub
    self._admin.reset_scrubbing_metadata(
  File "/root/tests/rptest/services/admin.py", line 1131, in reset_scrubbing_metadata
    return self._request(
  File "/root/tests/rptest/services/admin.py", line 363, in _request
    r.raise_for_status()
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://docker-rp-24:9644/v1/cloud_storage/reset_scrubbing_metadata/kafka/__consumer_offsets/1

JIRA Link: CORE-1573

The text was updated successfully, but these errors were encountered:

When `persisted_stm::sync()` method fails it is indicating that the current node is not longer a leader. The `sync()` executed before `replicate` call in archival stm `command_batch_builder` prevents replicate from being called. The end result for such an error is deterministic and we can translate the sync error to `not_leader` error code. Fixes: redpanda-data#14898 Signed-off-by: Michal Maslanka <michal@redpanda.com>

When `persisted_stm::sync()` method fails it is indicating that the current node is not longer a leader. The `sync()` executed before `replicate` call in archival stm `command_batch_builder` prevents replicate from being called. The end result for such an error is deterministic and we can translate the sync error to `not_leader` error code. Fixes: redpanda-data#14898 Signed-off-by: Michal Maslanka <michal@redpanda.com> (cherry picked from commit 4dfdc53)

abhijat · 2023-11-20T15:27:08Z

seen again in https://buildkite.com/redpanda/redpanda/builds/41402#018beb54-caf5-405c-b535-767865afcff5

====================================================================================================
test_id:    rptest.tests.e2e_shadow_indexing_test.EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures.cloud_storage_type=CloudStorageType.ABS
status:     FAIL
run time:   2 minutes 7.351 seconds


    HTTPError('500 Server Error: Internal Server Error for url: http://docker-rp-4:9644/v1/cloud_storage/reset_scrubbing_metadata/kafka/__consumer_offsets/0')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 269, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 159, in wrapped
    self.redpanda.maybe_do_internal_scrub()
  File "/root/tests/rptest/services/redpanda.py", line 3917, in maybe_do_internal_scrub
    results = self.wait_for_internal_scrub(cloud_partitions)
  File "/root/tests/rptest/services/redpanda.py", line 4022, in wait_for_internal_scrub
    self._admin.reset_scrubbing_metadata(
  File "/root/tests/rptest/services/admin.py", line 1145, in reset_scrubbing_metadata
    return self._request(
  File "/root/tests/rptest/services/admin.py", line 363, in _request
    r.raise_for_status()
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://docker-rp-4:9644/v1/cloud_storage/reset_scrubbing_metadata/kafka/__consumer_offsets/0

vbotbuildovich · 2023-12-13T20:21:41Z

*https://buildkite.com/redpanda/redpanda/builds/41747

vbotbuildovich · 2024-01-27T01:40:51Z

*https://buildkite.com/redpanda/redpanda/builds/44340

vbotbuildovich · 2024-01-30T00:15:07Z

*https://buildkite.com/redpanda/redpanda/builds/44413

vbotbuildovich · 2024-02-01T02:35:16Z

*https://buildkite.com/redpanda/redpanda/builds/44517

vbotbuildovich · 2024-02-02T00:13:12Z

*https://buildkite.com/redpanda/redpanda/builds/44580

vbotbuildovich · 2024-02-07T00:12:21Z

*https://buildkite.com/redpanda/redpanda/builds/44739

vbotbuildovich · 2024-02-18T00:13:43Z

*https://buildkite.com/redpanda/redpanda/builds/45088
*https://buildkite.com/redpanda/redpanda/builds/45089

vbotbuildovich · 2024-02-28T00:16:21Z

*https://buildkite.com/redpanda/redpanda/builds/45388

vbotbuildovich · 2024-03-04T18:34:28Z

*https://buildkite.com/redpanda/redpanda/builds/45546
*https://buildkite.com/redpanda/redpanda/builds/45585
*https://buildkite.com/redpanda/redpanda/builds/45605

vbotbuildovich · 2024-03-14T00:17:03Z

*https://buildkite.com/redpanda/redpanda/builds/46054#018e330b-421b-442b-a985-674ca018a46c

vbotbuildovich · 2024-03-18T21:15:27Z

*https://buildkite.com/redpanda/redpanda/builds/46389

vbotbuildovich · 2024-03-20T21:14:25Z

*https://buildkite.com/redpanda/redpanda/builds/46479

vbotbuildovich · 2024-03-29T04:41:22Z

*https://buildkite.com/redpanda/redpanda/builds/46966
*https://buildkite.com/redpanda/redpanda/builds/47006

vbotbuildovich · 2024-04-03T07:32:46Z

*https://buildkite.com/redpanda/redpanda/builds/47263

vbotbuildovich · 2024-04-03T07:33:37Z

*https://buildkite.com/redpanda/redpanda/builds/47263

vbotbuildovich · 2024-04-03T21:15:59Z

*https://buildkite.com/redpanda/redpanda/builds/47266

vbotbuildovich · 2024-04-04T21:17:14Z

*https://buildkite.com/redpanda/redpanda/builds/47353
*https://buildkite.com/redpanda/redpanda/builds/47374

vbotbuildovich · 2024-04-10T17:28:02Z

*https://buildkite.com/redpanda/redpanda/builds/47463

vbotbuildovich · 2024-04-22T23:00:34Z

*https://buildkite.com/redpanda/redpanda/builds/48074

vbotbuildovich · 2024-05-04T21:14:46Z

*https://buildkite.com/redpanda/vtools/builds/13484

vbotbuildovich · 2024-05-06T21:12:52Z

*https://buildkite.com/redpanda/redpanda/builds/48734

vbotbuildovich · 2024-05-11T21:15:22Z

*https://buildkite.com/redpanda/redpanda/builds/48966
*https://buildkite.com/redpanda/redpanda/builds/48963

vbotbuildovich · 2024-05-13T21:14:00Z

*https://buildkite.com/redpanda/redpanda/builds/48995

vbotbuildovich · 2024-06-06T21:07:31Z

*https://buildkite.com/redpanda/redpanda/builds/49943

vbotbuildovich · 2024-06-11T21:12:32Z

*https://buildkite.com/redpanda/redpanda/builds/48074
*https://buildkite.com/redpanda/vtools/builds/13484
*https://buildkite.com/redpanda/redpanda/builds/48734
*https://buildkite.com/redpanda/redpanda/builds/48963
*https://buildkite.com/redpanda/redpanda/builds/48966
*https://buildkite.com/redpanda/redpanda/builds/48995
*https://buildkite.com/redpanda/redpanda/builds/49943

vbotbuildovich · 2024-06-11T21:29:40Z

*https://buildkite.com/redpanda/redpanda/builds/48074
*https://buildkite.com/redpanda/vtools/builds/13484
*https://buildkite.com/redpanda/redpanda/builds/48734
*https://buildkite.com/redpanda/redpanda/builds/48963
*https://buildkite.com/redpanda/redpanda/builds/48966
*https://buildkite.com/redpanda/redpanda/builds/48995
*https://buildkite.com/redpanda/redpanda/builds/49943

vbotbuildovich · 2024-06-12T21:09:31Z

*https://buildkite.com/redpanda/redpanda/builds/48074
*https://buildkite.com/redpanda/vtools/builds/13484
*https://buildkite.com/redpanda/redpanda/builds/48734
*https://buildkite.com/redpanda/redpanda/builds/48963
*https://buildkite.com/redpanda/redpanda/builds/48966
*https://buildkite.com/redpanda/redpanda/builds/48995
*https://buildkite.com/redpanda/redpanda/builds/49943

vbotbuildovich · 2024-06-20T03:48:50Z

*https://buildkite.com/redpanda/redpanda/builds/50379

vbotbuildovich · 2024-06-27T13:00:23Z

*https://buildkite.com/redpanda/redpanda/builds/50551
*https://buildkite.com/redpanda/redpanda/builds/50567

vbotbuildovich · 2024-07-04T23:12:17Z

*https://buildkite.com/redpanda/redpanda/builds/51084

vbotbuildovich · 2024-07-05T19:00:43Z

*https://buildkite.com/redpanda/redpanda/builds/51133

vbotbuildovich · 2024-07-19T21:11:07Z

*https://buildkite.com/redpanda/redpanda/builds/51786

piyushredpanda · 2024-09-24T04:01:17Z

Closing older-bot-filed CI issues as we transition to a more reliable system.

StephanDollberg added ci-failure kind/bug Something isn't working labels Nov 10, 2023

mmaslankaprv self-assigned this Nov 15, 2023

mmaslankaprv mentioned this issue Nov 15, 2023

c/archival_stm: translate sync error to not_leader error code #14978

Merged

7 tasks

mmaslankaprv closed this as completed in #14978 Nov 16, 2023

vbotbuildovich mentioned this issue Nov 16, 2023

[v23.2.x] CI Failure (Internal Server Error) in EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures #14996

Closed

abhijat reopened this Nov 20, 2023

travisdowns mentioned this issue Dec 20, 2023

[v23.3.x] Fix 128K iobuf zero-copy #15781

Merged

piyushredpanda mentioned this issue Mar 14, 2024

[v23.3.x] storage: ensure monotonic stable offset updates #17099

Merged

dotnwat added the team/replication helper for jira sync label Apr 18, 2024

This was referenced May 9, 2024

[v24.1.x] Fix timequery returning wrong offset after trim-prefix which could lead to stuck consumers #18281

Merged

[v24.1.x] archival: clamp uploads to committed offset #18392

Merged

bharathv mentioned this issue Jun 10, 2024

transactions: port log state into producer state struct #18684

Merged

7 tasks

bashtanov mentioned this issue Jun 17, 2024

c/backend: shard_table to be able to notify subscribers #19623

Merged

7 tasks

piyushredpanda added team/devprod display on zenhub workspace for devprod team ci-rca/infra CI Root Cause Analysis - Infrastructure Issue labels Jun 18, 2024

oleiman mentioned this issue Sep 5, 2024

serde: Require T::serde_fields for all serde structs #23126

Merged

7 tasks

piyushredpanda closed this as completed Sep 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI Failure (Internal Server Error) in `EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures` #14898

CI Failure (Internal Server Error) in `EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures` #14898

StephanDollberg commented Nov 10, 2023 •

edited by jira bot

Loading

abhijat commented Nov 20, 2023

vbotbuildovich commented Dec 13, 2023

vbotbuildovich commented Jan 27, 2024

vbotbuildovich commented Jan 30, 2024

vbotbuildovich commented Feb 1, 2024

vbotbuildovich commented Feb 2, 2024

vbotbuildovich commented Feb 7, 2024

vbotbuildovich commented Feb 18, 2024

vbotbuildovich commented Feb 28, 2024

vbotbuildovich commented Mar 4, 2024

vbotbuildovich commented Mar 14, 2024

vbotbuildovich commented Mar 18, 2024

vbotbuildovich commented Mar 20, 2024

vbotbuildovich commented Mar 29, 2024

vbotbuildovich commented Apr 3, 2024

vbotbuildovich commented Apr 3, 2024

vbotbuildovich commented Apr 3, 2024

vbotbuildovich commented Apr 4, 2024

vbotbuildovich commented Apr 10, 2024

vbotbuildovich commented Apr 22, 2024

vbotbuildovich commented May 4, 2024

vbotbuildovich commented May 6, 2024

vbotbuildovich commented May 11, 2024

vbotbuildovich commented May 13, 2024

vbotbuildovich commented Jun 6, 2024

vbotbuildovich commented Jun 11, 2024

vbotbuildovich commented Jun 11, 2024

vbotbuildovich commented Jun 12, 2024

vbotbuildovich commented Jun 20, 2024

vbotbuildovich commented Jun 27, 2024

vbotbuildovich commented Jul 4, 2024

vbotbuildovich commented Jul 5, 2024

vbotbuildovich commented Jul 19, 2024

piyushredpanda commented Sep 24, 2024

CI Failure (Internal Server Error) in EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures #14898

CI Failure (Internal Server Error) in EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures #14898

Comments

StephanDollberg commented Nov 10, 2023 • edited by jira bot Loading

abhijat commented Nov 20, 2023

vbotbuildovich commented Dec 13, 2023

vbotbuildovich commented Jan 27, 2024

vbotbuildovich commented Jan 30, 2024

vbotbuildovich commented Feb 1, 2024

vbotbuildovich commented Feb 2, 2024

vbotbuildovich commented Feb 7, 2024

vbotbuildovich commented Feb 18, 2024

vbotbuildovich commented Feb 28, 2024

vbotbuildovich commented Mar 4, 2024

vbotbuildovich commented Mar 14, 2024

vbotbuildovich commented Mar 18, 2024

vbotbuildovich commented Mar 20, 2024

vbotbuildovich commented Mar 29, 2024

vbotbuildovich commented Apr 3, 2024

vbotbuildovich commented Apr 3, 2024

vbotbuildovich commented Apr 3, 2024

vbotbuildovich commented Apr 4, 2024

vbotbuildovich commented Apr 10, 2024

vbotbuildovich commented Apr 22, 2024

vbotbuildovich commented May 4, 2024

vbotbuildovich commented May 6, 2024

vbotbuildovich commented May 11, 2024

vbotbuildovich commented May 13, 2024

vbotbuildovich commented Jun 6, 2024

vbotbuildovich commented Jun 11, 2024

vbotbuildovich commented Jun 11, 2024

vbotbuildovich commented Jun 12, 2024

vbotbuildovich commented Jun 20, 2024

vbotbuildovich commented Jun 27, 2024

vbotbuildovich commented Jul 4, 2024

vbotbuildovich commented Jul 5, 2024

vbotbuildovich commented Jul 19, 2024

piyushredpanda commented Sep 24, 2024

CI Failure (Internal Server Error) in `EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures` #14898

CI Failure (Internal Server Error) in `EndToEndShadowIndexingTestWithDisruptions.test_write_with_node_failures` #14898

StephanDollberg commented Nov 10, 2023 •

edited by jira bot

Loading