CI Failure (timeout) in `test_archival_service_rpfixture` #14254

VladLazar · 2023-10-18T17:00:28Z

https://buildkite.com/redpanda/redpanda/builds/39032#018b384f-4683-4253-9ec8-dc24f384f602

This happened on #13896, so it could be related to that, but @nvartolomei mentioned he's seen the issue previously.

andijcr · 2023-10-20T14:49:04Z

maybe related to the partition::stop() block i'm seeing in #14065 ?

VladLazar · 2023-10-23T12:18:22Z

maybe related to the partition::stop() block i'm seeing in #14065 ?

Perhaps, but for #14065, it un-wedges itself eventually.

andijcr · 2023-11-02T14:51:33Z

as a datapoint, just observing the output in the terminal i see the test hanging here and then continue

likely from test_archival_policy_search_when_a_segment_is_compacted

INFO  2023-11-02 15:49:38,772 [shard 0:main] client_pool - client_pool.cc:164 - Stopping client pool: 2
INFO  2023-11-02 15:49:38,772 [shard 0:main] client_pool - client_pool.cc:184 - Stopped client pool
INFO  2023-11-02 15:49:38,772 [shard 0:main] kafka - server.cc:280 - kafka rpc protocol - Stopping 1 listeners
INFO  2023-11-02 15:49:38,772 [shard 0:main] kafka - server.cc:292 - kafka rpc protocol - Shutting down 0 connections
INFO  2023-11-02 15:49:38,772 [shard 0:main] rpc - server.cc:280 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2023-11-02 15:49:38,772 [shard 0:main] rpc - server.cc:292 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2023-11-02 15:49:38,772 [shard 0:main] client_pool - client_pool.cc:189 - Shutting down client pool: 20
INFO  2023-11-02 15:49:38,772 [shard 0:main] client_pool - client_pool.cc:203 - Shut down of client pool complete
INFO  2023-11-02 15:49:38,772 [shard 0:main] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:266 - Stopping
INFO  2023-11-02 15:49:38,772 [shard 0:main] raft - [group_id:1, {kafka/test-topic/42}] consensus.cc:266 - Stopping
INFO  2023-11-02 15:49:41,601 [shard 0:main] cluster - partition_balancer_planner.cc:1528 - counts rebalancing objective in domain 0:      1 ->      1
INFO  2023-11-02 15:49:41,602 [shard 0:main] cluster - partition_balancer_planner.cc:1528 - counts rebalancing objective in domain -1:      1 ->      1
INFO  2023-11-02 15:49:41,602 [shard 0:main] cluster - partition_balancer_backend.cc:420 - last status: in_progress; violations: unavailable nodes: 0, full nodes: 0; nodes to rebalance count: 1; on demand rebalance requested: false; updates in progress: 0; action counts: reassignments: 0, cancellations: 0, failed: 0; counts rebalancing finished: true, force refresh health report: false
WARN  2023-11-02 15:49:41,602 [shard 0:main] cluster - partition_balancer_backend.cc:443 - submitting finish_reallocations for node 1 failed, error: raft protocol shutting down

abhijat · 2023-11-27T12:37:18Z

Ran this on repeat in CI and in the debug build I did see this get blocked after one test failed:

t=1701077829694116: /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-03788b59bfc51b527-1/redpanda/vtools/src/v/archival/tests/ntp_archiver_test.cc(1780): fatal error: in "test_manifest_spillover": critical check uploaded.size() != 0 has failed
t=1701077829694116: INFO  2023-11-27 09:37:09,694 [shard 0:main] test - ntp_archiver_test.cc:1766 - Reconciling storage bucket
t=1701077829694116: test_archival_service_rpfixture: /vectorized/include/seastar/core/gate.hh:91: seastar::gate::~gate(): Assertion `!_count && "gate destroyed with outstanding requests"' failed.
t=1701077829694116: ==51==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7f226d2bc000; bottom 0x7ffd974b2000; size: 0xffffff24d5e0a000 (-941304537088)
t=1701077829694116: False positive error reports may follow
t=1701077829694116: For details see https://github.com/google/sanitizers/issues/189
t=1701077829694116: unknown location(0): fatal error: in "test_manifest_spillover": signal: SIGABRT (application abort requested)
t=1701077829694116: /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-03788b59bfc51b527-1/redpanda/vtools/src/v/archival/tests/ntp_archiver_test.cc(1780): last checkpoint
t=1701077829694116: /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-03788b59bfc51b527-1/redpanda/vtools/src/v/archival/tests/ntp_archiver_test.cc(1797): Leaving test case "test_manifest_spillover"; testing time: 3501920us
t=1701077829694116: /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-03788b59bfc51b527-1/redpanda/vtools/src/v/archival/tests/ntp_archiver_test.cc(1802): Entering test case "test_upload_with_gap_blocked"

This seems to have got stuck at around 3:07 PM IST, after having been started at 2:29 PM IST, so around 40 minutes. The build was cancelled manually at 6 PM IST so it did not appear to progress for the intervening 3.5 hours at least in buildkite logs.

test_manifest_spillover failed and then we get a subsequent assertion from the gate, which should be fixed with a defer to clean up if the test fails.

test_upload_with_gap_blocked is entered but no further logs are seen.

abhijat · 2023-11-27T14:10:42Z

Another run of buildkite with just the debug build and all logs turned on runs into some sort of internal error but downloading the raw logs again shows the test run stuck at

t=1701092691580116: DEBUG 2023-11-27 13:44:51,580 [shard 0:main] test - ntp_archiver_test.cc:1739 - apply spillover, spillover manifest size: 4096
t=1701092691580116: INFO  2023-11-27 13:44:51,580 [shard 0:main] test - ntp_archiver_test.cc:1763 - new_so: {nullopt}, new_kafka: {nullopt}, archive_so: -9223372036854775808, archive_kafka: -9223372036854775808, archive_clean: -9223372036854775808
t=1701092691580116: /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-03788b59bfc51b527-1/redpanda/vtools/src/v/archival/tests/ntp_archiver_test.cc(1780): fatal error: in "test_manifest_spillover": critical check uploaded.size() != 0 has failed
t=1701092691580116: INFO  2023-11-27 13:44:51,580 [shard 0:main] test - ntp_archiver_test.cc:1766 - Reconciling storage bucket
t=1701092691580116: test_archival_service_rpfixture: /vectorized/include/seastar/core/gate.hh:91: seastar::gate::~gate(): Assertion `!_count && "gate destroyed with outstanding requests"' failed.
t=1701092691580116: ==873==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7f8a8deff000; bottom 0x7ffc5a511000; size: 0xffffff8e339ee000 (-488760221696)
t=1701092691580116: False positive error reports may follow
t=1701092691580116: For details see https://github.com/google/sanitizers/issues/189
t=1701092691580116: unknown location(0): fatal error: in "test_manifest_spillover": signal: SIGABRT (application abort requested)
t=1701092691580116: /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-03788b59bfc51b527-1/redpanda/vtools/src/v/archival/tests/ntp_archiver_test.cc(1780): last checkpoint
t=1701092691581116: /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-03788b59bfc51b527-1/redpanda/vtools/src/v/archival/tests/ntp_archiver_test.cc(1797): Leaving test case "test_manifest_spillover"; testing time: 3498975us
t=1701092691581116: /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-03788b59bfc51b527-1/redpanda/vtools/src/v/archival/tests/ntp_archiver_test.cc(1802): Entering test case "test_upload_with_gap_blocked"

While the buildkite webpage is stuck at:

⚠️ Oops, something went wrong when we tried to fetch your log
We've been notified of this error, and we'll take a look at it very soon. In the mean time, perhaps try reloading the page?

abhijat · 2023-11-27T16:05:28Z

t=1701098806322116: DEBUG 2023-11-27 15:26:46,322 [shard 0:main] test - ntp_archiver_test.cc:1739 - apply spillover, spillover manifest size: 4096
t=1701098806322116: INFO  2023-11-27 15:26:46,322 [shard 0:main] test - ntp_archiver_test.cc:1763 - new_so: {nullopt}, new_kafka: {nullopt}, archive_so: -9223372036854775808, archive_kafka: -9223372036854775808, archive_clean: -9223372036854775808
t=1701098806322116: /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-03788b59bfc51b527-1/redpanda/vtools/src/v/archival/tests/ntp_archiver_test.cc(1780): fatal error: in "test_manifest_spillover": critical check uploaded.size() != 0 has failed
t=1701098806322116: INFO  2023-11-27 15:26:46,322 [shard 0:main] test - ntp_archiver_test.cc:1766 - Reconciling storage bucket
t=1701098806322116: test_archival_service_rpfixture: /vectorized/include/seastar/core/gate.hh:91: seastar::gate::~gate(): Assertion `!_count && "gate destroyed with outstanding requests"' failed.
t=1701098806322116: ==212==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7f46772eb000; bottom 0x7ffd44a01000; size: 0xffffff49328ea000 (-785130807296)
t=1701098806322116: False positive error reports may follow
t=1701098806322116: For details see https://github.com/google/sanitizers/issues/189
t=1701098806322116: unknown location(0): fatal error: in "test_manifest_spillover": signal: SIGABRT (application abort requested)
t=1701098806322116: /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-03788b59bfc51b527-1/redpanda/vtools/src/v/archival/tests/ntp_archiver_test.cc(1780): last checkpoint
t=1701098806322116: /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-03788b59bfc51b527-1/redpanda/vtools/src/v/archival/tests/ntp_archiver_test.cc(1797): Leaving test case "test_manifest_spillover"; testing time: 3499867us
t=1701098806322116: /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-03788b59bfc51b527-1/redpanda/vtools/src/v/archival/tests/ntp_archiver_test.cc(1802): Entering test case "test_upload_with_gap_blocked"

Another run failed at the same spot - blocked for 35 minutes. This seems like at least one spot where the test hangs. Will create a PR to fix this.

abhijat · 2023-11-28T05:05:29Z

Artificially failing the test at the same assertion point does not trigger the test hang locally. There are the following areas to investigate here:

fixing the test failure itself
cleaning up if the test does fail via shutting down the archiver etc
does this test hang or the next one (test_upload_with_gap_blocked)?

abhijat · 2023-11-28T05:56:34Z

The test hanging in CI (when it fails) has a different set of errors than when the test fails locally (artifically failed using a bogus comparison etc)

in CI (the gate related assert and the request to abort are visible):

t=1701077829694116: /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-03788b59bfc51b527-1/redpanda/vtools/src/v/archival/tests/ntp_archiver_test.cc(1780): fatal error: in "test_manifest_spillover": critical check uploaded.size() != 0 has failed
t=1701077829694116: INFO  2023-11-27 09:37:09,694 [shard 0:main] test - ntp_archiver_test.cc:1766 - Reconciling storage bucket
t=1701077829694116: test_archival_service_rpfixture: /vectorized/include/seastar/core/gate.hh:91: seastar::gate::~gate(): Assertion `!_count && "gate destroyed with outstanding requests"' failed.
t=1701077829694116: ==51==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7f226d2bc000; bottom 0x7ffd974b2000; size: 0xffffff24d5e0a000 (-941304537088)
t=1701077829694116: False positive error reports may follow
t=1701077829694116: For details see https://github.com/google/sanitizers/issues/189
t=1701077829694116: unknown location(0): fatal error: in "test_manifest_spillover": signal: SIGABRT (application abort requested)
t=1701077829694116: /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-03788b59bfc51b527-1/redpanda/vtools/src/v/archival/tests/ntp_archiver_test.cc(1780): last checkpoint

locally:

INFO  2023-11-28 10:56:25,963 [shard 0:main] test - ntp_archiver_test.cc:1769 - Found /10000000/meta/kafka/test-topic/42_0/manifest.bin.514.770.514.771.514.770
INFO  2023-11-28 10:56:25,964 [shard 0:main] test - ntp_archiver_test.cc:1769 - Found /10000000/meta/kafka/test-topic/42_0/manifest.bin.771.1027.771.1028.771.1027
/development/redpanda/src/v/archival/tests/ntp_archiver_test.cc(1780): fatal error: in "test_manifest_spillover": critical check uploaded.size() == 0 has failed
INFO  2023-11-28 10:56:25,966 [shard 0:main] client_pool - client_pool.cc:189 - Shutting down client pool: 2
INFO  2023-11-28 10:56:25,966 [shard 0:main] client_pool - client_pool.cc:203 - Shut down of client pool complete
INFO  2023-11-28 10:56:25,969 [shard 0:main] client_pool - client_pool.cc:164 - Stopping client pool: 2

In this test we add a set of segments to the archival STM before triggering spillover. For each added segment we should be able to see some logs for the added segment. In CI when the test fails, no such logs are seen:

t=1701077829693116: DEBUG 2023-11-27 09:37:09,692 [shard 0:main] test - ntp_archiver_test.cc:1700 - stm add segments
t=1701077829693116: DEBUG 2023-11-27 09:37:09,693 [shard 0:main] test - ntp_archiver_test.cc:1710 - stm add segments completed
t=1701077829693116: DEBUG 2023-11-27 09:37:09,693 [shard 0:main] test - ntp_archiver_test.cc:1715 - archival metadata stm last offset: 0
...
t=1701077829694116: INFO  2023-11-27 09:37:09,694 [shard 0:main] test - ntp_archiver_test.cc:1763 - new_so: {nullopt}, new_kafka: {nullopt}, archive_so: -9223372036854775808, archive_kafka: -9223372036854775808, archive_clean: -9223372036854775808

The updated offsets are also indicative of no segments being added. When the test is run locally these logs are visible:

INFO  2023-11-28 10:56:23,521 [shard 0:main] cluster - ntp: {kafka/test-topic/42} - archival_metadata_stm.cc:941 - new remote segment added (name: {0-1-v1.log}, meta: {is_compacted: false, size_bytes: 1, base_offset: 0, committed_offset: 0, base_timestamp: {timestamp: 0}, max_timestamp: {timestamp: 0}, delta_offset: 0, ntp_revision: 0, archiver_term: 1, segment_term: 1, delta_offset_end: 0, sname_format: {v1}, metadata_size_hint: 0}remote start_offset: 0, last_offset: 1536, highest_producer_id: -9223372036854775808)
INFO  2023-11-28 10:56:23,522 [shard 0:main] cluster - ntp: {kafka/test-topic/42} - archival_metadata_stm.cc:941 - new remote segment added (name: {1-1-v1.log}, meta: {is_compacted: false, size_bytes: 1, base_offset: 1, committed_offset: 1, base_timestamp: {timestamp: 1}, max_timestamp: {timestamp: 1}, delta_offset: 0, ntp_revision: 0, archiver_term: 1, segment_term: 1, delta_offset_end: 0, sname_format: {v1}, metadata_size_hint: 0}remote start_offset: 0, last_offset: 1536, highest_producer_id: -9223372036854775808)
INFO  2023-11-28 10:56:23,522 [shard 0:main] cluster - ntp: {kafka/test-topic/42} - archival_metadata_stm.cc:941 - new remote segment added (name: {2-1-v1.log}, meta: {is_compacted: false, size_bytes: 1, base_offset: 2, committed_offset: 2, base_timestamp: {timestamp: 2}, max_timestamp: {timestamp: 2}, delta_offset: 0, ntp_revision: 0, archiver_term: 1, segment_term: 1, delta_offset_end: 0, sname_format: {v1}, metadata_size_hint: 0}remote start_offset: 0, last_offset: 1536, highest_producer_id: -9223372036854775808)
INFO  2023-11-28 10:56:23,522 [shard 0:main] cluster - ntp: {kafka/test-topic/42} - archival_metadata_stm.cc:941 - new remote segment added (name: {3-1-v1.log}, meta: {is_compacted: false, size_bytes: 1, base_offset: 3, committed_offset: 3, base_timestamp: {timestamp: 3}, max_timestamp: {timestamp: 3}, delta_offset: 0, ntp_revision: 0, archiver_term: 1, segment_term: 1, delta_offset_end: 0, sname_format: {v1}, metadata_size_hint: 0}remote start_offset: 0, last_offset: 1536, highest_producer_id: -9223372036854775808)
...
INFO  2023-11-28 10:56:25,959 [shard 0:main] test - ntp_archiver_test.cc:1763 - new_so: {1028}, new_kafka: {1028}, archive_so: 0, archive_kafka: 0, archive_clean: 0

To recreate the same scenario (no segments added to STM) if the part to add segments to STM is removed, the test hangs locally just like in CI:

INFO  2023-11-28 11:25:47,664 [shard 0:main] raft - [group_id:1, {kafka/test-topic/42}] vote_stm.cc:281 - became the leader term: 2
DEBUG 2023-11-28 11:25:47,679 [shard 0:main] test - ntp_archiver_test.cc:1700 - stm add segments
DEBUG 2023-11-28 11:25:47,679 [shard 0:main] test - ntp_archiver_test.cc:1710 - stm add segments completed
DEBUG 2023-11-28 11:25:47,679 [shard 0:main] test - ntp_archiver_test.cc:1715 - archival metadata stm last offset: 0
TRACE 2023-11-28 11:25:47,679 [shard 0:main] features - feature_table.cc:493 - Feature cloud_storage_scrubbing already active
DEBUG 2023-11-28 11:25:47,680 [shard 0:main] archival - ntp_archiver_service.cc:257 - created ntp_archiver {kafka/test-topic/42} in term 2
DEBUG 2023-11-28 11:25:47,680 [shard 0:main] test - ntp_archiver_test.cc:1739 - apply spillover, spillover manifest size: 4096
/development/redpanda/src/v/archival/tests/ntp_archiver_test.cc(1780): fatal error: in "test_manifest_spillover": critical check uploaded.size() != 0 has failed
DEBUG 2023-11-28 11:25:47,680 [shard 0:main] archival - [fiber96 kafka/test-topic/42] - ntp_archiver_service.cc:2450 - Manifest size: 312, manifest size limit (x2): 8192
INFO  2023-11-28 11:25:47,680 [shard 0:main] test - ntp_archiver_test.cc:1763 - new_so: {nullopt}, new_kafka: {nullopt}, archive_so: -9223372036854775808, archive_kafka: -9223372036854775808, archive_clean: -9223372036854775808
unknown location(0): fatal error: in "test_manifest_spillover": signal: SIGABRT (application abort requested)
/development/redpanda/src/v/archival/tests/ntp_archiver_test.cc(1780): last checkpoint
/development/redpanda/src/v/archival/tests/ntp_archiver_test.cc(1797): Leaving test case "test_manifest_spillover"; testing time: 2833986us
/development/redpanda/src/v/archival/tests/ntp_archiver_test.cc(1802): Entering test case "test_upload_with_gap_blocked"
INFO  2023-11-28 11:25:47,680 [shard 0:main] test - ntp_archiver_test.cc:1766 - Reconciling storage bucket
test_archival_service_rpfixture: /development/redpanda/vbuild/debug/clang/rp_deps_install/include/seastar/core/gate.hh:91: seastar::gate::~gate(): Assertion `!_count && "gate destroyed with outstanding requests"' failed.
==409617==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7f16eb9bc000; bottom 0x7fff34ac0000; size: 0xffffff17b6efc000 (-997658214400)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189

The test run hangs at this point, so the issue seems related closely to the STM state.

abhijat · 2023-11-28T06:06:03Z

The assertion which hangs the test run is from the scrubber in the archiver being destroyed, the scrubber seems to have some requests in flight when the gate is destroyed. Adding a defer to stop the archiver seems to fix the hang, but why the original test test_manifest_spillover fails to sometimes add segments remains to be seen. That can be done via a separate ticket perhaps.

dotnwat · 2023-12-06T23:43:01Z

I see this in #15344 it looks like it is the same thing.

abhijat · 2024-02-12T16:04:05Z

I see this in #15344 it looks like it is the same thing.

It looks like a different issue:

t=1701892909662/var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0d3b71f9868f3b66e-1/redpanda/redpanda/src/v/archival/tests/ntp_archiver_test.cc(1802): fatal error: in "test_manifest_spillover": critical check uploaded.size() != 0 has failed

piyushredpanda · 2024-03-23T21:50:40Z

This issue hasn't reoccurred for more than 2 months; closing.

VladLazar added kind/bug Something isn't working ci-failure area/cloud-storage Shadow indexing subsystem rpunit unit test ci-failure (not ducktape) labels Oct 18, 2023

VladLazar mentioned this issue Oct 23, 2023

archival: same term sync-ing #13896

Merged

7 tasks

mmaslankaprv mentioned this issue Oct 27, 2023

Fixed Raft reconfiguration unit test #14459

Merged

7 tasks

ztlpn mentioned this issue Oct 27, 2023

APIs for disabling topics/partitions and listing them #14461

Merged

7 tasks

andijcr self-assigned this Nov 2, 2023

StephanDollberg mentioned this issue Nov 7, 2023

kafka: Add fetch plan and execute latency metric #13485

Merged

7 tasks

andrwng mentioned this issue Nov 8, 2023

storage: enable sliding window compaction by default #14330

Merged

7 tasks

rockwotj mentioned this issue Nov 9, 2023

transform/probe: cleanup metrics #14831

Merged

7 tasks

piyushredpanda unassigned andijcr Nov 17, 2023

abhijat self-assigned this Nov 17, 2023

rockwotj mentioned this issue Nov 20, 2023

transform: fix ducktape tests #15009

Merged

7 tasks

abhijat mentioned this issue Nov 28, 2023

archival/test: Ensure archiver is stopped #15169

Merged

7 tasks

abhijat closed this as completed in #15169 Nov 28, 2023

vbotbuildovich mentioned this issue Nov 28, 2023

[v23.2.x] CI Failure (timeout) in test_archival_service_rpfixture #15176

Closed

dotnwat mentioned this issue Dec 6, 2023

clang-tidy: tweak find_program hints #15344

Merged

7 tasks

dotnwat reopened this Dec 6, 2023

piyushredpanda closed this as completed Mar 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI Failure (timeout) in `test_archival_service_rpfixture` #14254

CI Failure (timeout) in `test_archival_service_rpfixture` #14254

VladLazar commented Oct 18, 2023

andijcr commented Oct 20, 2023

VladLazar commented Oct 23, 2023

andijcr commented Nov 2, 2023 •

edited

Loading

abhijat commented Nov 27, 2023

abhijat commented Nov 27, 2023

abhijat commented Nov 27, 2023

abhijat commented Nov 28, 2023

abhijat commented Nov 28, 2023

abhijat commented Nov 28, 2023

dotnwat commented Dec 6, 2023

abhijat commented Feb 12, 2024

piyushredpanda commented Mar 23, 2024

CI Failure (timeout) in test_archival_service_rpfixture #14254

CI Failure (timeout) in test_archival_service_rpfixture #14254

Comments

VladLazar commented Oct 18, 2023

andijcr commented Oct 20, 2023

VladLazar commented Oct 23, 2023

andijcr commented Nov 2, 2023 • edited Loading

abhijat commented Nov 27, 2023

abhijat commented Nov 27, 2023

abhijat commented Nov 27, 2023

abhijat commented Nov 28, 2023

abhijat commented Nov 28, 2023

abhijat commented Nov 28, 2023

dotnwat commented Dec 6, 2023

abhijat commented Feb 12, 2024

piyushredpanda commented Mar 23, 2024

CI Failure (timeout) in `test_archival_service_rpfixture` #14254

CI Failure (timeout) in `test_archival_service_rpfixture` #14254

andijcr commented Nov 2, 2023 •

edited

Loading