CI Failure (bad_deltas anomaly) in `RedpandaUpgradeTest.test_workloads_through_releases` #11842

VladLazar · 2023-07-03T14:32:26Z

https://buildkite.com/redpanda/redpanda/builds/32382#01891a7b-8348-4279-b724-633f25042597

Module: rptest.tests.workload_upgrade_runner_test
Class:  RedpandaUpgradeTest
Method: test_workloads_through_releases

test_id:    rptest.tests.workload_upgrade_runner_test.RedpandaUpgradeTest.test_workloads_through_releases
status:     FAIL
run time:   15 minutes 7.355 seconds


    RuntimeError("Object storage scrub detected fatal anomalies of type {'ntpr_bad_deltas'}")
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/utils/mode_checks.py", line 63, in f
    return func(*args, **kwargs)
  File "/root/tests/rptest/services/cluster.py", line 159, in wrapped
    self.redpanda.stop_and_scrub_object_storage()
  File "/root/tests/rptest/services/redpanda.py", line 3708, in stop_and_scrub_object_storage
    raise RuntimeError(
RuntimeError: Object storage scrub detected fatal anomalies of type {'ntpr_bad_deltas'}

The text was updated successfully, but these errors were encountered:

VladLazar · 2023-07-03T14:39:13Z

https://buildkite.com/redpanda/vtools/builds/8306#01890e37-4e3c-4410-8295-fb5937d43c23

jcsp · 2023-07-03T14:46:47Z

This could either be a bug in a historic redpanda or a bug in the tool's handling of legacy manifests which miss some deltas. Until we know otherwise, let's assume it's a real issue.

andijcr · 2023-07-03T15:12:14Z

from test_log.debug:

    "ntpr_bad_deltas": [
      "kafka/topic-MinimalWorkload-31ff1f08-6d03-44c8-878e-0a7581c16ce0/0_15"
    ],
    
    ...
      "partitions": {
    "kafka/topic-MinimalWorkload-31ff1f08-6d03-44c8-878e-0a7581c16ce0/0_15": {
      "bytes": 19621,
      "raw_start_offset": 0,
      "raw_last_offset": 60,
      "kafka_lwm": 0,
      "kafka_hwm": 12,
      "lifecycle_status": null
    },

this i produced by this class

class MinimalWorkload(PWorkload):
    def __init__(self, ctx: RedpandaTest) -> None:
        self.ctx = ctx
        self.topic = TopicSpec(
            name=f"topic-{self.__class__.__name__}-{str(uuid.uuid4())}",
            replication_factor=3)

    def begin(self):
        self.ctx.client().create_topic(self.topic)

    def end(self):
        self.ctx.client().delete_topic(self.topic.name)

    def on_cluster_upgraded(self, version: tuple[int, int, int]) -> int:
        offset = RpkTool(self.ctx.redpanda).produce(topic=self.topic.name,
                                                    key=f"{version}",
                                                    msg=str(uuid.uuid4()))
        self.ctx.logger.info(f"produced to {self.topic.name} at {offset=}")
        return PWorkload.DONE

Which generates a simple message on each cluster version:

[INFO  - 2023-07-03 07:26:05,845 - workload_upgrade_runner_test - setUp - lineno:211]: going through these versions: [(22, 1, 11), (22, 1, 10), (22, 1, 11), (22, 2, 13), (22, 2, 12), (22, 2, 13), (22, 3, 22), (22, 3, 21), (22, 3, 22), (23, 1, 13), (23, 1, 12), (23, 1, 13), (23, 2, 1)]

so 13 messages in total.
loglines on for this topic:
MinimalWorkload.log.txt

andijcr · 2023-07-05T12:56:03Z

#11848

running the check after every upgrade step, shows that ntpr_bad_deltas is generated after installing and checking HEAD.

this is 40000000_meta_kafka_topic-MinimalWorkload-fcb57d39-5b31-4c2a-84d6-a063912da293_0_15_manifest.json
, but it seems fine. I'm modifying the test to catch stderr from the tool

{
  "version": 1,
  "namespace": "kafka",
  "topic": "topic-MinimalWorkload-fcb57d39-5b31-4c2a-84d6-a063912da293",
  "partition": 0,
  "revision": 15,
  "last_offset": 65,
  "insync_offset": 67,
  "start_offset": 0,
  "segments": {
    "0-1-v1.log": {
      "is_compacted": false,
      "size_bytes": 723,
      "committed_offset": 1,
      "base_offset": 0,
      "base_timestamp": 1688558529552,
      "max_timestamp": 1688558529552,
      "delta_offset": 0,
      "archiver_term": 2,
      "segment_term": 1
    },
    "2-2-v1.log": {
      "is_compacted": false,
      "size_bytes": 893,
      "committed_offset": 4,
      "base_offset": 2,
      "base_timestamp": 1688558538599,
      "max_timestamp": 1688558540400,
      "delta_offset": 1,
      "archiver_term": 3,
      "segment_term": 2
    },
    "5-3-v1.log": {
      "is_compacted": false,
      "size_bytes": 893,
      "committed_offset": 7,
      "base_offset": 5,
      "base_timestamp": 1688558551240,
      "max_timestamp": 1688558551240,
      "delta_offset": 3,
      "archiver_term": 4,
      "segment_term": 3
    },
    "8-4-v1.log": {
      "is_compacted": false,
      "size_bytes": 778,
      "committed_offset": 9,
      "base_offset": 8,
      "base_timestamp": 1688558568085,
      "max_timestamp": 1688558568085,
      "delta_offset": 5,
      "archiver_term": 5,
      "segment_term": 4
    },
    "10-4-v1.log": {
      "is_compacted": false,
      "size_bytes": 115,
      "committed_offset": 10,
      "base_offset": 10,
      "base_timestamp": 1688558603445,
      "max_timestamp": 1688558603445,
      "delta_offset": 7,
      "archiver_term": 5,
      "segment_term": 4
    },
    "11-5-v1.log": {
      "is_compacted": false,
      "size_bytes": 889,
      "committed_offset": 13,
      "base_offset": 11,
      "base_timestamp": 1688558604763,
      "max_timestamp": 1688558604763,
      "delta_offset": 7,
      "archiver_term": 6,
      "segment_term": 5
    },
    "14-6-v1.log": {
      "is_compacted": false,
      "size_bytes": 780,
      "committed_offset": 15,
      "base_offset": 14,
      "base_timestamp": 1688558616529,
      "max_timestamp": 1688558616529,
      "delta_offset": 10,
      "archiver_term": 7,
      "segment_term": 6
    },
    "16-7-v1.log": {
      "is_compacted": false,
      "size_bytes": 895,
      "committed_offset": 18,
      "base_offset": 16,
      "base_timestamp": 1688558634572,
      "max_timestamp": 1688558634572,
      "delta_offset": 12,
      "archiver_term": 8,
      "segment_term": 7
    },
    "19-8-v1.log": {
      "is_compacted": false,
      "size_bytes": 780,
      "committed_offset": 20,
      "base_offset": 19,
      "base_timestamp": 1688558636115,
      "max_timestamp": 1688558636115,
      "delta_offset": 14,
      "archiver_term": 9,
      "segment_term": 8
    },
    "21-9-v1.log": {
      "is_compacted": false,
      "size_bytes": 780,
      "committed_offset": 22,
      "base_offset": 21,
      "base_timestamp": 1688558643409,
      "max_timestamp": 1688558643409,
      "delta_offset": 16,
      "archiver_term": 10,
      "segment_term": 9
    },
    "23-9-v1.log": {
      "is_compacted": false,
      "size_bytes": 115,
      "committed_offset": 23,
      "base_offset": 23,
      "base_timestamp": 1688558658893,
      "max_timestamp": 1688558658893,
      "delta_offset": 18,
      "archiver_term": 10,
      "segment_term": 9
    },
    "24-10-v1.log": {
      "is_compacted": false,
      "size_bytes": 890,
      "committed_offset": 26,
      "base_offset": 24,
      "base_timestamp": 1688558660205,
      "max_timestamp": 1688558660205,
      "delta_offset": 18,
      "archiver_term": 11,
      "segment_term": 10
    },
    "27-11-v1.log": {
      "is_compacted": false,
      "size_bytes": 781,
      "committed_offset": 28,
      "base_offset": 27,
      "base_timestamp": 1688558679033,
      "max_timestamp": 1688558679033,
      "delta_offset": 21,
      "archiver_term": 12,
      "segment_term": 11,
      "delta_offset_end": 23,
      "sname_format": 2
    },
    "29-12-v1.log": {
      "is_compacted": false,
      "size_bytes": 916,
      "committed_offset": 31,
      "base_offset": 29,
      "base_timestamp": 1688558720824,
      "max_timestamp": 1688558720824,
      "delta_offset": 23,
      "archiver_term": 13,
      "segment_term": 12,
      "delta_offset_end": 25,
      "sname_format": 2
    },
    "32-13-v1.log": {
      "is_compacted": false,
      "size_bytes": 801,
      "committed_offset": 33,
      "base_offset": 32,
      "base_timestamp": 1688558730400,
      "max_timestamp": 1688558730400,
      "delta_offset": 25,
      "archiver_term": 14,
      "segment_term": 13,
      "delta_offset_end": 27,
      "sname_format": 2
    },
    "34-14-v1.log": {
      "is_compacted": false,
      "size_bytes": 801,
      "committed_offset": 35,
      "base_offset": 34,
      "base_timestamp": 1688558746342,
      "max_timestamp": 1688558746342,
      "delta_offset": 27,
      "archiver_term": 15,
      "segment_term": 14,
      "delta_offset_end": 29,
      "sname_format": 2
    },
    "36-15-v1.log": {
      "is_compacted": false,
      "size_bytes": 801,
      "committed_offset": 37,
      "base_offset": 36,
      "base_timestamp": 1688558758614,
      "max_timestamp": 1688558758614,
      "delta_offset": 29,
      "archiver_term": 16,
      "segment_term": 15,
      "delta_offset_end": 31,
      "sname_format": 2
    },
    "38-15-v1.log": {
      "is_compacted": false,
      "size_bytes": 115,
      "committed_offset": 38,
      "base_offset": 38,
      "base_timestamp": 1688558771863,
      "max_timestamp": 1688558771863,
      "delta_offset": 31,
      "archiver_term": 16,
      "segment_term": 15,
      "delta_offset_end": 31,
      "sname_format": 2
    },
    "39-16-v1.log": {
      "is_compacted": false,
      "size_bytes": 932,
      "committed_offset": 41,
      "base_offset": 39,
      "base_timestamp": 1688558793052,
      "max_timestamp": 1688558793052,
      "delta_offset": 31,
      "archiver_term": 17,
      "segment_term": 16,
      "delta_offset_end": 34,
      "sname_format": 2
    },
    "42-17-v1.log": {
      "is_compacted": false,
      "size_bytes": 916,
      "committed_offset": 44,
      "base_offset": 42,
      "base_timestamp": 1688558819043,
      "max_timestamp": 1688558819043,
      "delta_offset": 34,
      "archiver_term": 18,
      "segment_term": 17,
      "delta_offset_end": 36,
      "sname_format": 2
    },
    "45-18-v1.log": {
      "is_compacted": false,
      "size_bytes": 801,
      "committed_offset": 46,
      "base_offset": 45,
      "base_timestamp": 1688558832873,
      "max_timestamp": 1688558832873,
      "delta_offset": 36,
      "archiver_term": 19,
      "segment_term": 18,
      "delta_offset_end": 38,
      "sname_format": 2
    },
    "47-19-v1.log": {
      "is_compacted": false,
      "size_bytes": 801,
      "committed_offset": 48,
      "base_offset": 47,
      "base_timestamp": 1688558848396,
      "max_timestamp": 1688558848396,
      "delta_offset": 38,
      "archiver_term": 20,
      "segment_term": 19,
      "delta_offset_end": 40,
      "sname_format": 2
    },
    "49-20-v1.log": {
      "is_compacted": false,
      "size_bytes": 801,
      "committed_offset": 50,
      "base_offset": 49,
      "base_timestamp": 1688558856094,
      "max_timestamp": 1688558856094,
      "delta_offset": 40,
      "archiver_term": 21,
      "segment_term": 20,
      "delta_offset_end": 42,
      "sname_format": 2
    },
    "51-20-v1.log": {
      "is_compacted": false,
      "size_bytes": 115,
      "committed_offset": 51,
      "base_offset": 51,
      "base_timestamp": 1688558874201,
      "max_timestamp": 1688558874201,
      "delta_offset": 42,
      "archiver_term": 21,
      "segment_term": 20,
      "delta_offset_end": 42,
      "sname_format": 2
    },
    "52-21-v1.log": {
      "is_compacted": false,
      "size_bytes": 932,
      "committed_offset": 54,
      "base_offset": 52,
      "base_timestamp": 1688558883747,
      "max_timestamp": 1688558883747,
      "delta_offset": 42,
      "archiver_term": 22,
      "segment_term": 21,
      "delta_offset_end": 45,
      "sname_format": 2
    },
    "55-22-v1.log": {
      "is_compacted": false,
      "size_bytes": 801,
      "committed_offset": 56,
      "base_offset": 55,
      "base_timestamp": 1688558891949,
      "max_timestamp": 1688558891949,
      "delta_offset": 45,
      "archiver_term": 23,
      "segment_term": 22,
      "delta_offset_end": 47,
      "sname_format": 2
    },
    "57-22-v1.log": {
      "is_compacted": false,
      "size_bytes": 115,
      "committed_offset": 57,
      "base_offset": 57,
      "base_timestamp": 1688558895942,
      "max_timestamp": 1688558895942,
      "delta_offset": 47,
      "archiver_term": 23,
      "segment_term": 22,
      "delta_offset_end": 47,
      "sname_format": 2
    },
    "58-23-v1.log": {
      "is_compacted": false,
      "size_bytes": 932,
      "committed_offset": 60,
      "base_offset": 58,
      "base_timestamp": 1688558910333,
      "max_timestamp": 1688558910333,
      "delta_offset": 47,
      "archiver_term": 24,
      "segment_term": 23,
      "delta_offset_end": 50,
      "sname_format": 2
    },
    "61-24-v1.log": {
      "is_compacted": false,
      "size_bytes": 916,
      "committed_offset": 63,
      "base_offset": 61,
      "base_timestamp": 1688558918102,
      "max_timestamp": 1688558918102,
      "delta_offset": 50,
      "archiver_term": 25,
      "segment_term": 24,
      "delta_offset_end": 52,
      "sname_format": 2
    },
    "64-25-v1.log": {
      "is_compacted": false,
      "size_bytes": 801,
      "committed_offset": 65,
      "base_offset": 64,
      "base_timestamp": 1688558927819,
      "max_timestamp": 1688558927819,
      "delta_offset": 52,
      "archiver_term": 26,
      "segment_term": 25,
      "delta_offset_end": 54,
      "sname_format": 2
    }
  }
}

piyushredpanda · 2023-07-07T21:35:53Z

@andijcr what is next here?

piyushredpanda · 2023-07-08T03:15:10Z

This is showing up as failing a lot.

andijcr · 2023-07-10T14:15:17Z

finally got some more output out of rp-storage-tool

[kafka/topic-MinimalWorkload-373f4aa2-e4c5-4cc1-a8e0-27f8e7915612/0_15] Segment 24 has delta lower than previous
[2023-07-10T11:18:06Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-373f4aa2-e4c5-4cc1-a8e0-27f8e7915612/0_15] Segment 25 has delta lower than previous
[2023-07-10T11:18:06Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-373f4aa2-e4c5-4cc1-a8e0-27f8e7915612/0_15] Segment 2 has delta lower than previous
[2023-07-10T11:18:06Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-373f4aa2-e4c5-4cc1-a8e0-27f8e7915612/0_15] Segment 5 has delta lower than previous
[2023-07-10T11:18:06Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-373f4aa2-e4c5-4cc1-a8e0-27f8e7915612/0_15] Segment 7 has delta lower than previous
[2023-07-10T11:18:06Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-373f4aa2-e4c5-4cc1-a8e0-27f8e7915612/0_15] Segment 8 has delta lower than previous
[2023-07-10T11:18:06Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-373f4aa2-e4c5-4cc1-a8e0-27f8e7915612/0_15] Segment 11 has delta lower than previous
[2023-07-10T11:18:06Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-373f4aa2-e4c5-4cc1-a8e0-27f8e7915612/0_15] Segment 13 has delta lower than previous
[2023-07-10T11:18:06Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-373f4aa2-e4c5-4cc1-a8e0-27f8e7915612/0_15] Segment 14 has delta lower than previous
[2023-07-10T11:18:06Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-373f4aa2-e4c5-4cc1-a8e0-27f8e7915612/0_15] Segment 17 has delta lower than previous
[2023-07-10T11:18:06Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-373f4aa2-e4c5-4cc1-a8e0-27f8e7915612/0_15] Segment 20 has delta lower than previous

unfortunately, cloud_diagnostic.zip does not contains the file, but nothing that can't be fixed in ducktape

andijcr · 2023-07-10T21:17:36Z

It's caused by rp-storage-tool not accounting for old segment_meta with a defaulted delta_offset_end. In redpanda, this is already handled in partition_manifest by checking the value of delta_offset_end != model::offset to decide if it's valid or not.

in the decoded manifest, delta_offset_end that have the value 9223372036854775808 should be ignored. (it should be a negative number, std::numeric_limits<int64_t>::min(), not sure if the error is in the datatype used or in the json handling of int64_t)

rp-storage-tool stderr output: preoutput=b'
[2023-07-10T15:39:39Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-f845d5ef-79fc-4fe9-bd7a-b29937805023/0_15] Segment 2 has delta lower than previous
[2023-07-10T15:39:39Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-f845d5ef-79fc-4fe9-bd7a-b29937805023/0_15] Segment 5 has delta lower than previous
[2023-07-10T15:39:39Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-f845d5ef-79fc-4fe9-bd7a-b29937805023/0_15] Segment 8 has delta lower than previous
[2023-07-10T15:39:39Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-f845d5ef-79fc-4fe9-bd7a-b29937805023/0_15] Segment 10 has delta lower than previous
[2023-07-10T15:39:39Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-f845d5ef-79fc-4fe9-bd7a-b29937805023/0_15] Segment 11 has delta lower than previous
[2023-07-10T15:39:39Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-f845d5ef-79fc-4fe9-bd7a-b29937805023/0_15] Segment 14 has delta lower than previous
[2023-07-10T15:39:39Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-f845d5ef-79fc-4fe9-bd7a-b29937805023/0_15] Segment 17 has delta lower than previous
[2023-07-10T15:39:39Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-f845d5ef-79fc-4fe9-bd7a-b29937805023/0_15] Segment 19 has delta lower than previous
[2023-07-10T15:39:39Z WARN  rp_storage_tool::bucket_reader] [kafka/topic-MinimalWorkload-f845d5ef-79fc-4fe9-bd7a-b29937805023/0_15] Segment 21 has delta lower than previous

manifest v1, json
manifest_v1.json.txt
looking at the first error message:

    "0-1-v1.log": {
      "is_compacted": false,
      "size_bytes": 726,
      "committed_offset": 1,
      "base_offset": 0,
      "base_timestamp": 1689002716441,
      "max_timestamp": 1689002716441,
      "delta_offset": 0,
      "archiver_term": 2,
      "segment_term": 1
    },
    "2-2-v1.log": {
      "is_compacted": false,
      "size_bytes": 896,
      "committed_offset": 4,
      "base_offset": 2,
      "base_timestamp": 1689002743839,
      "max_timestamp": 1689002744739,
      "delta_offset": 1,
      "archiver_term": 3,
      "segment_term": 2
    },

segment.base_offset=2 does not have a predecessor with delta_offset_end, so i could not trigger rp-storage-tool.

manifest v2 in bin format (and decoded json representation)
30000000_meta_kafka_topic-MinimalWorkload-f845d5ef-79fc-4fe9-bd7a-b29937805023_0_15_manifest.bin.0.21.0.22.1689002716441.1689002895505deleteme.txt
manifest_v2.bin.json.txt

  "segments": {
    "0-1-v1.log": {
      "base_offset": 0,
      "committed_offset": 1,
      "is_compacted": false,
      "size_bytes": 726,
      "archiver_term": 2,
      "delta_offset": 0,
      "base_timestamp": 1689002716441,
      "max_timestamp": 1689002716441,
      "ntp_revision": 15,
      "sname_format": 1,
      "segment_term": 1,
      "delta_offset_end": 9223372036854775808
    },
    "8-4-v1.log": {
      "base_offset": 8,
      "committed_offset": 9,
      "is_compacted": false,
      "size_bytes": 781,
      "archiver_term": 5,
      "delta_offset": 5,
      "base_timestamp": 1689002805826,
      "max_timestamp": 1689002805826,
      "ntp_revision": 15,
      "sname_format": 1,
      "segment_term": 4,
      "delta_offset_end": 9223372036854775808
    },
    "10-4-v1.log": {
      "base_offset": 10,
      "committed_offset": 10,
      "is_compacted": false,
      "size_bytes": 115,
      "archiver_term": 5,
      "delta_offset": 7,
      "base_timestamp": 1689002825850,
      "max_timestamp": 1689002825850,
      "ntp_revision": 15,
      "sname_format": 1,
      "segment_term": 4,
      "delta_offset_end": 9223372036854775808
    },
    "14-6-v1.log": {
      "base_offset": 14,
      "committed_offset": 16,
      "is_compacted": false,
      "size_bytes": 898,
      "archiver_term": 7,
      "delta_offset": 10,
      "base_timestamp": 1689002868026,
      "max_timestamp": 1689002868026,
      "ntp_revision": 15,
      "sname_format": 1,
      "segment_term": 6,
      "delta_offset_end": 9223372036854775808
    },
    "2-2-v1.log": {
      "base_offset": 2,
      "committed_offset": 4,
      "is_compacted": false,
      "size_bytes": 896,
      "archiver_term": 3,
      "delta_offset": 1,
      "base_timestamp": 1689002743839,
      "max_timestamp": 1689002744739,
      "ntp_revision": 15,
      "sname_format": 1,
      "segment_term": 2,
      "delta_offset_end": 9223372036854775808
    },

piyushredpanda · 2023-07-11T05:21:19Z

@andijcr : Sorry not clear, so a tool defect and no real RP issue?

andijcr · 2023-07-11T07:06:09Z

@andijcr : Sorry not clear, so a tool defect and no real RP issue?

tool defect

VladLazar added kind/bug Something isn't working ci-failure area/cloud-storage Shadow indexing subsystem labels Jul 3, 2023

VladLazar mentioned this issue Jul 3, 2023

CI Failure (RuntimeError("Object storage scrub detected fatal anomalies of type {'ntpr_bad_deltas'}")) in rptest.tests.workload_upgrade_runner_test.RedpandaUpgradeTest.test_workloads_through_releases #11841

Closed

jcsp added the sev/medium Bugs that do not meet criteria for high or critical, but are more severe than low. label Jul 3, 2023

andijcr assigned andijcr and unassigned andijcr Jul 3, 2023

jcsp mentioned this issue Jul 3, 2023

tests: refactor franz_go_verifiable_test & fix its use of tiny cache size #11836

Merged

7 tasks

andijcr self-assigned this Jul 4, 2023

andijcr mentioned this issue Jul 5, 2023

Tests: partial upgrade with rollback #11848

Open

7 tasks

NyaliaLui mentioned this issue Jul 7, 2023

sr: retry create ACLs for the ephemeral principal #11563

Merged

7 tasks

piyushredpanda mentioned this issue Jul 8, 2023

Fix for delete-records tests that don't handle RpkExceptions #11963

Merged

7 tasks

jcsp mentioned this issue Jul 10, 2023

storage: fix coroutine iteration over _segs #11956

Merged

7 tasks

This was referenced Jul 10, 2023

Wait for leader to be elected before finishing abort cancel #11905

Merged

Made prevote phase not resetting last heartbeat timestamp #11726

Merged

andijcr mentioned this issue Jul 10, 2023

t/segment_meta_cstore_test: fix oob access in test vector #11972

Merged

7 tasks

jcsp mentioned this issue Jul 11, 2023

tools: fix legacy metadata handling in rp-storage-tool #12013

Merged

7 tasks

VladLazar added sev/low Bugs which are non-functional paper cuts, e.g. typos, issues in log messages and removed sev/medium Bugs that do not meet criteria for high or critical, but are more severe than low. labels Jul 11, 2023

jcsp closed this as completed in #12013 Jul 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI Failure (bad_deltas anomaly) in `RedpandaUpgradeTest.test_workloads_through_releases` #11842

CI Failure (bad_deltas anomaly) in `RedpandaUpgradeTest.test_workloads_through_releases` #11842

VladLazar commented Jul 3, 2023

VladLazar commented Jul 3, 2023

jcsp commented Jul 3, 2023

andijcr commented Jul 3, 2023

andijcr commented Jul 5, 2023

piyushredpanda commented Jul 7, 2023

piyushredpanda commented Jul 8, 2023

andijcr commented Jul 10, 2023

andijcr commented Jul 10, 2023

piyushredpanda commented Jul 11, 2023

andijcr commented Jul 11, 2023

CI Failure (bad_deltas anomaly) in RedpandaUpgradeTest.test_workloads_through_releases #11842

CI Failure (bad_deltas anomaly) in RedpandaUpgradeTest.test_workloads_through_releases #11842

Comments

VladLazar commented Jul 3, 2023

VladLazar commented Jul 3, 2023

jcsp commented Jul 3, 2023

andijcr commented Jul 3, 2023

andijcr commented Jul 5, 2023

piyushredpanda commented Jul 7, 2023

piyushredpanda commented Jul 8, 2023

andijcr commented Jul 10, 2023

andijcr commented Jul 10, 2023

piyushredpanda commented Jul 11, 2023

andijcr commented Jul 11, 2023

CI Failure (bad_deltas anomaly) in `RedpandaUpgradeTest.test_workloads_through_releases` #11842

CI Failure (bad_deltas anomaly) in `RedpandaUpgradeTest.test_workloads_through_releases` #11842