Ducktape test to execute workloads through multi-release upgrades #8253

andijcr · 2023-01-16T16:21:02Z

A framework to execute tests on a sequence of redpanda versions, composed of 2 components:

PWorkload, an interface to implement
RedpandaUpgradeTest, a test that runs a collection of PWorkloads through a cluster while upgrading it

A Workload can implement PWorkload, spin up an external producer/consumer, and check progress on the external service in a stable and upgrading cluster.
Workloads run in parallel, so care should be taken not to disrupt other workloads, like changing cluster properties or removing nodes.

Various tests implementing a upgrading+testing pattern can be moved to this test to reduce the number of tests and total ci execution time while doing a more rigorous test.

Fixes #7310

Backports Required

UX Changes

Release Notes

none

tests/rptest/tests/workload_upgrade_runner_test.py

andrwng

Just a couple thoughts on things to consider w.r.t versioning, nothing really blocking. I think I need to digest the workload protocol and adapters a bit more and let them sit, but at first glance this looks great!

tests/rptest/services/redpanda_installer.py

tests/rptest/services/workload_protocol.py

tests/rptest/tests/workload_upgrade_runner_test.py

tests/rptest/services/redpanda_installer.py

tests/rptest/tests/redpanda_test.py

tests/rptest/tests/workload_upgrade_runner_test.py

andijcr · 2023-06-28T09:36:21Z

https://buildkite.com/redpanda/redpanda/builds/32089#0188fed7-8481-4e6d-92ae-d25de019d22f
failures are
#10836
#11449
#11169

and a failure related to the pr in debug mode:

File "/root/tests/rptest/tests/workload_producer_consumer.py", line 86, in end
    consumer.wait()

edit: actually the timeout is before the check in the test itself, it's related to kgo consumer

andijcr · 2023-06-28T09:52:53Z

/ci-repeat 3 debug skip-debug dt-repeat=3 tests/rptest/tests/workload_upgrade_runner_test.py::RedpandaUpgradeTest.test_workloads_through_releases

VladLazar

The test code looks fine to me. Does the new test pass reliably-ish?

VladLazar · 2023-06-28T15:48:57Z

tests/rptest/tests/redpanda_test.py

+                    timeout_sec=30,
+                    backoff_sec=1)
+
+            old_version = current_version


nit: is this assignment needed?

old_version is passed on line 258 to mid_upgrade_check() , for the nodes that are not yet updated

VladLazar · 2023-06-28T15:52:36Z

tests/rptest/services/redpanda_installer.py

+        head_line = self.head_version()[0:2]
+        oldest_supported_line = (head_line[0] - 1, head_line[1])
+        latest_unsupported_line = (oldest_supported_line[0],
+                                   oldest_supported_line[1] - 1)
+        if latest_unsupported_line[1] == 0:
+            # if going back, version vX.0 is v(X-1).3
+            latest_unsupported_line = (latest_unsupported_line[0] - 1, 3)
+        return latest_unsupported_line


nit: it would be nice to wrap this version back and forth in a type (i.e. RedpandaVersionTriple becomes a dataclass and this stuff is wrapped in a member)

The pain with a dataclass is that we use "head" as a version in the codebase, and sometimes we need to convert it to and actual triple. That requires RedpandaInstaller to do it, and a query to a running instance of redpanda in principle, breaking a bit the encapsulation of a dataclass.

VladLazar · 2023-06-28T16:18:15Z

tests/rptest/tests/workload_producer_consumer.py

+        # old release have a bug where cloud_storage_enabled is sticky. forcing a leadership transfer is a workaround for this
+        #Admin(self.ctx.redpanda).partition_transfer_leadership("kafka", self.topic.name, 1)


I remember discussing this. Did this solution work?

Forgot to remove the comment, but it didn't appear to work reliably. The solution used here is to just start with v22.3 for cloud storage

VladLazar · 2023-06-28T16:18:59Z

tests/rptest/services/workload_protocol.py

+        return PWorkload.DONE
+
+    @abstractmethod
+    def progress(self, version: RedpandaVersionTriple) -> int:


nit: I'd rename this to something like on_cluster_upgraded

VladLazar · 2023-06-28T16:19:21Z

tests/rptest/services/workload_protocol.py

+        """
+        return
+
+    def partial_progress(self, versions: dict[Any,


nit: I'd rename this to on_partial_cluster_upgrade

VladLazar · 2023-06-28T16:22:15Z

tests/rptest/services/workload_protocol.py

+    def partial_progress(self, versions: dict[Any,
+                                              RedpandaVersionTriple]) -> int:
+        """
+        This method is called while upgrading a cluster, after each node is upgraded.


nit: Is this comment correct? Looks like this is called after half the cluster was upgraded

Right, comment out of sync

and some other python related typing

this callback is used to run checks mid-upgrade

This Protocol defines the interface that a workload has to implement to run inside the workload upgrade runner, introduced in the next commit. Only the methods marked as abstract needs to be implemented, the other are optional. additionally, tests/rptest/tests/workload_dummy.py provides an example implementation

andijcr · 2023-06-29T08:22:51Z

force push: fixed a missing function rename in the code

andijcr · 2023-06-29T13:50:56Z

/ci-repeat 1 debug tests/rptest/tests/workload_producer_consumer.py

the tests setup a producer and consumer, ensures that data gets written in cloud storage, and checks the content of the partition manifest to ensure progress

this test is a runner for a collection PWorkload. it will create an upgrade path, insert patch downgrades, setup a cluster and run the workloads concurrently against the cluster. at the end of the test it will report failed workloads. WorkloadAdapter is a wrapper to keep track workload state and to store any thrown exception

the method is used in a loop by RedpandaUpgradeTest and it's a bit noisy

andijcr · 2023-06-29T14:45:27Z

/ci-repeat 1 debug tests/rptest/tests/workload_upgrade_runner_test.py

andijcr · 2023-06-29T15:47:20Z

force push: comment fix and new commit to skip-debug since

andijcr · 2023-06-29T20:09:18Z

https://buildkite.com/redpanda/redpanda/builds/32226#0189080c-5ffe-40e2-b5be-257439b688c7
issues are
#11062
#11508

and one for this test, in debug mode. test is marked as @skip_debug_mode, so there is some weird interaction

====================================================================================================
test_id:    rptest.tests.workload_upgrade_runner_test.RedpandaUpgradeTest.test_workloads_through_releases
status:     FAIL
run time:   17.606 seconds


    JSONDecodeError('Expecting value: line 1 column 1 (char 0)')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 155, in wrapped
    self.redpanda.stop_and_scrub_object_storage()
  File "/root/tests/rptest/services/redpanda.py", line 3657, in stop_and_scrub_object_storage
    report = json.loads(output)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

edit: last issue was solved by moving @skip_debug_build to the top of the stack of decorators

transition v23.1 -> v23.2 seems to be flaky in debug mode and a producer/consumer workload running

andijcr · 2023-06-30T12:16:26Z

https://buildkite.com/redpanda/redpanda/builds/32267#01890bd8-4bf1-4453-b99b-d35995690819
issues
#9697
#11790

VladLazar

LGTM

andijcr mentioned this pull request Feb 15, 2023

tools: Update offline_log_viewer for serde-encoded controller logs & test it #5293

Closed

andijcr force-pushed the feat/test/upgrade_tests branch from 57e5257 to 4f2d12a Compare March 10, 2023 11:33

andijcr force-pushed the feat/test/upgrade_tests branch 2 times, most recently from e6c2394 to eb08344 Compare March 24, 2023 12:12

andijcr marked this pull request as ready for review March 24, 2023 12:13

andijcr added area/redpanda area/tests labels Mar 24, 2023

andijcr requested review from andrwng and jcsp March 24, 2023 12:23

andijcr force-pushed the feat/test/upgrade_tests branch from eb08344 to a55ba9c Compare March 24, 2023 16:08

github-actions bot removed the area/redpanda label Mar 24, 2023

andijcr changed the title ~~Draft: Feat/test/upgrade tests~~ test that executes workloads during multi-release upgrades Mar 24, 2023

andijcr changed the title ~~test that executes workloads during multi-release upgrades~~ Ducktape test to execute workloads through multi-release upgrades Mar 27, 2023

jcsp reviewed Apr 6, 2023

View reviewed changes

tests/rptest/tests/workload_upgrade_runner_test.py Outdated Show resolved Hide resolved

andijcr mentioned this pull request Apr 6, 2023

Add an upgrade test to check the behaviour of partition_manifest serialization while performing an upgrade #9891

Closed

andijcr force-pushed the feat/test/upgrade_tests branch 2 times, most recently from e75422a to c086131 Compare April 26, 2023 15:42

andijcr requested review from jcsp, NyaliaLui and VladLazar April 26, 2023 16:06

andijcr force-pushed the feat/test/upgrade_tests branch from c086131 to 3178ba8 Compare April 26, 2023 18:32

andrwng reviewed Apr 26, 2023

View reviewed changes

andijcr force-pushed the feat/test/upgrade_tests branch from 3178ba8 to 877e100 Compare April 27, 2023 11:28

andijcr requested a review from andrwng May 17, 2023 21:56

andijcr force-pushed the feat/test/upgrade_tests branch from 877e100 to d6acf1c Compare May 18, 2023 11:55

VladLazar reviewed May 29, 2023

View reviewed changes

andijcr force-pushed the feat/test/upgrade_tests branch from a662522 to 1e112fe Compare June 12, 2023 15:00

andijcr force-pushed the feat/test/upgrade_tests branch from 1e112fe to d1c1ead Compare June 20, 2023 15:34

andijcr force-pushed the feat/test/upgrade_tests branch from d1c1ead to a246d09 Compare June 27, 2023 20:35

andijcr force-pushed the feat/test/upgrade_tests branch from a246d09 to 1c0352a Compare June 28, 2023 13:46

VladLazar reviewed Jun 28, 2023

View reviewed changes

andijcr added 4 commits June 28, 2023 23:00

tests: RedpandaVersion typing

52abd72

and some other python related typing

tests/redpanda_test: upgrade_through_versions now accepts a callback

f332fca

this callback is used to run checks mid-upgrade

tsts/workload_license: tests enterprise license during upgrades

a1b9b56

andijcr force-pushed the feat/test/upgrade_tests branch 2 times, most recently from faadc8d to be61a1f Compare June 29, 2023 07:45

andijcr force-pushed the feat/test/upgrade_tests branch from e7d7849 to 1c590e8 Compare June 29, 2023 13:48

andijcr added 3 commits June 29, 2023 16:29

tests/workload_producer_consumer: ProducerConsumerWorkload

59af831

the tests setup a producer and consumer, ensures that data gets written in cloud storage, and checks the content of the partition manifest to ensure progress

t/redpanda_installer: cache for highest_from_previous_feature_version

16289e7

the method is used in a loop by RedpandaUpgradeTest and it's a bit noisy

andijcr force-pushed the feat/test/upgrade_tests branch from 1c590e8 to c8d2a70 Compare June 29, 2023 15:46

andijcr requested a review from VladLazar June 29, 2023 20:14

t/workload_upgrade_runner_test: disable test in debug mode

0913b54

transition v23.1 -> v23.2 seems to be flaky in debug mode and a producer/consumer workload running

andijcr force-pushed the feat/test/upgrade_tests branch from c8d2a70 to 0913b54 Compare June 30, 2023 09:26

VladLazar approved these changes Jun 30, 2023

View reviewed changes

jcsp approved these changes Jun 30, 2023

View reviewed changes

jcsp merged commit 5b5b726 into redpanda-data:dev Jun 30, 2023

andijcr deleted the feat/test/upgrade_tests branch June 30, 2023 15:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ducktape test to execute workloads through multi-release upgrades #8253

Ducktape test to execute workloads through multi-release upgrades #8253

andijcr commented Jan 16, 2023 •

edited

Loading

andrwng left a comment

andijcr commented Jun 28, 2023 •

edited

Loading

andijcr commented Jun 28, 2023

VladLazar left a comment

VladLazar Jun 28, 2023

andijcr Jun 28, 2023

VladLazar Jun 28, 2023

andijcr Jun 28, 2023

VladLazar Jun 28, 2023

andijcr Jun 28, 2023

VladLazar Jun 28, 2023

andijcr Jun 28, 2023

VladLazar Jun 28, 2023

andijcr Jun 28, 2023

VladLazar Jun 28, 2023

andijcr Jun 28, 2023

andijcr commented Jun 29, 2023

andijcr commented Jun 29, 2023

andijcr commented Jun 29, 2023

andijcr commented Jun 29, 2023

andijcr commented Jun 29, 2023 •

edited

Loading

andijcr commented Jun 30, 2023

VladLazar left a comment

		# old release have a bug where cloud_storage_enabled is sticky. forcing a leadership transfer is a workaround for this
		#Admin(self.ctx.redpanda).partition_transfer_leadership("kafka", self.topic.name, 1)

Ducktape test to execute workloads through multi-release upgrades #8253

Ducktape test to execute workloads through multi-release upgrades #8253

Conversation

andijcr commented Jan 16, 2023 • edited Loading

Backports Required

UX Changes

Release Notes

andrwng left a comment

Choose a reason for hiding this comment

andijcr commented Jun 28, 2023 • edited Loading

andijcr commented Jun 28, 2023

VladLazar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andijcr commented Jun 29, 2023

andijcr commented Jun 29, 2023

andijcr commented Jun 29, 2023

andijcr commented Jun 29, 2023

andijcr commented Jun 29, 2023 • edited Loading

andijcr commented Jun 30, 2023

VladLazar left a comment

Choose a reason for hiding this comment

andijcr commented Jan 16, 2023 •

edited

Loading

andijcr commented Jun 28, 2023 •

edited

Loading

andijcr commented Jun 29, 2023 •

edited

Loading