CI Failure (rpk cluster maintenance disable timed out) in PartitionBalancerTest.test_maintenance_mode

[dlex]

on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/29202#01882328-a2b0-4c44-935f-ec0b02cd5046

Module: rptest.tests.partition_balancer_test
Class:  PartitionBalancerTest
Method: test_maintenance_mode
Arguments:
{
  "kill_same_node": false
}

test_id:    rptest.tests.partition_balancer_test.PartitionBalancerTest.test_maintenance_mode.kill_same_node=False
status:     FAIL
run time:   1 minute 17.140 seconds


    RpkException('command /var/lib/buildkite-agent/builds/buildkite-amd64-xfs-builders-i-09b5b635b195e82c5-1/redpanda/redpanda/vbuild/redpanda_installs/ci/bin/rpk --api-urls docker-rp-18:9644,docker-rp-12:9644,docker-rp-13:9644,docker-rp-14:9644,docker-rp-11:9644 cluster maintenance disable 4 timed out')
Traceback (most recent call last):
  File "/root/tests/rptest/clients/rpk.py", line 847, in _execute
    output, error = p.communicate(input=stdin, timeout=timeout)
  File "/usr/lib/python3.10/subprocess.py", line 1152, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.10/subprocess.py", line 2004, in _communicate
    self._check_timeout(endtime, orig_timeout, stdout, stderr)
  File "/usr/lib/python3.10/subprocess.py", line 1196, in _check_timeout
    raise TimeoutExpired(
subprocess.TimeoutExpired: Command '['/var/lib/buildkite-agent/builds/buildkite-amd64-xfs-builders-i-09b5b635b195e82c5-1/redpanda/redpanda/vbuild/redpanda_installs/ci/bin/rpk', '--api-urls', 'docker-rp-18:9644,docker-rp-12:9644,docker-rp-13:9644,docker-rp-14:9644,docker-rp-11:9644', 'cluster', 'maintenance', 'disable', '4']' timed out after 30 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 49, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/partition_balancer_test.py", line 799, in test_maintenance_mode
    rpk.cluster_maintenance_disable(node)
  File "/root/tests/rptest/clients/rpk.py", line 890, in cluster_maintenance_disable
    return self._execute(cmd)
  File "/root/tests/rptest/clients/rpk.py", line 850, in _execute
    raise RpkException(f"command {' '.join(cmd)} timed out")

[VladLazar]

FAIL test: PartitionBalancerTest.test_maintenance_mode.kill_same_node=False (1/67 runs)
  failure at 2023-05-22T14:42:14.910Z: RpkException('command /var/lib/buildkite-agent/builds/buildkite-bk-amd64-xfs-builders-i-0b99a34fc5d3377e4-1/redpanda/redpanda/vbuild/redpanda_installs/ci/bin/rpk --api-urls docker-rp-11:9644,docker-rp-12:9644,docker-rp-13:9644,docker-rp-14:9644,docker-rp-15:9644 cluster maintenance disable 3 timed out')
      on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/29583#018843b9-86de-484f-aad1-807a682ce987

[NyaliaLui]

https://buildkite.com/redpanda/redpanda/builds/29687#018848f7-3797-4c99-b906-6f972363e67f

[andijcr]

https://buildkite.com/redpanda/vtools/builds/7904#01887db9-189b-4f9b-a6ff-44bcae6b9c3a

[andijcr]

https://buildkite.com/redpanda/vtools/builds/8001#01889c9e-a0a1-4dca-97c0-f974b062cc80

[michael-redpanda]

https://buildkite.com/redpanda/redpanda/builds/31334#0188bdb2-22f0-4034-ab6b-3775fcc09ccb

[abhijat]

https://buildkite.com/redpanda/redpanda/builds/31793#0188e3c2-e539-4eff-bd44-3f1305eca933

[dotnwat]

https://buildkite.com/redpanda/redpanda/builds/31966#0188f8ac-7501-4000-91d2-28462d7b6a5c