CI Failure (timeout in kafka-topics.sh create topic) in RackAwarePlacementTest.test_replica_placement in DEBUG builds

[andijcr]

https://buildkite.com/redpanda/redpanda/builds/30776#018894e7-2322-4fdc-afec-cb028f9cc9da
https://buildkite.com/redpanda/redpanda/builds/30776#018894e7-2322-4fdc-afec-cb028f9cc9da

Module: rptest.tests.rack_aware_replica_placement_test
Class:  RackAwarePlacementTest
Method: test_replica_placement
Arguments:
{
  "num_partitions": 400,
  "num_topics": 2,
  "rack_layout_str": "ooooFF",
  "replication_factor": 3
}

====================================================================================================
test_id:    rptest.tests.rack_aware_replica_placement_test.RackAwarePlacementTest.test_replica_placement.rack_layout_str=ooooFF.num_partitions=400.replication_factor=3.num_topics=2
status:     FAIL
run time:   2 minutes 5.245 seconds


    CalledProcessError(1, ['/opt/kafka-3.0.0/bin/kafka-topics.sh', '--bootstrap-server', 'docker-rp-5:9092,docker-rp-16:9092,docker-rp-4:9092,docker-rp-17:9092,docker-rp-14:9092,docker-rp-15:9092', '--create', '--topic', 'topic-rrtjhdlqrb', '--partitions', '400', '--replication-factor', '3', '--config', 'cleanup.policy=delete'])
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 79, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/rack_aware_replica_placement_test.py", line 147, in test_replica_placement
    self._create_topic(topic)
  File "/root/tests/rptest/tests/rack_aware_replica_placement_test.py", line 50, in _create_topic
    self.client().create_topic(topic)
  File "/root/tests/rptest/clients/default.py", line 56, in create_topic
    client.create_topic(spec)
  File "/root/tests/rptest/clients/kafka_cli_tools.py", line 84, in create_topic
    return self._run("kafka-topics.sh", args)
  File "/root/tests/rptest/clients/kafka_cli_tools.py", line 402, in _run
    return self._execute(cmd)
  File "/root/tests/rptest/clients/kafka_cli_tools.py", line 412, in _execute
    res = subprocess.check_output(cmd,
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/opt/kafka-3.0.0/bin/kafka-topics.sh', '--bootstrap-server', 'docker-rp-5:9092,docker-rp-16:9092,docker-rp-4:9092,docker-rp-17:9092,docker-rp-14:9092,docker-rp-15:9092', '--create', '--topic', 'topic-rrtjhdlqrb', '--partitions', '400', '--replication-factor', '3', '--config', 'cleanup.policy=delete']' returned non-zero exit status 1.

[andijcr]

https://buildkite.com/redpanda/redpanda/builds/30944#01889eed-271f-4b23-9c89-2c4d75045fe0
https://buildkite.com/redpanda/redpanda/builds/30951#01889fb4-791f-42f2-bbce-9641c33f10e5

[michael-redpanda]

https://buildkite.com/redpanda/redpanda/builds/31065#0188ae42-0537-40fd-a4ac-afa2b4cd4c75a
https://buildkite.com/redpanda/redpanda/builds/31045#0188acc9-4842-4b11-ab19-336de3b87f11
https://buildkite.com/redpanda/redpanda/builds/30988#0188a23f-4fc5-44f8-8467-e61982af21cf
https://buildkite.com/redpanda/redpanda/builds/31035#0188a900-fe9f-4f99-88d5-dea3ad7cea3a
https://buildkite.com/redpanda/redpanda/builds/31018#0188a409-e48d-48fd-840b-3c4a820e5a0ba

[michael-redpanda]

https://buildkite.com/redpanda/redpanda/builds/31188#0188b564-dd6c-4c1c-9d48-2b0c9ace4255

[ztlpn]

https://buildkite.com/redpanda/redpanda/builds/31141#0188b20d-4335-4f5e-8e36-51f7d252894d
https://buildkite.com/redpanda/redpanda/builds/31184#0188b53e-0446-41ae-a3e7-f45dc7e02d11

[michael-redpanda]

https://buildkite.com/redpanda/redpanda/builds/31188#0188b564-dd6c-4c1c-9d48-2b0c9ace4255

[ztlpn]

https://buildkite.com/redpanda/redpanda/builds/31355#0188bf9a-3d4c-4ceb-a6ad-9e3da54b5f50

[ztlpn]

https://buildkite.com/redpanda/redpanda/builds/31432#0188c34d-5cc4-4357-8e82-ccca09538b5b

[travisdowns]

Underlying cause is that topic creation fails due to timeout, I think this can only be seen in the debug log:

[DEBUG - 2023-06-19 00:48:52,926 - kafka_cli_tools - create_topic - lineno:67]: Creating topic: topic-qpvznbjaqd
[DEBUG - 2023-06-19 00:48:52,927 - kafka_cli_tools - _execute - lineno:410]: Executing command: ['/opt/kafka-3.0.0/bin/kafka-topics.sh', '--bootstrap-server', 'docker-rp-9:9092,docker-rp-6:9092,docker-rp-19:9092,docker-rp-5:9092,docker-rp-7:9092,docker-rp-8:9092', '--create', '--topic', 'topic-qpvznbjaqd', '--partitions', '400', '--replication-factor', '3', '--config', 'cleanup.policy=delete']
[DEBUG - 2023-06-19 00:49:54,807 - kafka_cli_tools - _execute - lineno:418]: Error (1) executing command: Error while executing topic command : Call(callName=createTopics, deadlineMs=1687135794401, tries=1, nextAllowedTryMs=1687135794559) timed out at 1687135794459 after 1 attempt(s)
[2023-06-19 00:49:54,464] ERROR org.apache.kafka.common.errors.TimeoutException: Call(callName=createTopics, deadlineMs=1687135794401, tries=1, nextAllowedTryMs=1687135794559) timed out at 1687135794459 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.DisconnectException: Cancelled createTopics request with correlation id 3 due to node 1 being disconnected
 (kafka.admin.TopicCommand$)

Looks like the timeout occurred about 62 seconds after issuing the command (so probably a 60 second timeout) which is a long time to create a 400 partition topic.

[travisdowns]

Three more instances overnight:

FAIL test: RackAwarePlacementTest.test_replica_placement.rack_layout_str=ooooFF.num_partitions=400.replication_factor=3.num_topics=2 (3/31 runs)
failure at 2023-06-19T01:19:57.960Z: CalledProcessError(1, ['/opt/kafka-3.0.0/bin/kafka-topics.sh', '--bootstrap-server', 'docker-rp-9:9092,docker-rp-6:9092,docker-rp-19:9092,docker-rp-5:9092,docker-rp-7:9092,docker-rp-8:9092', '--create', '--topic', 'topic-qpvznbjaqd', '--partitions', '400', '--replication-factor', '3', '--config', 'cleanup.policy=delete'])
on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/31567#0188d102-709f-4dfa-a1bf-47647df79717
failure at 2023-06-19T14:53:48.777Z: CalledProcessError(1, ['/opt/kafka-3.0.0/bin/kafka-topics.sh', '--bootstrap-server', 'docker-rp-8:9092,docker-rp-23:9092,docker-rp-24:9092,docker-rp-21:9092,docker-rp-18:9092,docker-rp-22:9092', '--create', '--topic', 'topic-xvrrqvacjj', '--partitions', '400', '--replication-factor', '3', '--config', 'cleanup.policy=delete'])
on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/31599#0188d3ec-265a-4582-997f-aad70ddfe771
failure at 2023-06-19T07:07:44.877Z: CalledProcessError(1, ['/opt/kafka-3.0.0/bin/kafka-topics.sh', '--bootstrap-server', 'docker-rp-2:9092,docker-rp-1:9092,docker-rp-11:9092,docker-rp-12:9092,docker-rp-18:9092,docker-rp-19:9092', '--create', '--topic', 'topic-kieseoggdk', '--partitions', '400', '--replication-factor', '3', '--config', 'cleanup.policy=delete'])
on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/31580#0188d241-8759-487b-8fb1-dba0aae8251c

[travisdowns]

One more:

FAIL test: RackAwarePlacementTest.test_replica_placement.rack_layout_str=xxYYzz.num_partitions=400.replication_factor=3.num_topics=2 (1/31 runs)
failure at 2023-06-19T16:03:55.178Z: CalledProcessError(1, ['/opt/kafka-3.0.0/bin/kafka-topics.sh', '--bootstrap-server', 'docker-rp-20:9092,docker-rp-10:9092,docker-rp-9:9092,docker-rp-19:9092,docker-rp-17:9092,docker-rp-8:9092', '--create', '--topic', 'topic-zfyjepslfe', '--partitions', '400', '--replication-factor', '3', '--config', 'cleanup.policy=delete'])
on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/31600#0188d42b-92b5-463b-9b9e-9303916f63a4

[bharathv]

Looks like it is only failing with debug build + 400 partition combination.. perhaps one of those slow debug build issues, I'll dig more tomorrow.

[michael-redpanda]

https://buildkite.com/redpanda/redpanda/builds/31404#0188c1f0-beae-4585-8d13-9c74b0115d81

[michael-redpanda]

https://buildkite.com/redpanda/redpanda/builds/31611#0188d489-fb1c-4494-ad16-477c664829b7

[bharathv]

Disabled via #11590