CI Failure (startup failure) in ScalingUpTest.test_adding_multiple_nodes_to_the_cluster

[NyaliaLui]

https://buildkite.com/redpanda/redpanda/builds/29955#018856ae-e969-4d34-92a8-a9272a38abd3

Module: rptest.tests.scaling_up_test
Class:  ScalingUpTest
Method: test_adding_multiple_nodes_to_the_cluster
Arguments:
{
  "partition_count": 20
}

test_id:    rptest.tests.scaling_up_test.ScalingUpTest.test_adding_multiple_nodes_to_the_cluster.partition_count=20
status:     FAIL
run time:   1 minute 30.717 seconds


    TimeoutError('Redpanda service docker-rp-6 failed to start within 20 sec')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 49, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/scaling_up_test.py", line 195, in test_adding_multiple_nodes_to_the_cluster
    self.redpanda.start_node(n)
  File "/root/tests/rptest/services/redpanda.py", line 1796, in start_node
    self.start_service(node, start_rp)
  File "/root/tests/rptest/services/redpanda.py", line 1868, in start_service
    start()
  File "/root/tests/rptest/services/redpanda.py", line 1787, in start_rp
    wait_until(
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Redpanda service docker-rp-6 failed to start within 20 sec

[NyaliaLui]

Looking at the logs of docker-rp-6, there were alot of unknown RPC methods

DEBUG 2023-05-26 06:31:56,908 [shard 0] rpc - rpc_server.cc:176 - Received a request for an unknown method 3615903018 from 172.16.16.16:55488
DEBUG 2023-05-26 06:31:56,931 [shard 0] rpc - rpc_server.cc:176 - Received a request for an unknown method 3615903018 from 172.16.16.10:54508
DEBUG 2023-05-26 06:31:56,957 [shard 0] rpc - rpc_server.cc:176 - Received a request for an unknown method 3615903018 from 172.16.16.27:50604
DEBUG 2023-05-26 06:31:56,976 [shard 0] rpc - could not parse header from client: 172.16.16.21:56646
DEBUG 2023-05-26 06:31:56,976 [shard 0] rpc - could not parse header from client: 172.16.16.21:63766

[piyushredpanda]

this can be closed due to debug build wonkiness now, @mmaslankaprv ?

[michael-redpanda]

https://buildkite.com/redpanda/redpanda/builds/31404#0188c1f0-beae-4585-8d13-9c74b0115d81

[mmaslankaprv]

marking as sev/low as this is slow debug build issue

[dotnwat]

https://buildkite.com/redpanda/redpanda/builds/31964#0188f895-dca6-4916-a815-73974db564cd

[r-vasquez]

Getting this in v23.2.x branch https://buildkite.com/redpanda/redpanda/builds/42189#018c3039-27eb-4902-a25a-a791214c6c11/6-3827 but it seems that the fix it's in v23.2 branch.