CI Failure (NodeCrash) in ShutdownTest.test_timely_shutdown_with_failures

[andijcr]

https://buildkite.com/redpanda/redpanda/builds/34603

Module: rptest.tests.timely_shutdown_test
Class: ShutdownTest
Method: test_timely_shutdown_with_failures

test_id:    ShutdownTest.test_timely_shutdown_with_failures
status:     FAIL
run time:   215.602 seconds

<NodeCrash docker-rp-2: Redpanda process unexpectedly stopped>
Traceback (most recent call last):
  File "/root/tests/rptest/services/cluster.py", line 82, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/timely_shutdown_test.py", line 105, in test_timely_shutdown_with_failures
    self.redpanda.restart_nodes(leader)
  File "/root/tests/rptest/services/redpanda.py", line 891, in restart_nodes
    list(
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/tests/rptest/services/redpanda.py", line 892, in <lambda>
    executor.map(lambda n: self.stop_node(n, timeout=stop_timeout),
  File "/root/tests/rptest/services/redpanda.py", line 3141, in stop_node
    wait_until(
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Redpanda node docker-rp-2 failed to stop in 30 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 103, in wrapped
    redpanda.raise_on_crash(log_allow_list=log_allow_list)
  File "/root/tests/rptest/services/redpanda.py", line 2838, in raise_on_crash
    raise NodeCrash(crashes)
rptest.services.utils.NodeCrash: <NodeCrash docker-rp-2: Redpanda process unexpectedly stopped>

docker-rp-2 shows a normal log, with no apparent failure, this might be a test debug timing issue

[rockwotj]

https://buildkite.com/redpanda/redpanda/builds/34889#0189e69e-34f8-4cf9-abc2-0631a0074333

[StephanDollberg]

https://buildkite.com/redpanda/redpanda/builds/34824
https://buildkite.com/redpanda/redpanda/builds/35050

[abhijat]

https://buildkite.com/redpanda/redpanda/builds/35861#018a4032-0e22-4c3f-9d87-43ccee26cb76

[rockwotj]

https://buildkite.com/redpanda/redpanda/builds/36455#018a674d-b930-45e5-b66f-8f871efff5dd

[dotnwat]

docker-rp-2 shows a normal log, with no apparent failure, this might be a test debug timing issue

yeh, agree. timing issue sounds right, or something else on the ducktape side--the logs look fine.

[piyushredpanda]

This has been happening a lot -- requesting @nvartolomei to take a look.

[nvartolomei]

Investigation and proposed fix #13739

[bharathv]

@nvartolomei can you please paste the crashing stack here for future reference (pattern matching existing/fixed issues etc).

[nvartolomei]

@bharathv there is no crash observed. The only externally observed behavior what I see is that the system didn't shut down "timely".