CI Failure (Assertion _state && !_state->available()' failed) in RandomNodeOperationsTest.test_node_operations`

[andrwng]

https://buildkite.com/redpanda/redpanda/builds/23336#0186573d-03a1-4205-aaef-1f0b8f2f9bd4

Module: rptest.tests.random_node_operations_test
Class:  RandomNodeOperationsTest
Method: test_node_operations
Arguments:
{
  "enable_failures": true
}

test_id:    rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True
status:     FAIL
run time:   8 minutes 55.854 seconds


    <BadLogLines nodes=docker-rp-14(1) example="redpanda: /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/future.cc:85: void seastar::internal::promise_base::clear(): Assertion `_state && !_state->available()' failed.">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 476, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/utils/mode_checks.py", line 63, in f
    return func(*args, **kwargs)
  File "/root/tests/rptest/services/cluster.py", line 67, in wrapped
    self.redpanda.raise_on_bad_logs(allow_list=log_allow_list)
  File "/root/tests/rptest/services/redpanda.py", line 1763, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-14(1) example="redpanda: /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/future.cc:85: void seastar::internal::promise_base::clear(): Assertion `_state && !_state->available()' failed.">

[andrwng]

Checked out e314110

TRACE 2023-02-15 23:29:03,349 [shard 0] kvstore - kvstore.cc:198 - Apply op: update: key={bytes:35} value={{bytes=21, fragments=1}}
redpanda: /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/future.cc:85: void seastar::internal::promise_base::clear(): Assertion `_state && !_state->available()' failed.
Aborting on shard 0.
Backtrace:
  0x5b572c6
  0x5bbacc2
  /opt/redpanda_installs/ci/lib/libc.so.6+0x42abf
  /opt/redpanda_installs/ci/lib/libc.so.6+0x92e3b
  /opt/redpanda_installs/ci/lib/libc.so.6+0x42a15
  /opt/redpanda_installs/ci/lib/libc.so.6+0x2c82e
  /opt/redpanda_installs/ci/lib/libc.so.6+0x2c75a
  /opt/redpanda_installs/ci/lib/libc.so.6+0x3b595
  0x5aba364
  0x506a535
  0x50c74aa
  0x504ede2
  0x1e90b6a
  0x5b753ff
  0x5b790d7
  0x5b764a9
  0x5a9a4f1
  0x5a9860f
  0x1d9e7de
  0x5e8b5cd
  /opt/redpanda_installs/ci/lib/libc.so.6+0x2d58f
  /opt/redpanda_installs/ci/lib/libc.so.6+0x2d648
  0x1d98aa4

[Backtrace #0]
__shared_ptr_emplace<const std::__1::__fs::filesystem::path &, const std::__1::__fs::filesystem::path &> at /home/andrew/xfs/Repos/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__memory/shared_ptr.h:289
 (inlined by) std::__1::shared_ptr<std::__1::__fs::filesystem::filesystem_error::_Storage> std::__1::allocate_shared<std::__1::__fs::filesystem::filesystem_error::_Storage, std::__1::allocator<std::__1::__fs::filesystem::filesystem_error::_Storage>, std::__1::__fs::filesystem::path const&, std::__1::__fs::filesystem::path const&, void>(std::__1::allocator<std::_
_1::__fs::filesystem::filesystem_error::_Storage> const&, std::__1::__fs::filesystem::path const&, std::__1::__fs::filesystem::path const&) at /home/andrew/xfs/Repos/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__memory/shared_ptr.h:954
 (inlined by) std::__1::shared_ptr<std::__1::__fs::filesystem::filesystem_error::_Storage> std::__1::make_shared<std::__1::__fs::filesystem::filesystem_error::_Storage, std::__1::__fs::filesystem::path const&, std::__1::__fs::filesystem::path const&, void>(std::__1::__fs::filesystem::path const&, std::__1::__fs::filesystem::path const&) at /home/andrew/xfs/Repos
/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__memory/shared_ptr.h:963
 (inlined by) filesystem_error at /home/andrew/xfs/Repos/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__filesystem/filesystem_error.h:46
 (inlined by) seastar::syscall_result<int>::throw_fs_exception(seastar::basic_sstring<char, unsigned int, 15u, true> const&, std::__1::__fs::filesystem::path const&, std::__1::__fs::filesystem::path const&) const at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/syscall_result.hh:44
 (inlined by) seastar::syscall_result<int>::throw_fs_exception_if_error(seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&) const at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/
core/syscall_result.hh:55
 (inlined by) operator() at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1791
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::reactor::link_file(std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_10::operator()() const::{lambda(seastar::syscall_result<int>)#1}, seastar::syscall_result<int> >(seastar::reactor::li
nk_file(std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_10::operator()() const::{lambda(seastar::syscall_result<int>)#1}&&, seastar::syscall_result<int>&&) at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future
.hh:2149
 (inlined by) seastar::future<void> seastar::future<seastar::syscall_result<int> >::then_impl<seastar::reactor::link_file(std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_10::operator()() const::{lambda(seastar::syscall_result<int>)#1}, seastar::future<void> >(seastar::reactor::l
ink_file(std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_10::operator()() const::{lambda(seastar::syscall_result<int>)#1}&&) at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1615
 (inlined by) seastar::internal::future_result<seastar::reactor::link_file(std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_10::operator()() const::{lambda(seastar::syscall_result<int>)#1}, seastar::syscall_result<int> >::future_type seastar::internal::call_then_impl<seastar::fut
ure<seastar::syscall_result<int> > >::run<seastar::reactor::link_file(std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_10::operator()() const::{lambda(seastar::syscall_result<int>)#1}>(seastar::future<seastar::syscall_result<int> >&, seastar::reactor::link_file(std::__1::basic_st
ring_view<char, std::__1::char_traits<char> >, std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_10::operator()() const::{lambda(seastar::syscall_result<int>)#1}&&) at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/core/future.hh:1248
... lots of futures
(inlined by) operator() at /home/andrew/xfs/Repos/redpanda/src/v/storage/disk_log_impl.cc:154
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}, seastar::lw_shared_ptr<storage::segment>&>(storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}&&, seast
ar::lw_shared_ptr<storage::segment>&) at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:2149
 (inlined by) auto seastar::futurize_invoke<storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}, seastar::lw_shared_ptr<storage::segment>&>(storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}&&, seastar::lw_shared_ptr<storage::segment>&) at /
home/andrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:2180
 (inlined by) seastar::future<void> seastar::parallel_for_each<seastar::circular_buffer<seastar::lw_shared_ptr<storage::segment>, std::__1::allocator<seastar::lw_shared_ptr<storage::segment> > >::cbiterator<seastar::circular_buffer<seastar::lw_shared_ptr<storage::segment>, std::__1::allocator<seastar::lw_shared_ptr<storage::segment> > >, seastar::lw_shared_ptr<s
torage::segment> >, seastar::circular_buffer<seastar::lw_shared_ptr<storage::segment>, std::__1::allocator<seastar::lw_shared_ptr<storage::segment> > >::cbiterator<seastar::circular_buffer<seastar::lw_shared_ptr<storage::segment>, std::__1::allocator<seastar::lw_shared_ptr<storage::segment> > >, seastar::lw_shared_ptr<storage::segment> >, storage::disk_log_impl:
:close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}>(seastar::circular_buffer<seastar::lw_shared_ptr<storage::segment>, std::__1::allocator<seastar::lw_shared_ptr<storage::segment> > >::cbiterator<seastar::circular_buffer<seastar::lw_shared_ptr<storage::segment>, std::__1::allocator<seastar::lw_shared_ptr<storage::segment> >
 >, seastar::lw_shared_ptr<storage::segment> >, seastar::circular_buffer<seastar::lw_shared_ptr<storage::segment>, std::__1::allocator<seastar::lw_shared_ptr<storage::segment> > >::cbiterator<seastar::circular_buffer<seastar::lw_shared_ptr<storage::segment>, std::__1::allocator<seastar::lw_shared_ptr<storage::segment> > >, seastar::lw_shared_ptr<storage::segment
> >, storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}&&) at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/loop.hh:566
 (inlined by) seastar::future<void> seastar::internal::parallel_for_each_impl<storage::segment_set&, storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}>(storage::segment_set&, storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}&&) at /home/a
ndrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/loop.hh:616
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::future<void> (*&)(storage::segment_set&, storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}&&), storage::segment_set&, storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_pt
r<storage::segment>&)#1}>(seastar::future<void> (*&)(storage::segment_set&, storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}&&), storage::segment_set&, storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}&&) at /home/andrew/xfs/Repos/redpan
da/vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:2149
 (inlined by) auto seastar::futurize_invoke<seastar::future<void> (*&)(storage::segment_set&, storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}&&), storage::segment_set&, storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}>(seastar::future<
void> (*&)(storage::segment_set&, storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}&&), storage::segment_set&, storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}&&) at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/in
clude/seastar/core/future.hh:2180
 (inlined by) seastar::future<void> seastar::parallel_for_each<storage::segment_set&, storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}>(storage::segment_set&, storage::disk_log_impl::close()::$_19::operator()() const::{lambda(seastar::lw_shared_ptr<storage::segment>&)#1}&&) at /home/andrew/xfs/Repos
/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/loop.hh:631
 (inlined by) operator() at /home/andrew/xfs/Repos/redpanda/src/v/storage/disk_log_impl.cc:152
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<storage::disk_log_impl::close()::$_19>(storage::disk_log_impl::close()::$_19&&) at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:2149
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<storage::disk_log_impl::close()::$_19>(storage::disk_log_impl::close()::$_19&&, seastar::internal::monostate) at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:1993
 (inlined by) seastar::future<void> seastar::future<void>::then_impl<storage::disk_log_impl::close()::$_19, seastar::future<void> >(storage::disk_log_impl::close()::$_19&&) at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:1615
 (inlined by) seastar::internal::future_result<storage::disk_log_impl::close()::$_19, void>::future_type seastar::internal::call_then_impl<seastar::future<void> >::run<storage::disk_log_impl::close()::$_19>(seastar::future<void>&, storage::disk_log_impl::close()::$_19&&) at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core
/future.hh:1248
 (inlined by) seastar::future<void> seastar::future<void>::then<storage::disk_log_impl::close()::$_19, seastar::future<void> >(storage::disk_log_impl::close()::$_19&&) at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:1534
 (inlined by) storage::disk_log_impl::close() at /home/andrew/xfs/Repos/redpanda/src/v/storage/disk_log_impl.cc:151
std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> > std::__1::__pad_and_output<char, std::__1::char_traits<char> >(std::__1::ostreambuf_iterator<char, std::__1::char_traits<char> >, char const*, char const*, char const*, std::__1::ios_base&, char) at /home/andrew/xfs/Repos/redpanda/vbuild/llvm/install/bin/../include/c++/v1/locale:1411
 (inlined by) std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::__put_character_sequence<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long) at /home/andrew/xfs/Repos/redpanda/vbuild/llvm/install/bin/../include/c++/v1/ostream:726
 (inlined by) std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::operator<< <std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*) at /home/andrew/xfs/Repos/redpanda/vbuild/llvm/install/bin/../include/c++/v1/ostream:865
 (inlined by) detail::operator<<(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, detail::base_named_type<long, model::model_offset_type, std::__1::integral_constant<bool, true> > const&) at /home/andrew/xfs/Repos/redpanda/src/v/utils/named_type.h:118
 (inlined by) storage::operator<<(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, storage::disk_log_appender const&) at /home/andrew/xfs/Repos/redpanda/src/v/storage/disk_log_appender.cc:154
storage::log_manager::manage(storage::ntp_config) [clone .destroy] at log_manager.cc:?
 (inlined by) seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>::return_all() at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/semaphore.hh:469
 (inlined by) ~semaphore_units at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/semaphore.hh:449
 (inlined by) void std::__1::__destroy_at<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, 0>(seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at /home/andrew/xfs/Repos/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__memory/construct_at.h:56
 (inlined by) void std::__1::destroy_at<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>, 0>(seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock>*) at /home/andrew/xfs/Repos/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__memory/construct_at.h:81
 (inlined by) seastar::future_state<seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::__1::chrono::steady_clock> >::clear() at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:621
 (inlined by) ~future_state at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:628
 (inlined by) ~future at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/future.hh:1351
 (inlined by) ~awaiter at /home/andrew/xfs/Repos/redpanda/vbuild/release/clang/rp_deps_install/include/seastar/core/coroutine.hh:157
 (inlined by) storage::log_manager::manage(storage::ntp_config) at /home/andrew/xfs/Repos/redpanda/src/v/storage/log_manager.cc:306

[jcsp]

Seen again here:
https://buildkite.com/redpanda/redpanda/builds/23318#0186566f-04ca-4875-a4c9-0b9edbd1f087

I also notice this shortly before the crash:

WARN  2023-02-15 20:10:05,299 [shard 1] seastar - Exceptional future ignored: storage::segment_closed_exception (segment_closed exception), backtrace: 0x5def084 0x5ad1973 0x50857a7 0x504951c 0x504a980 0x1f8fd48 0x5b8bb6f 0x5b8f847 0x5bd2b95 0x5b2d93f /opt/redpanda_installs/ci/lib/libc.so.6+0x91016 /opt/redpanda_installs/ci/lib/libc.so.6+0x1166cf

[andijcr]

could it be an allocation failure?
the first backtrace is interesting, in trying to create a shared_ptr. this is the top of the backtrace

__shared_ptr_emplace<const std::__1::__fs::filesystem::path &, const std::__1::__fs::filesystem::path &> at /home/andrew/xfs/Repos/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__memory/shared_ptr.h:289 (inlined by) std::__1::shared_ptr<std::__1::__fs::filesystem::filesystem_error::_Storage> std::__1::allocate_shared<std::__1::__fs::filesystem::filesystem_error::_Storage, std::__1::allocator<std::__1::__fs::filesystem::filesystem_error::_Storage>, std::__1::__fs::filesystem::path const&, std::__1::__fs::filesystem::path const&, void>(std::__1::allocator<std::__1::__fs::filesystem::filesystem_error::_Storage> const&, std::__1::__fs::filesystem::path const&, std::__1::__fs::filesystem::path const&) at /home/andrew/xfs/Repos/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__memory/shared_ptr.h:954 (inlined by) std::__1::shared_ptr<std::__1::__fs::filesystem::filesystem_error::_Storage> std::__1::make_shared<std::__1::__fs::filesystem::filesystem_error::_Storage, std::__1::__fs::filesystem::path const&, std::__1::__fs::filesystem::path const&, void>(std::__1::__fs::filesystem::path const&, std::__1::__fs::filesystem::path const&) at /home/andrew/xfs/Repos
/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__memory/shared_ptr.h:963
 (inlined by) filesystem_error at /home/andrew/xfs/Repos/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__filesystem/filesystem_error.h:46

and this is the initiator in our code

(inlined by) operator() at /home/andrew/xfs/Repos/redpanda/src/v/storage/disk_log_impl.cc:154

funny thing is that this is the code @ line 154

             return h->close().handle_exception(
                [&errors, h](std::exception_ptr e) {
                    vlog(stlog.error, "Error closing segment:{} - {}", e, h);
                    errors = true;
                });

[VladLazar]

I had a look at https://buildkite.com/redpanda/redpanda/builds/23318#0186566f-04ca-4875-a4c9-0b9edbd1f087:

Looks like we have a zombie topic of sorts. We attempted to delete kafka/fuzzy-operator-1683-ktdcuh/0, but it doesn't look like this deletion propagated properly to the storage layer. Then, when redpanda tries to stop it trips up on this topic.

cluster - controller_backend.cc:941 - [{kafka/fuzzy-operator-1683-ktdcuh/0}] (retry 0) executing operation: {type: deletion, revision: 1673, assignment: { id: 0, group_id: 211, replicas: {{node_id: 7, shard: 1}, {node_id: 4, shard: 0}, {node_id: 6, shard: 0}} }, previous assignment: {nullopt}}     
controller_backend.cc:941 - [{kafka/fuzzy-operator-1683-ktdcuh/0}] (retry 0) executing operation: {type: deletion, revision: 1673, assignment: { id: 0, group_id: 211, replicas: {{node_id: 7, shard: 1}, {node_id: 4, shard: 0}, {node_id: 6, shard: 0}} }, previous assignment: {nullopt}}
controller_backend.cc:863 - [{kafka/fuzzy-operator-1683-ktdcuh/0}] (retry 0) finished operation: {type: deletion, revision: 1673, assignment: { id: 0, group_id: 211, replicas: {{node_id: 7, shard: 1}, {node_id: 4, shard: 0}, {node_id: 6, shard: 0}} }, previous assignment: {nullopt}} 
raft - [group_id:211, {kafka/fuzzy-operator-1683-ktdcuh/0}] consensus.cc:252 - Stopping

log_manager.cc:462 - Cleaning up ntp directory "/var/lib/redpanda/data/kafka/fuzzy-operator-1683-ktdcuh/0_1521"
log_manager.cc:513 - Trying to clean up topic directory "/var/lib/redpanda/data/kafka/fuzzy-operator-1683-ktdcuh"
cluster - controller_backend.cc:885 - [{kafka/fuzzy-operator-1683-ktdcuh/0}] exception while executing partition operation: {delta: {type: deletion, ntp: {kafka/fuzzy-operator-1683-ktdcuh/0}, offset: 1673, new_assignment: {         id: 0, group_id: 211, replicas: {{node_id: 7, shard: 1}, {node_id: 4, shard: 0}, {node_id: 6, shard: 0}} }, previous_replica_set: {nullopt}}, retries: 0} - std::__1::__fs::filesystem::filesystem_error (error system:2, filesystem error: open failed: No such file or         directory ["/var/lib/redpanda/data/kafka/fuzzy-operator-1683-ktdcuh/0_1521"])

cluster - controller_backend.cc:941 - [{kafka/fuzzy-operator-1683-ktdcuh/0}] (retry 1) executing operation: {type: deletion, revision: 1673, assignment: { id: 0, group_id: 211, replicas: {{node_id: 7, shard: 1}, {node_id: 4,shard: 0}, {node_id: 6, shard: 0}} }, previous assignment: {nullopt}}
cluster - controller_backend.cc:863 - [{kafka/fuzzy-operator-1683-ktdcuh/0}] (retry 1) finished operation: {type: deletion, revision: 1673, assignment: { id: 0, group_id: 211, replicas: {{node_id: 7, shard: 1}, {node_id: 4, shard: 0}, {node_id: 6, shard: 0}} }, previous assignment: {nullopt}}

state_machine.cc:160 - State machine for ntp={kafka/fuzzy-operator-1683-ktdcuh/0} caught exception seastar::broken_promise (broken promise)

storage - disk_log_impl.cc:132 - closing log {offsets: {start_offset:0, committed_offset:0, committed_offset_term:1, dirty_offset:0, dirty_offset_term:1, last_term_start_offset:0}, max_collectible_offset: -922337203685477580        8, is_closed: false, segments: [{size: 1, [{offset_tracker:{term:1, base_offset:0, committed_offset:0, dirty_offset:0}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/kafka/fuzzy-operator-1683-ktdcuh/0_1521/0-1-v1.l        og, (611 bytes)}, writer=nullptr, cache={cache_size=1}, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/fuzzy-operator-1683-ktdcuh/0_1521/0-1-v1.base_index, offsets:{0}, index:{header_bitflags:0, base_offset:{0}, max_offse        t:{0}, base_timestamp:{timestamp: 1676491731666}, max_timestamp:{timestamp: 1676491731666}, batch_timestamps_are_monotonic:1, index(0,0,0)}, step:32768, needs_persistence:0}}]}], config: {ntp: {kafka/fuzzy-operator-1683-ktdcuh/0}, base_dir: /var/lib/redpanda/data,         overrides: {compaction_strategy: {nullopt}, cleanup_policy_bitflags: {delete}, segment_size: {nullopt}, retention_bytes: {}, retention_time_ms: {}, recovery_enabled: false, retention_local_target_bytes: {}, retention_local_target_ms: {}, remote_delete: {true}, se        gment_ms: {}}, revision: 1521, initial_revision: 1521}}
seastar - Exceptional future ignored: storage::segment_closed_exception (segment_closed exception), backtrace: 0x5def084 0x5ad1973 0x50857a7 0x504951c 0x504a980 0x1f8fd48 0x5b8bb6f 0x5b8f847 0x5bd2b95 0x5b2d93f /opt/redpanda        _installs/ci/lib/libc.so.6+0x91016 /opt/redpanda_installs/ci/lib/libc.so.6+0x1166cf

segment.cc:89 - closing segment: {offset_tracker:{term:1, base_offset:0, committed_offset:0, dirty_offset:0}, compacted_segment=0, finished_self_compaction=0, generation={1}, reader={/var/lib/redpanda/data/kafka/fu        zzy-operator-1683-ktdcuh/0_1521/0-1-v1.log, (611 bytes)}, writer=nullptr, cache={cache_size=1}, compaction_index:nullopt, closed=1, tombstone=0, index={file:/var/lib/redpanda/data/kafka/fuzzy-operator-1683-ktdcuh/0_1521/0-1-v1.base_index, offsets:{0}, index:{heade        r_bitflags:0, base_offset:{0}, max_offset:{0}, base_timestamp:{timestamp: 1676491731666}, max_timestamp:{timestamp: 1676491731666}, batch_timestamps_are_monotonic:1, index(0,0,0)}, step:32768, needs_persistence:0}}
disk_log_impl.cc:170 - closed {kafka/fuzzy-operator-1683-ktdcuh/0}, last clean segment is /var/lib/redpanda/data/kafka/fuzzy-operator-1683-ktdcuh/0_1521/0-1-v1.log
log_manager.cc:177 - writing clean record for: {kafka/fuzzy-operator-1683-ktdcuh/0} /var/lib/redpanda/data/kafka/fuzzy-operator-1683-ktdcuh/0_1521/0-1-v1.log
kvstore - kvstore.cc:198 - Apply op: insert: key={bytes:45} value={{bytes=20, fragments=1}}

redpanda: /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/future.cc:85: void seastar::internal::promise_base::clear(): Assertion `_state && !_state->available()' failed.

[VladLazar]

Perhaps topic deletion is failing because it's somehow racing with orphan topic deletion.
@ZeDRoman could you take a look?

log_manager.cc:462 - Cleaning up ntp directory "/var/lib/redpanda/data/kafka/fuzzy-operator-1683-ktdcuh/0_1521"

cluster - controller_backend.cc:885 - [{kafka/fuzzy-operator-1683-ktdcuh/0}] exception while executing partition operation: {delta: {type: deletion, ntp: {kafka/fuzzy-operator-1683-ktdcuh/0}, offset: 1673, new_assignment: { id: 0, group_id: 211, replicas: {{node_id: 7, shard: 1}, {node_id: 4, shard: 0}, {node_id: 6, shard: 0}} }, previous_replica_set: {nullopt}}, retries: 0} - std::__1::__fs::filesystem::filesystem_error (error system:2, filesystem error: open failed: No such file or directory ["/var/lib/redpanda/data/kafka/fuzzy-operator-1683-ktdcuh/0_1521"])

[bharathv]

Moving to sev/medium for now as the problematic patch got reverted.