Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI Failure (Crash: Failed to allocate) in MemoryStressTest.test_fetch_with_many_partitions #11304

Closed
rystsov opened this issue Jun 8, 2023 · 10 comments
Closed

CI Failure (Crash: Failed to allocate) in MemoryStressTest.test_fetch_with_many_partitions #11304

rystsov opened this issue Jun 8, 2023 · 10 comments
Assignees
@dlex
Labels
ci-failure kind/bug Something isn't working sev/high loss of availability, pathological performance degradation, recoverable corruption

Comments

@rystsov
Copy link
Contributor

rystsov commented Jun 8, 2023

https://buildkite.com/redpanda/redpanda/builds/30854#01889a79-3d42-4b02-8500-00ea6604d469

Module: rptest.tests.memory_stress_test
Class:  MemoryStressTest
Method: test_fetch_with_many_partitions
Arguments:
{
  "memory_share_for_fetch": 0.8
}

test_id:    rptest.tests.memory_stress_test.MemoryStressTest.test_fetch_with_many_partitions.memory_share_for_fetch=0.8
status:     FAIL
run time:   2 minutes 2.985 seconds


    <BadLogLines nodes=docker-rp-7(2) example="ERROR 2023-06-08 10:30:58,132 [shard 0] seastar_memory - Dumping seastar memory diagnostics">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 139, in wrapped
    redpanda.raise_on_bad_logs(
  File "/root/tests/rptest/services/redpanda.py", line 2673, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-7(2) example="ERROR 2023-06-08 10:30:58,132 [shard 0] seastar_memory - Dumping seastar memory diagnostics">
@rystsov rystsov added kind/bug Something isn't working ci-failure labels Jun 8, 2023
@rystsov rystsov mentioned this issue Jun 8, 2023
Merged
7 tasks
@rystsov
Copy link
Contributor Author

rystsov commented Jun 8, 2023 

ERROR 2023-06-08 10:30:58,133 [shard 0] seastar - Failed to allocate 131072 bytes
Aborting on shard 0.
Backtrace:
  0x6de0583
  0x6e334db
  /opt/redpanda_installs/ci/lib/libc.so.6+0x42abf
  /opt/redpanda_installs/ci/lib/libc.so.6+0x92e3b
  /opt/redpanda_installs/ci/lib/libc.so.6+0x42a15
  /opt/redpanda_installs/ci/lib/libc.so.6+0x2c82e
  0x6d0c261
  0x6d1aae5
  0x6bb7999
  0x6bb8608
  0x6dd8c0f
  0x6ddc127
  0x6dd9476
  0x6cdbb40
  0x6cd9f38
  0x22273c5
  0x70d4bdb
  /opt/redpanda_installs/ci/lib/libc.so.6+0x2d58f
  /opt/redpanda_installs/ci/lib/libc.so.6+0x2d648
  0x2220f24

@rystsov rystsov changed the title CI Failure (Dumping seastar memory diagnostics) in MemoryStressTest.test_fetch_with_many_partitions CI Failure (Crash: Failed to allocate) in MemoryStressTest.test_fetch_with_many_partitions Jun 8, 2023
@andijcr
Copy link
Contributor

andijcr commented Jun 9, 2023

https://buildkite.com/redpanda/redpanda/builds/30951#01889fbf-4d78-4127-8af1-cf2fc3ab02a5

ower_bound: 3
TRACE 2023-06-09 11:06:32,208 [shard 0] storage - readers_cache.cc:328 - {kafka/topic-mlpnhzbysi/363} - removing reader: [0,2] lower_bound: 3
TRACE 2023-06-09 11:06:32,208 [shard 0] storage - segment_reader.cc:153 - ::put segment file /var/lib/redpanda/data/kafka/topic-mlpnhzbysi/60_12/0-1-v1.log, refcount=1
DEBUG 2023-06-09 11:06:32,208 [shard 0] storage - segment_reader.cc:157 - Closing segment file /var/lib/redpanda/data/kafka/topic-mlpnhzbysi/60_12/0-1-v1.log
TRACE 2023-06-09 11:06:32,209 [shard 0] storage - readers_cache.cc:328 - {kafka/topic-mlpnhzbysi/368} - removing reader: [0,2] lower_bound: 3
TRACE 2023-06-09 11:06:32,209 [shard 0] storage - readers_cache.cc:328 - {kafka/topic-mlpnhzbysi/150} - removing reader: [0,2] lower_bound: 3
TRACE 2023-06-09 11:06:32,210 [shard 0] storage - readers_cache.cc:328 - {kafka/topic-mlpnhzbysi/318} - removing reader: [0,2] lower_bound: 3
TRACE 2023-06-09 11:06:32,210 [shard 0] storage - readers_cache.cc:328 - {kafka/topic-mlpnhzbysi/250} - removing reader: [0,2] lower_bound: 3
DEBUG 2023-06-09 11:06:32,214 [shard 0] seastar_memory - Failed to allocate 131072 bytes at 0x708ed83 0x6d0c0fd 0x6d1ab25 0x6bb26ae 0x5faddb9 0x5fadc0c 0x5fb0620 0x5ec0d61 0x5eb4b14 0x5ea3c81 0x6dd8c4f 0x6ddc167 0x6dd94b6 0x6cdbb80 0x6cd9f78 0x22273c5 0x70d4c1b /opt/redpanda_installs/ci/lib/libc.so.6+0x2d58f /opt/redpanda_installs/ci/lib/libc.so.6+0x2d648 0x2220f24
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>, storage::continuous_batch_parser::consume_records()::$_0, seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>> seastar::future<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void>>>::then_impl_nrvo<storage::continuous_batch_parser::consume_records()::$_0, seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>>(storage::continuous_batch_parser::consume_records()::$_0&&)::'lambda'(seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>&&, storage::continuous_batch_parser::consume_records()::$_0&, seastar::future_state<boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void>>>&&), boost::outcome_v2::basic_result<iobuf, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<iobuf, std::__1::error_code, void>>>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>, storage::continuous_batch_parser::consume_one()::$_0::operator()(boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>) const::'lambda'(boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>), seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>> seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>::then_impl_nrvo<storage::continuous_batch_parser::consume_one()::$_0::operator()(boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>) const::'lambda'(boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>), seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>>(storage::continuous_batch_parser::consume_one()::$_0::operator()(boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>) const::'lambda'(boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>)&&)::'lambda'(seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>&&, storage::continuous_batch_parser::consume_one()::$_0::operator()(boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>) const::'lambda'(boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>)&, seastar::future_state<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>&&), boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag>>, storage::continuous_batch_parser::consume()::$_1::operator()() const::'lambda'(boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>), seastar::future<seastar::bool_class<seastar::stop_iteration_tag>> seastar::future<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>::then_impl_nrvo<storage::continuous_batch_parser::consume()::$_1::operator()() const::'lambda'(boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>), seastar::future<seastar::bool_class<seastar::stop_iteration_tag>>>(storage::continuous_batch_parser::consume()::$_1::operator()() const::'lambda'(boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>)&&)::'lambda'(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag>>&&, storage::continuous_batch_parser::consume()::$_1::operator()() const::'lambda'(boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>)&, seastar::future_state<boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>&&), boost::outcome_v2::basic_result<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::bool_class<storage::stop_parser_tag>, std::__1::error_code, void>>>
   --------
   seastar::internal::repeater<storage::continuous_batch_parser::consume()::$_1>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<unsigned long, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<unsigned long, std::__1::error_code, void>>>, storage::continuous_batch_parser::consume()::$_2, seastar::future<boost::outcome_v2::basic_result<unsigned long, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<unsigned long, std::__1::error_code, void>>> seastar::future<void>::then_impl_nrvo<storage::continuous_batch_parser::consume()::$_2, seastar::future<boost::outcome_v2::basic_result<unsigned long, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<unsigned long, std::__1::error_code, void>>>>(storage::continuous_batch_parser::consume()::$_2&&)::'lambda'(seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<unsigned long, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<unsigned long, std::__1::error_code, void>>>&&, storage::continuous_batch_parser::consume()::$_2&, seastar::future_state<seastar::internal::monostate>&&), void>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>, storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_0, seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>> seastar::future<boost::outcome_v2::basic_result<unsigned long, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<unsigned long, std::__1::error_code, void>>>::then_impl_nrvo<storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_0, seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>>(storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_0&&)::'lambda'(seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>&&, storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_0&, seastar::future_state<boost::outcome_v2::basic_result<unsigned long, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<unsigned long, std::__1::error_code, void>>>&&), boost::outcome_v2::basic_result<unsigned long, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<unsigned long, std::__1::error_code, void>>>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>, seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>> seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>::handle_exception_type<storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_1>(storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_1&&)::'lambda'(storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_1&&), seastar::futurize<storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_1>::type seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>::then_wrapped_nrvo<seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>, seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>> seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>::handle_exception_type<storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_1>(storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_1&&)::'lambda'(storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_1&&)>(seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>> seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>::handle_exception_type<storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_1>(storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_1&&)::'lambda'(storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_1&&)&&)::'lambda'(seastar::internal::promise_base_with_type<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>&&, seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>> seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>::handle_exception_type<storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_1>(storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_1&&)::'lambda'(storage::log_segment_batch_reader::read_some(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_1&&)&, seastar::future_state<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>&&), boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>
   --------
   seastar::internal::coroutine_traits_base<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>::promise_type
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>, storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_4, seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>> seastar::future<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>::then_impl_nrvo<storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_4, seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>>(storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_4&&)::'lambda'(seastar::internal::promise_base_with_type<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>&&, storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_4&, seastar::future_state<boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>&&), boost::outcome_v2::basic_result<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, std::__1::error_code, void>>>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>, seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>> seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>::handle_exception<storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_5>(storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_5&&)::'lambda'(storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_5&&), seastar::futurize<storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_5>::type seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>::then_wrapped_nrvo<seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>, seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>> seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>::handle_exception<storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_5>(storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_5&&)::'lambda'(storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_5&&)>(seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>> seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>::handle_exception<storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_5>(storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_5&&)::'lambda'(storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_5&&)&&)::'lambda'(seastar::internal::promise_base_with_type<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>&&, seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>> seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>::handle_exception<storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_5>(storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_5&&)::'lambda'(storage::log_reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::$_5&&)&, seastar::future_state<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>&&), std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>, kafka::replicated_partition::make_reader(storage::log_reader_config, std::__1::optional<std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>>)::reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>), seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>> seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>::then_impl_nrvo<kafka::replicated_partition::make_reader(storage::log_reader_config, std::__1::optional<std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>>)::reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>), seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>>(kafka::replicated_partition::make_reader(storage::log_reader_config, std::__1::optional<std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>>)::reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>)&&)::'lambda'(seastar::internal::promise_base_with_type<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>&&, kafka::replicated_partition::make_reader(storage::log_reader_config, std::__1::optional<std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>>)::reader::do_load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>)&, seastar::future_state<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>&&), std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<void>, model::record_batch_reader::impl::load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>), seastar::future<void> seastar::future<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>::then_impl_nrvo<model::record_batch_reader::impl::load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>), seastar::future<void>>(model::record_batch_reader::impl::load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>)&&)::'lambda'(seastar::internal::promise_base_with_type<void>&&, model::record_batch_reader::impl::load_slice(std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>)&, seastar::future_state<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>&&), std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch>>, model::record_batch_reader::foreign_data_t>>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag>>, auto model::record_batch_reader::impl::do_action<kafka::kafka_batch_serializer, auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&)>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&)&&)::'lambda'()::operator()() const::'lambda'(), auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&) seastar::future<void>::then_impl_nrvo<auto model::record_batch_reader::impl::do_action<kafka::kafka_batch_serializer, auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&)>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&)&&)::'lambda'()::operator()() const::'lambda'(), seastar::future<seastar::bool_class<seastar::stop_iteration_tag>>>(kafka::kafka_batch_serializer&&)::'lambda'(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag>>&&, auto model::record_batch_reader::impl::do_action<kafka::kafka_batch_serializer, auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&)>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&)&&)::'lambda'()::operator()() const::'lambda'()&, seastar::future_state<seastar::internal::monostate>&&), void>
   --------
   seastar::internal::repeater<auto model::record_batch_reader::impl::do_action<kafka::kafka_batch_serializer, auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&)>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&)&&)::'lambda'()>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<kafka::kafka_batch_serializer::result>, auto model::record_batch_reader::impl::do_action<kafka::kafka_batch_serializer, auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&)>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&)&&)::'lambda0'(), auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&) seastar::future<void>::then_impl_nrvo<auto model::record_batch_reader::impl::do_action<kafka::kafka_batch_serializer, auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&)>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&)&&)::'lambda0'(), seastar::future<kafka::kafka_batch_serializer::result>>(kafka::kafka_batch_serializer&&)::'lambda'(seastar::internal::promise_base_with_type<kafka::kafka_batch_serializer::result>&&, auto model::record_batch_reader::impl::do_action<kafka::kafka_batch_serializer, auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&)>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, auto model::record_batch_reader::impl::do_consume<kafka::kafka_batch_serializer>(kafka::kafka_batch_serializer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>)::'lambda'(kafka::kafka_batch_serializer&)&&)::'lambda0'()&, seastar::future_state<seastar::internal::monostate>&&), void>
   --------
   seastar::internal::do_with_state<std::__1::tuple<kafka::kafka_batch_serializer>, seastar::future<kafka::kafka_batch_serializer::result>>
ERROR 2023-06-09 11:06:32,214 [shard 0] seastar_memory - Dumping seastar memory diagnostics
Used memory:   509M
Free memory:   3M
Total memory:  512M
Hard failures: 1

Small pools:
objsz spansz usedobj memory unused wst%
    8     4K      1k    16K     5K   33
   10     4K       1     8K     8K   99
   12     4K      12    12K    12K   98
   14     4K       2     8K     8K   99
   16     4K      6k    96K     6K    5
   32     4K    142k     4M    23K    0
   32     4K      9k   296K    16K    5
   32     4K     10k   304K     3K    0
   32     4K      8k   332K    92K   27
   48     4K      9k   412K     5K    1
   48     4K      6k   408K   108K   26
   64     4K     18k     1M   134K   10
   64     4K    389k    24M    11K    0
   80     4K     31k     2M    78K    3
   96     4K      1k   292K   162K   55
  112     4K      2k   284K    86K   30
  128     4K     125    40K    24K   60
  160     4K     920   192K    48K   25
  192     4K     25k     5M   143K    2
  224     4K      3k   748K    35K    4
  256     4K     576   176K    32K   18
  320     8K     738   248K    17K    6
  384     8K     468   376K   200K   53
  448     4K     967   508K    85K   16
  512     4K      93    96K    50K   51
  640    16K      12   128K   120K   93
  768    16K      2k     2M   309K   16
  896     8K      2k     2M    96K    5
 1024     4K      54   184K   130K   70
 1280    32K     522   768K   115K   14
 1536    32K     432   768K   120K   15
 1792    16K       8   144K   130K   89
 2048     8K      67   152K    18K   11
 2560    64K     629     2M   155K    8
 3072    64K     411     1M    45K    3
 3584    32K       8   288K   259K   89
 4096    16K     752     3M    32K    1
 5120   128K       8   384K   340K   88
 6144   128K       1   512K   504K   98
 7168    64K      23   576K   413K   71
 8192    32K      1k    11M     1M   10
10240    64K      10     1M   920K   89
12288    64K       7   960K   876K   91
14336   128K       2     1M     1M   97
16384    64K      3k    52M     1M    2

Page spans:
index  size  free  used spans
    0    4K   40K   41M   10k
    1    8K  160K    2M   324
    2   16K  272K    5M   340
    3   32K    1M   17M   573
    4   64K    1M   58M   944
    5  128K    0B  322M    3k
    6  256K    0B  768K     3
    7  512K    0B    0B     0
    8    1M    0B    0B     0
    9    2M    0B    0B     0
   10    4M    0B    8M     2
   11    8M    0B    8M     1
   12   16M    0B   48M     3
   13   32M    0B    0B     0
   14   64M    0B    0B     0
   15  128M    0B    0B     0
   16  256M    0B    0B     0
   17  512M    0B    0B     0
   18    1G    0B    0B     0
   19    2G    0B    0B     0
   20    4G    0B    0B     0
   21    8G    0B    0B     0
   22   16G    0B    0B     0
   23   32G    0B    0B     0
   24   64G    0B    0B     0
   25  128G    0B    0B     0
   26  256G    0B    0B     0
   27  512G    0B    0B     0
   28    1T    0B    0B     0
   29    2T    0B    0B     0
   30    4T    0B    0B     0
   31    8T    0B    0B     0

ERROR 2023-06-09 11:06:32,215 [shard 0] seastar - Failed to allocate 131072 bytes
Aborting on shard 0.
Backtrace:
  0x6de05c3
  0x6e3351b
  /opt/redpanda_installs/ci/lib/libc.so.6+0x42abf
  /opt/redpanda_installs/ci/lib/libc.so.6+0x92e3b
  /opt/redpanda_installs/ci/lib/libc.so.6+0x42a15
  /opt/redpanda_installs/ci/lib/libc.so.6+0x2c82e
  0x6d0c2a1
  0x6d1ab25
  0x6bb26ae
  0x5faddb9
  0x5fadc0c
  0x5fb0620
  0x5ec0d61
  0x5eb4b14
  0x5ea3c81
  0x6dd8c4f
  0x6ddc167
  0x6dd94b6
  0x6cdbb80
  0x6cd9f78
  0x22273c5
  0x70d4c1b
  /opt/redpanda_installs/ci/lib/libc.so.6+0x2d58f
  /opt/redpanda_installs/ci/lib/libc.so.6+0x2d648
  0x2220f24

@michael-redpanda
Copy link
Contributor

https://buildkite.com/redpanda/redpanda/builds/31060#0188ae40-ef78-40d9-ad82-ad1240b54cc6
https://buildkite.com/redpanda/redpanda/builds/31062#0188ae47-aece-4a56-919f-c9cec2f865c8
https://buildkite.com/redpanda/redpanda/builds/31012#0188a3f3-bc97-49b7-b008-ecc7cec55500
https://buildkite.com/redpanda/redpanda/builds/31018#0188a406-8be0-4850-92e8-022281259762

@VladLazar VladLazar mentioned this issue Jun 13, 2023
Merged
7 tasks
@ztlpn
Copy link
Contributor

ztlpn commented Jun 14, 2023

https://buildkite.com/redpanda/redpanda/builds/31161#0188b380-d536-4158-a61e-1f14bcebec2e

@dotnwat dotnwat mentioned this issue Jun 14, 2023
Merged
7 tasks
@michael-redpanda
Copy link
Contributor

https://buildkite.com/redpanda/redpanda/builds/31293#0188baa3-5c8f-4b20-a345-89db13e9a23f

@ztlpn ztlpn mentioned this issue Jun 15, 2023
Merged
7 tasks
@michael-redpanda michael-redpanda added the sev/high loss of availability, pathological performance degradation, recoverable corruption label Jun 16, 2023
@michael-redpanda michael-redpanda self-assigned this Jun 16, 2023
@michael-redpanda
Copy link
Contributor

Easily reproducible in ducktape

@rockwotj rockwotj mentioned this issue Jun 16, 2023
Merged
7 tasks
@ztlpn
Copy link
Contributor

ztlpn commented Jun 16, 2023

https://buildkite.com/redpanda/redpanda/builds/31436#0188c3ad-93ee-4e13-8074-3698fbe0c4b2

@piyushredpanda
Copy link
Contributor

I wonder if #11458 is different?

@BenPope BenPope mentioned this issue Jun 20, 2023
Closed
7 tasks
@michael-redpanda michael-redpanda mentioned this issue Jun 20, 2023
Closed
@michael-redpanda
Copy link
Contributor

Work for #11458 should be done under this issue

@r-vasquez r-vasquez mentioned this issue Jun 20, 2023
Merged
1 task
@dlex
Copy link
Contributor

dlex commented Jun 20, 2023

The work has already started in #11458, so closing this one as a duplicate.

@dlex dlex closed this as completed Jun 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dlex dlex

Labels
ci-failure kind/bug Something isn't working sev/high loss of availability, pathological performance degradation, recoverable corruption
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@andijcr @rystsov @ztlpn @dlex @piyushredpanda @michael-redpanda