Flaky test: tpch test on cluster failed for reading a body from connection: broken pipe #7986

Xuanwo · 2022-09-30T06:42:15Z

Summary

--- /workspace/tests/suites/0_stateless/13_tpch/13_0007_q7.result        2022-09-30 05:37:31.856084133 +0000
+++ /workspace/tests/suites/0_stateless/13_tpch/13_0007_q7.stdout        2022-09-30 05:38:54.725091643 +0000
@@ -1,4 +1 @@
-FRANCE        GERMANY        1995        4637235.15
-FRANCE        GERMANY        1996        5224779.573
-GERMANY        FRANCE        1995        6232818.703
-GERMANY        FRANCE        1996        5557312.112
+ERROR 1105 (HY000) at line 2: Code: 1067, displayText = error reading a body from connection: broken pipe (while in processor thread 0).

https://github.com/datafuselabs/databend/actions/runs/3156329493/jobs/5136357403

Xuanwo · 2022-10-24T06:16:31Z

@zhang2014 This query seems stable in recent CI, maybe it has been resolved?

xudong963 · 2022-10-24T06:18:53Z

@zhang2014 This query seems stable in recent CI, maybe it has been resolved?

A recent case from @BohuTANG #8398 (comment)

Xuanwo · 2022-10-25T01:21:18Z

--- /workspace/tests/suites/0_stateless/13_tpch/13_0003_q3.result	2022-10-25 01:10:26.021571701 +0000
+++ /workspace/tests/suites/0_stateless/13_tpch/13_0003_q3.stdout	2022-10-25 01:12:40.323415686 +0000
@@ -1,5 +1 @@
-223140	355369.0698	1995-03-14	0
-584291	354494.7318	1995-02-21	0
-405063	353125.4577	1995-03-03	0
-573861	351238.27699999994	1995-03-09	0
-554757	349181.7426	1995-03-14	0
+ERROR 1105 (HY000) at line 2: Code: 1067, displayText = error reading a body from connection: broken pipe, source: Some(hyper::Error(Body, Error { kind: Io(Kind(BrokenPipe)) })).

This error is from tonic::Status, the connection between the query and meta's grpc service.

Maybe the conn for grpc has been closed unexpectedly?

Cc @zhang2014

zhang2014 · 2022-10-25T03:28:03Z

--- /workspace/tests/suites/0_stateless/13_tpch/13_0003_q3.result	2022-10-25 01:10:26.021571701 +0000
+++ /workspace/tests/suites/0_stateless/13_tpch/13_0003_q3.stdout	2022-10-25 01:12:40.323415686 +0000
@@ -1,5 +1 @@
-223140	355369.0698	1995-03-14	0
-584291	354494.7318	1995-02-21	0
-405063	353125.4577	1995-03-03	0
-573861	351238.27699999994	1995-03-09	0
-554757	349181.7426	1995-03-14	0
+ERROR 1105 (HY000) at line 2: Code: 1067, displayText = error reading a body from connection: broken pipe, source: Some(hyper::Error(Body, Error { kind: Io(Kind(BrokenPipe)) })).

This error is from tonic::Status, the connection between the query and meta's grpc service.

Maybe the conn for grpc has been closed unexpectedly?

Cc @zhang2014

Let's distinguish the meta rpc or flight rpc(add some message).

xudong963 · 2022-10-25T08:05:21Z

#8445

xudong963 · 2022-10-26T14:27:16Z

#8477 some stable cases

Xuanwo · 2022-11-07T18:08:03Z

The error happened while:

2022-11-07T17:41:10.513426Z ERROR databend_query::servers::mysql::writers::query_result_writer: result row write failed: Code: 1067, displayText = error reading a body from connection: connection reset, source: Some(hyper::Error(Body, Error { kind: Io(Kind(ConnectionReset)) })).

Xuanwo · 2022-11-08T02:39:19Z

Finally, we got a backtrace of this error (https://github.com/datafuselabs/databend/actions/runs/3415714548/jobs/5685251065):

2022-11-08T02:13:56.936274Z ERROR databend_query::api::rpc::packets::packet_data: src/query/service/src/api/rpc/packets/packet_data.rs:98: Got error code data packet: Code: 1067, displayText = error reading a body from connection: connection reset, source: Some(hyper::Error(Body, Error { kind: Io(Kind(ConnectionReset)) })).

   0: common_exception::exception_code::<impl common_exception::exception::ErrorCode>::UnknownException
             at ./src/common/exception/src/exception_code.rs:38:71
   1: common_exception::exception_into::<impl core::convert::From<tonic::status::Status> for common_exception::exception::ErrorCode>::from
             at ./src/common/exception/src/exception_into.rs:234:28
   2: {async_fn#0}
             at ./src/query/service/src/api/rpc/flight_client.rs:374:36
   3: poll<databend_query::api::rpc::flight_client::{impl#9}::recv::{async_fn_env#0}>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/core/src/future/mod.rs:91:19
   4: {async_fn#0}
             at ./src/query/service/src/api/rpc/flight_client.rs:192:64
   5: poll<databend_query::api::rpc::flight_client::{impl#2}::recv::{async_fn_env#0}>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/core/src/future/mod.rs:91:19
   6: {async_block#0}
             at ./src/query/service/src/api/rpc/exchange/exchange_transform.rs:283:66
   7: poll<databend_query::api::rpc::exchange::exchange_transform::{impl#2}::async_process::{async_block_env#0}>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/core/src/future/mod.rs:91:19
   8: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<(), common_exception::exception::ErrorCode>> + core::marker::Send), alloc::alloc::Global>>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/core/src/future/future.rs:124:9
   9: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<(), common_exception::exception::ErrorCode>> + core::marker::Send), alloc::alloc::Global>>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/core/src/future/future.rs:124:9
  10: poll<alloc::boxed::Box<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<(), common_exception::exception::ErrorCode>> + core::marker::Send), alloc::alloc::Global>>, alloc::alloc::Global>>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/core/src/future/future.rs:124:9
  11: poll_unpin<core::pin::Pin<alloc::boxed::Box<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<(), common_exception::exception::ErrorCode>> + core::marker::Send), alloc::alloc::Global>>, alloc::alloc::Global>>>
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/futures-util-0.3.25/src/future/future/mod.rs:562:9
  12: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/futures-util-0.3.25/src/future/select.rs:104:35
  13: {async_block#0}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<(), common_exception::exception::ErrorCode>> + core::marker::Send), alloc::alloc::Global>>>
             at ./src/query/service/src/pipelines/executor/processor_async_task.rs:56:55
  14: poll<databend_query::pipelines::executor::processor_async_task::{impl#0}::create::{async_block_env#0}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<(), common_exception::exception::ErrorCode>> + core::marker::Send), alloc::alloc::Global>>>>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/core/src/future/mod.rs:91:19
  15: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<(), common_exception::exception::ErrorCode>> + core::marker::Send), alloc::alloc::Global>>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/core/src/future/future.rs:124:9
  16: poll_unpin<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<(), common_exception::exception::ErrorCode>> + core::marker::Send), alloc::alloc::Global>>>
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/futures-util-0.3.25/src/future/future/mod.rs:562:9
  17: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/futures-util-0.3.25/src/future/select.rs:108:35
  18: {async_block#1}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<(), common_exception::exception::ErrorCode>> + core::marker::Send), alloc::alloc::Global>>>
             at ./src/query/service/src/pipelines/executor/processor_async_task.rs:72:67
  19: poll<databend_query::pipelines::executor::processor_async_task::{impl#0}::create::{async_block_env#1}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<(), common_exception::exception::ErrorCode>> + core::marker::Send), alloc::alloc::Global>>>>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/core/src/future/mod.rs:91:19
  20: <databend_query::pipelines::executor::processor_async_task::ProcessorAsyncTask as core::future::future::Future>::poll::{{closure}}
             at ./src/query/service/src/pipelines/executor/processor_async_task.rs:111:17
  21: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/core/src/panic/unwind_safe.rs:271:9
  22: do_call<core::panic::unwind_safe::AssertUnwindSafe<databend_query::pipelines::executor::processor_async_task::{impl#1}::poll::{closure_env#0}>, core::task::poll::Poll<core::result::Result<(), common_exception::exception::ErrorCode>>>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/panicking.rs:483:40
  23: __rust_try
  24: try<core::task::poll::Poll<core::result::Result<(), common_exception::exception::ErrorCode>>, core::panic::unwind_safe::AssertUnwindSafe<databend_query::pipelines::executor::processor_async_task::{impl#1}::poll::{closure_env#0}>>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/panicking.rs:447:19
  25: std::panic::catch_unwind
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/panic.rs:137:14
  26: poll
             at ./src/query/service/src/pipelines/executor/processor_async_task.rs:110:13
  27: {closure#0}<databend_query::pipelines::executor::processor_async_task::ProcessorAsyncTask>
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/core.rs:184:17
  28: with_mut<tokio::runtime::task::core::Stage<databend_query::pipelines::executor::processor_async_task::ProcessorAsyncTask>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#3}::poll::{closure_env#0}<databend_query::pipelines::executor::processor_async_task::ProcessorAsyncTask>>
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/loom/std/unsafe_cell.rs:14:9
  29: tokio::runtime::task::core::CoreStage<T>::poll
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/core.rs:174:13
  30: tokio::runtime::task::harness::poll_future::{{closure}}
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/harness.rs:480:19
  31: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/core/src/panic/unwind_safe.rs:271:9
  32: do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<databend_query::pipelines::executor::processor_async_task::ProcessorAsyncTask, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::worker::Shared>>>, core::task::poll::Poll<()>>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/panicking.rs:483:40
  33: __rust_try
  34: try<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<databend_query::pipelines::executor::processor_async_task::ProcessorAsyncTask, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::worker::Shared>>>>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/panicking.rs:447:19
  35: std::panic::catch_unwind
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/panic.rs:137:14
  36: tokio::runtime::task::harness::poll_future
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/harness.rs:468:18
  37: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/harness.rs:104:27
  38: tokio::runtime::task::harness::Harness<T,S>::poll
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/harness.rs:57:15
  39: tokio::runtime::task::raw::poll
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/raw.rs:194:5
  40: tokio::runtime::task::raw::RawTask::poll
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/raw.rs:134:18
  41: tokio::runtime::task::LocalNotified<S>::run
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/mod.rs:385:9
  42: {closure#0}
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/scheduler/multi_thread/worker.rs:421:13
  43: tokio::coop::with_budget::{{closure}}
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/coop.rs:102:9
  44: try_with<core::cell::Cell<tokio::coop::Budget>, tokio::coop::with_budget::{closure_env#0}<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core, alloc::alloc::Global>, ()>, tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure_env#0}>, core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core, alloc::alloc::Global>, ()>>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/thread/local.rs:446:16
  45: std::thread::local::LocalKey<T>::with
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/thread/local.rs:422:9
  46: with_budget<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core, alloc::alloc::Global>, ()>, tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure_env#0}>
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/coop.rs:95:5
  47: budget<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core, alloc::alloc::Global>, ()>, tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure_env#0}>
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/coop.rs:72:5
  48: run_task
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/scheduler/multi_thread/worker.rs:420:9
  49: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/scheduler/multi_thread/worker.rs:387:24
  50: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/scheduler/multi_thread/worker.rs:372:17
  51: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/macros/scoped_tls.rs:61:9
  52: tokio::runtime::scheduler::multi_thread::worker::run
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/scheduler/multi_thread/worker.rs:369:5
  53: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/scheduler/multi_thread/worker.rs:348:45
  54: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/blocking/task.rs:42:21
  55: {closure#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/core.rs:184:17
  56: with_mut<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#3}::poll::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>>
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/loom/std/unsafe_cell.rs:14:9
  57: tokio::runtime::task::core::CoreStage<T>::poll
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/core.rs:174:13
  58: tokio::runtime::task::harness::poll_future::{{closure}}
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/harness.rs:480:19
  59: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/core/src/panic/unwind_safe.rs:271:9
  60: do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::NoopSchedule>>, core::task::poll::Poll<()>>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/panicking.rs:483:40
  61: __rust_try
  62: try<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::NoopSchedule>>>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/panicking.rs:447:19
  63: std::panic::catch_unwind
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/panic.rs:137:14
  64: tokio::runtime::task::harness::poll_future
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/harness.rs:468:18
  65: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/harness.rs:104:27
  66: tokio::runtime::task::harness::Harness<T,S>::poll
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/harness.rs:57:15
  67: tokio::runtime::task::raw::poll
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/raw.rs:194:5
  68: tokio::runtime::task::raw::RawTask::poll
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/raw.rs:134:18
  69: tokio::runtime::task::UnownedTask<S>::run
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/task/mod.rs:422:9
  70: tokio::runtime::blocking::pool::Task::run
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/blocking/pool.rs:111:9
  71: tokio::runtime::blocking::pool::Inner::run
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/blocking/pool.rs:346:17
  72: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /opt/rust/cargo/registry/src/index.crates.io-93a9e06d8945f2c0/tokio-1.21.2/src/runtime/blocking/pool.rs:321:13
  73: __rust_begin_short_backtrace<tokio::runtime::blocking::pool::{impl#5}::spawn_thread::{closure_env#0}, ()>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/sys_common/backtrace.rs:121:18
  74: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/thread/mod.rs:551:17
  75: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/core/src/panic/unwind_safe.rs:271:9
  76: do_call<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#5}::spawn_thread::{closure_env#0}, ()>>, ()>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/panicking.rs:483:40
  77: __rust_try
  78: try<(), core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#5}::spawn_thread::{closure_env#0}, ()>>>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/panicking.rs:447:19
  79: std::panic::catch_unwind
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/panic.rs:137:14
  80: {closure#1}<tokio::runtime::blocking::pool::{impl#5}::spawn_thread::{closure_env#0}, ()>
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/thread/mod.rs:550:30
  81: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/core/src/ops/function.rs:510:5
  82: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/alloc/src/boxed.rs:2000:9
  83: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/alloc/src/boxed.rs:2000:9
  84: std::sys::unix::thread::Thread::new::thread_start
             at /rustc/7eef946fc0e0eff40e588eab77b09b287accbec3/library/std/src/sys/unix/thread.rs:108:17
  85: start_thread
  86: clone

Xuanwo · 2022-11-08T08:55:36Z

This test has been disabled in #8699

So we can have more time to fix it.

xudong963 · 2022-11-11T08:01:35Z

How's it going? I want to join you to fix it together.

Xuanwo · 2022-11-11T08:24:06Z

How's it going? I want to join you to fix it together.

No progress so far. The latest progress is we can get the error backtrace: it seems our server shutdown too early.

xudong963 · 2022-12-04T16:58:01Z

Difficult to find a case that can reproduce stably 😭

zhang2014 self-assigned this Sep 30, 2022

Xuanwo added this to Xuanwo's Work Sep 30, 2022

Xuanwo moved this to 📋 Backlog in Xuanwo's Work Sep 30, 2022

Xuanwo moved this from 📋 Backlog to 🪤 In Waiting in Xuanwo's Work Oct 3, 2022

Xuanwo mentioned this issue Oct 25, 2022

Flaky Stateless Test Case: 13_tpch/13_0021_q21 #8398

Closed

Xuanwo mentioned this issue Oct 25, 2022

sql failed with message "(1105, 'Code: 1067, displayText = error reading a body from connection: connection reset (while in processor thread 6).')" #8221

Closed

2 tasks

Xuanwo mentioned this issue Oct 25, 2022

Add some message if flight error #8433

Closed

Xuanwo mentioned this issue Oct 26, 2022

flaky test: 15_0007_right_anti_join: error reading a body from connection: connection reset, source: Some(hyper::Error(Body, Error { kind: Io(Kind(ConnectionReset)) })) #8445

Closed

This was referenced Oct 28, 2022

fix: Move flight service client init to global io runtime #8511

Merged

flaky q5 in stateless test #8534

Closed

Xuanwo changed the title ~~Flaky test: 13_tpch/13_0007_q7~~ Flaky test: tpch test on cluster failed for reading a body from connection: broken pipe Oct 29, 2022

xudong963 mentioned this issue Nov 6, 2022

flaky test: 13_0005_q5.sql #8657

Closed

BohuTANG mentioned this issue Nov 7, 2022

chore(input format): more tests and diagnostic info. #8664

Merged

This was referenced Nov 7, 2022

ci: Enable backtrace to debug cluster related issue #8672

Merged

chore: Add error message for mysql result writer #8688

Merged

xudong963 self-assigned this Nov 11, 2022

This was referenced Nov 14, 2022

q9 is flaky in stateless cluster test #8786

Closed

feat: geo functions #8481

Merged

zhang2014 mentioned this issue Dec 6, 2022

fix(cluster): try fix broken pipe or connect reset #9104

Merged

BohuTANG closed this as completed in #9104 Dec 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flaky test: tpch test on cluster failed for reading a body from connection: broken pipe #7986

Flaky test: tpch test on cluster failed for reading a body from connection: broken pipe #7986

Xuanwo commented Sep 30, 2022

Xuanwo commented Oct 24, 2022

xudong963 commented Oct 24, 2022

Xuanwo commented Oct 25, 2022

zhang2014 commented Oct 25, 2022

xudong963 commented Oct 25, 2022 •

edited

Loading

xudong963 commented Oct 26, 2022

Xuanwo commented Nov 7, 2022

Xuanwo commented Nov 8, 2022 •

edited

Loading

Xuanwo commented Nov 8, 2022

xudong963 commented Nov 11, 2022

Xuanwo commented Nov 11, 2022

xudong963 commented Dec 4, 2022

Flaky test: tpch test on cluster failed for reading a body from connection: broken pipe #7986

Flaky test: tpch test on cluster failed for reading a body from connection: broken pipe #7986

Comments

Xuanwo commented Sep 30, 2022

Xuanwo commented Oct 24, 2022

xudong963 commented Oct 24, 2022

Xuanwo commented Oct 25, 2022

zhang2014 commented Oct 25, 2022

xudong963 commented Oct 25, 2022 • edited Loading

xudong963 commented Oct 26, 2022

Xuanwo commented Nov 7, 2022

Xuanwo commented Nov 8, 2022 • edited Loading

Xuanwo commented Nov 8, 2022

xudong963 commented Nov 11, 2022

Xuanwo commented Nov 11, 2022

xudong963 commented Dec 4, 2022

xudong963 commented Oct 25, 2022 •

edited

Loading

Xuanwo commented Nov 8, 2022 •

edited

Loading