rt(alt): fix a number of concurrency bugs

[carllerche]

Expands loom coverage and fixes a number of bugs.

Closes #5888
Depends on: tokio-rs/loom#323

[carllerche]

This PR makes the local queue capacity configurable but does not expose the option. At some point, it might, but right now, it is for tests.

[carllerche]

I will revert his. I don't think it is needed anymore.

[carllerche]

Edit: this was left over debugging code gone wrong.

Looks like there is another spot it is failing

thread 'tokio-runtime-worker' panicked at 'steal=596; real=600; tail=600', tokio/src/runtime/scheduler/multi_thread_alt/queue.rs:133:13
stack backtrace:
   0: rust_begin_unwind
             at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:593:5
   1: core::panicking::panic_fmt
             at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/panicking.rs:67:14
   2: tokio::runtime::scheduler::multi_thread_alt::queue::Local<T>::push_back
             at ./src/runtime/scheduler/multi_thread_alt/queue.rs:133:13
   3: tokio::runtime::scheduler::multi_thread_alt::worker::Worker::next_remote_task_batch_synced
             at ./src/runtime/scheduler/multi_thread_alt/worker.rs:816:9
   4: tokio::runtime::scheduler::multi_thread_alt::worker::Worker::next_remote_task_batch
             at ./src/runtime/scheduler/multi_thread_alt/worker.rs:785:26
   5: tokio::runtime::scheduler::multi_thread_alt::worker::Worker::next_notified_task
             at ./src/runtime/scheduler/multi_thread_alt/worker.rs:753:9
   6: tokio::runtime::scheduler::multi_thread_alt::worker::Worker::next_task
             at ./src/runtime/scheduler/multi_thread_alt/worker.rs:702:26
   7: tokio::runtime::scheduler::multi_thread_alt::worker::Worker::run
             at ./src/runtime/scheduler/multi_thread_alt/worker.rs:579:35
   8: tokio::runtime::scheduler::multi_thread_alt::worker::run::{{closure}}::{{closure}}
             at ./src/runtime/scheduler/multi_thread_alt/worker.rs:511:23
   9: tokio::runtime::context::scoped::Scoped<T>::set
             at ./src/runtime/context/scoped.rs:40:9
  10: tokio::runtime::context::set_scheduler::{{closure}}
             at ./src/runtime/context.rs:176:26
  11: std::thread::local::LocalKey<T>::try_with
             at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/thread/local.rs:270:16
  12: std::thread::local::LocalKey<T>::with
             at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/thread/local.rs:246:9
  13: tokio::runtime::context::set_scheduler
             at ./src/runtime/context.rs:176:9
  14: tokio::runtime::scheduler::multi_thread_alt::worker::run::{{closure}}
             at ./src/runtime/scheduler/multi_thread_alt/worker.rs:507:9
  15: tokio::runtime::context::runtime::enter_runtime
             at ./src/runtime/context/runtime.rs:65:16
  16: tokio::runtime::scheduler::multi_thread_alt::worker::run
             at ./src/runtime/scheduler/multi_thread_alt/worker.rs:496:5
  17: tokio::runtime::scheduler::multi_thread_alt::worker::create::{{closure}}
             at ./src/runtime/scheduler/multi_thread_alt/worker.rs:343:49
  18: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at ./src/runtime/blocking/task.rs:42:21
  19: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at ./src/runtime/task/core.rs:334:17
  20: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at ./src/loom/std/unsafe_cell.rs:16:9
  21: tokio::runtime::task::core::Core<T,S>::poll
             at ./src/runtime/task/core.rs:323:13
  22: tokio::runtime::task::harness::poll_future::{{closure}}
             at ./src/runtime/task/harness.rs:485:19
  23: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/panic/unwind_safe.rs:271:9
  24: std::panicking::try::do_call
             at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:500:40
  25: __rust_try
  26: std::panicking::try
             at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:464:19
  27: std::panic::catch_unwind
             at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panic.rs:142:14
  28: tokio::runtime::task::harness::poll_future
             at ./src/runtime/task/harness.rs:473:18
  29: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at ./src/runtime/task/harness.rs:208:27
  30: tokio::runtime::task::harness::Harness<T,S>::poll
             at ./src/runtime/task/harness.rs:153:15
  31: tokio::runtime::task::raw::poll
             at ./src/runtime/task/raw.rs:276:5
  32: tokio::runtime::task::raw::RawTask::poll
             at ./src/runtime/task/raw.rs:200:18
  33: tokio::runtime::task::UnownedTask<S>::run
             at ./src/runtime/task/mod.rs:437:9
  34: tokio::runtime::blocking::pool::Task::run
             at ./src/runtime/blocking/pool.rs:159:9
  35: tokio::runtime::blocking::pool::Inner::run
             at ./src/runtime/blocking/pool.rs:513:17
  36: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at ./src/runtime/blocking/pool.rs:471:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
worker thread panicking; aborting process

[carllerche]

The broken CI depends on #5908

[carllerche]

This is added so the scheduler can avoid the extra thread if there are no resources to drive.