[core] Move GCS InternalKV workload to dedicated thread. #47736
Conversation
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
rynewang
force-pushed
the
dedicated-kv-ioctx
branch
2 times, most recently
from
aadcd14
to
c84fd80
Compare
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
|
Note: TSAN captured a new data race: GetAllJobInfo sends RPCs to workers (main thread) and KVMultiGet (the new ioctx thread). Both modifies a counter that implements a "gather all these tasks and then run callback" logic. I don't want a new mutex just for this; so I used atomics. So now, both threads access to a same atomic<size_t>. They perform atomic |
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
src/ray/gcs/gcs_server/gcs_server.cc
Outdated
| RAY_LOG(INFO) << "main_service_ Event stats:\n\n" | ||
| << main_service_.stats().StatsString() << "\n\n"; | ||
| RAY_LOG(INFO) << "pubsub_io_context_ Event stats:\n\n" | ||
| << pubsub_io_context_.GetIoService().stats().StatsString() << "\n\n"; | ||
| RAY_LOG(INFO) << "kv_io_context_ Event stats:\n\n" | ||
| << kv_io_context_.GetIoService().stats().StatsString() << "\n\n"; | ||
| RAY_LOG(INFO) << "task_io_context_ Event stats:\n\n" | ||
| << task_io_context_.GetIoService().stats().StatsString() << "\n\n"; | ||
| RAY_LOG(INFO) << "ray_syncer_io_context_ Event stats:\n\n" | ||
| << ray_syncer_io_context_.GetIoService().stats().StatsString() | ||
| << "\n\n"; |
There was a problem hiding this comment.
This is user-facing, let's make this user-friendly -- instead of using field name describe what they are
There was a problem hiding this comment.
I placed it this way for ease of searching logs - I don't think Ray users (other than Core developers) will ever need to read this...
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
src/ray/common/asio/asio_util.h
Outdated
| * The constructor takes a thread name and starts the thread. | ||
| * The destructor stops the io_service and joins the thread. | ||
| */ | ||
| class InstrumentedIoContextWithThread { |
There was a problem hiding this comment.
should be IO instead of Io? lol
Signed-off-by: Ruiyang Wang <rywang014@gmail.com>
Every time GetInternalConfig reads from table_storage, but it's never mutated. Moves to internal kv as a simple in-mem get (no more read from redis). This itself should slightly update performance. But with #47736 it should improve start up latency a lot in thousand-node clusters. In theory we can remove it all for good, instead just put it as an InternalKV entry. but let's do things step by step. Signed-off-by: Ruiyang Wang <rywang014@gmail.com> Signed-off-by: Ruiyang Wang <56065503+rynewang@users.noreply.github.com>
Most GCS services are on a same asio thread. On really big clusters, e.g. thousands of nodes, event loop queuing time can be an issue. Note we already have task_events and pubsub in dedicated threads and respective event loops and we can dedup code.
Creates a dedicated io context with thread for InternalKV workload, including
Notably gcs_table_storage_ is not on kv_io_service_, because that's used by all other non-KV services and we want it be isolated from the busy kv_io_service_.
This change makes
PeriodicalRunnermulti-thread. Makes it thread-safe with atomic bool.