Even distribution clients between threads

[artful88533]

Hello!
We have 3 separate redis instances. Now in top cpu used 80% and raising every month. So we decided to migrate to keydb 2 nodes with active-replica and multiple threads.
I've configured keydb with 8 threads, but the client's distribution not evenly. Also, we are using 3 haproxy load-balancing nodes on the front.

Is it normal that one thread load to 99.9%?

Run parameters

      --port 7000
      --bind 0.0.0.0
      --server-threads 8
      --server-thread-affinity true
      --active-replica yes
      --replicaof 10.64.128.52
      --client-output-buffer-limit slave 0 0 0
      --save 3600 1
      --save 1800 10
      --save 300 10000

INFO

# Server
redis_version:6.0.13
redis_git_sha1:a6782a3c
redis_git_dirty:1
redis_build_id:4c36a0f29d258da9
redis_mode:standalone
os:Linux 4.4.0-130-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:7.5.0
process_id:1
run_id:2b33ff722ed116b61bdb82d4deeb4eff35091aa5
tcp_port:7000
uptime_in_seconds:22569
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:7209256
executable:/data/keydb-server
config_file:

# Clients
connected_clients:114
client_recent_max_input_buffer:10
client_recent_max_output_buffer:3776
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
current_client_thread:2
thread_0_clients:47
thread_1_clients:50
thread_2_clients:17
thread_3_clients:1
thread_4_clients:0
thread_5_clients:0
thread_6_clients:0
thread_7_clients:0

# Memory
used_memory:254998400
used_memory_human:243.19M
used_memory_rss:435576832
used_memory_rss_human:415.40M
used_memory_peak:470049472
used_memory_peak_human:448.27M
used_memory_peak_perc:54.25%
used_memory_overhead:12247260
used_memory_startup:9233520
used_memory_dataset:242751140
used_memory_dataset_perc:98.77%
allocator_allocated:255528568
allocator_active:419897344
allocator_resident:432103424
total_system_memory:135180136448
total_system_memory_human:125.90G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.64
allocator_frag_bytes:164368776
allocator_rss_ratio:1.03
allocator_rss_bytes:12206080
rss_overhead_ratio:1.01
rss_overhead_bytes:3473408
mem_fragmentation_ratio:1.71
mem_fragmentation_bytes:180655200
mem_not_counted_for_evict:6136
mem_replication_backlog:1048576
mem_clients_slaves:20962
mem_clients_normal:1942018
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:3903991
rdb_bgsave_in_progress:0
rdb_last_save_time:1601044675
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:6
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:31645696
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:69163
total_commands_processed:1008373790
instantaneous_ops_per_sec:50002
total_net_input_bytes:225514588370
total_net_output_bytes:152321011451
instantaneous_input_kbps:10988.76
instantaneous_output_kbps:7643.44
rejected_connections:0
sync_full:3
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:272135
keyspace_misses:1
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:15960
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0

# Replication
role:active-replica
master_global_link_status:up
Master 0:
master_host:10.64.128.52
master_port:7000
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:115738676320
slave_priority:100
slave_read_only:0
connected_slaves:1
slave0:ip=10.64.128.52,port=7000,state=online,offset=296224675393,lag=1
master_replid:6245a3de3ff281e0373712e37c58f029f322388b
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:296231698469
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:296230649894
repl_backlog_histlen:1048576

# CPU
used_cpu_sys:15476.256000
used_cpu_user:40963.700000
used_cpu_sys_children:27.788000
used_cpu_user_children:186.476000
server_threads:8
long_lock_waits:35726

# Modules

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=2,expires=0,avg_ttl=0

[JohnSully]

setting the config min-clients-per-thread to 0 will distribute clients across all threads evenly. By default we will try to bunch at least 50 clients to a thread.

Usually, bunching a small number of clients results in lower overall latency as there is fixed overhead that needs to be overcome.

[JohnSully]

I think the default of 50 is a bit too high though. I set that on a high clocked desktop CPU. Server CPUs tend to prefer a lower number, of around 20 or so.

[artful88533]

@JohnSully I've set min-clients-per-thread to 20, clients distribution and cpu looks better now. Thank you!

thread_0_clients:17
thread_1_clients:20
thread_2_clients:20
thread_3_clients:20
thread_4_clients:13
thread_5_clients:18
thread_6_clients:0
thread_7_clients:0

One more question. Why CPU with keydb instances so loaded? Now it's half of all traffic.
Redis use about 120% CPU
Keydb use about 280% CPU x 2 instances
Maybe I can customize something else?

[JohnSully]

Its a timely question, the reason it uses so much CPU is because its waiting in a spinlock.

As of the next release we’ve dramatically reduced the lock contention and CPU usage is much lower.

If you want a sneak peek give the unstable branch a try.

[artful88533]

I think, I can add more threads and waiting for new release. Thanks for answer!

[JohnSully]

This bug tracks lowering the min-clients-per-thread setting to 20.

[artful88533]

When are you planning to release the next stable version?

[JohnSully]

Hey @artful88533 I just tagged the release. Docker images are being built now and will be up soon. This fix is included.

[artful88533]

Unfortunately, after update, I don't see any improvement in CPU usage :(

min-clients-per-thread setting 10

# Server
redis_version:6.0.16
redis_git_sha1:63c47a09
redis_git_dirty:1
redis_build_id:f20f71e10920f6e0
redis_mode:standalone
os:Linux 4.4.0-130-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:7.5.0
process_id:1
run_id:ea714018c29beb2651df9b6b78c61cedbb55e64c
tcp_port:7000
uptime_in_seconds:18847
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:7545904
executable:/data/keydb-server
config_file:

# Clients
connected_clients:123
client_recent_max_input_buffer:16670
client_recent_max_output_buffer:21127468360
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
current_client_thread:3
thread_0_clients:11
thread_1_clients:11
thread_2_clients:11
thread_3_clients:11
thread_4_clients:10
thread_5_clients:10
thread_6_clients:10
thread_7_clients:10
thread_8_clients:10
thread_9_clients:10
thread_10_clients:10
thread_11_clients:10

# Memory
used_memory:22082053960
used_memory_human:20.57G
used_memory_rss:18078797824
used_memory_rss_human:16.84G
used_memory_peak:22082053960
used_memory_peak_human:20.57G
used_memory_peak_perc:100.01%
used_memory_overhead:21139989779
used_memory_startup:9317912
used_memory_dataset:942064181
used_memory_dataset_perc:4.27%
allocator_allocated:22081420168
allocator_active:23446245376
allocator_resident:23631138816
total_system_memory:135180136448
total_system_memory_human:125.90G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.06
allocator_frag_bytes:1364825208
allocator_rss_ratio:1.01
allocator_rss_bytes:184893440
rss_overhead_ratio:0.77
rss_overhead_bytes:-5552340992
mem_fragmentation_ratio:0.82
mem_fragmentation_bytes:-4001752960
mem_not_counted_for_evict:21134731776
mem_replication_backlog:1048576
mem_clients_slaves:21127485546
mem_clients_normal:2135385
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:4378337
rdb_bgsave_in_progress:0
rdb_last_save_time:1601381356
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:26
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:360980480
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:58116
total_commands_processed:563716651
instantaneous_ops_per_sec:79214
total_net_input_bytes:126242348093
total_net_output_bytes:73732563241
instantaneous_input_kbps:17917.43
instantaneous_output_kbps:5230.29
rejected_connections:0
sync_full:3
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:512488
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:384307
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0

# Replication
role:active-replica
master_global_link_status:up
master_host:10.64.128.52
master_port:7000
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:64219874738
slave_priority:100
slave_read_only:0
connected_slaves:1
slave0:ip=10.64.128.52,port=7000,state=online,offset=153289693346,lag=1
master_replid:c9e75cf4b840dedb0543bbc079e048bcf8b4d6f2
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:165636466785
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:165635418210
repl_backlog_histlen:1048576

# CPU
used_cpu_sys:10442.432000
used_cpu_user:52810.068000
used_cpu_sys_children:22.924000
used_cpu_user_children:130.292000
server_threads:12
long_lock_waits:154235

# Modules

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=6,expires=0,avg_ttl=0

[JohnSully]

@artful88533 You really want the minimum number of threads necessary for your workload. Usually 8 threads is more than sufficient for workloads with more clients than you have there.

Each thread has overhead and using too many can reduce performance. A handful of clients will not saturate a thread, remember Redis handled many clients on its single thread - KeyDB’s server threads are essentially clones of that each one is capable of handling many clients.

[JohnSully]

I originally implemented min-clients-per-thread because of a tendency to select too many threads.

I’d recommend running memtier and observing what setup works best on your hardware when tweaking that setting.

[artful88533]

Hi!
Before I've tried to run keydb on virtual machines (Xenserver).
Now I've installed keydb master-master cluster directly on physical servers. It works much better! =)