kv: integrate raft async storage writes #94165

Fixes cockroachdb#17500. Waiting on github.com/cockroachdb/pebble/pull/2117. This commit integrates with the `AsyncStorageWrites` functionality that we added to Raft in github.com/etcd-io/raft/pull/8. \## Approach The commit makes the minimal changes needed to integrate with async storage writes and pull fsyncs out of the raft state machine loop. It does not make an effort to extract the non-durable portion of raft log writes or raft log application onto separate goroutine pools, as was described in cockroachdb#17500. Those changes will also be impactful, but they're non trivial and bump into a pipelining vs. batching trade-off, so they are left as future work items (TODO(nvanbenschoten): open new issues). With this change, asynchronous Raft log syncs are enabled by the new `DB.ApplyNoSyncWait` Pebble API introduced in github.com/cockroachdb/pebble/pull/2117. The `handleRaftReady` state machine loop continues to initiate Raft log writes, but it uses the Pebble API to offload waiting on durability to a separate goroutine. This separate goroutine then sends the corresponding `MsgStorageAppend`'s response messages where they need to go (locally and/or to the Raft leader) when the fsync completes. The async storage writes functionality in Raft makes this all safe. \## Benchmark Results The result of this change is reduced interference between Raft proposals. As a result, it reduces end-to-end commit latency. github.com/etcd-io/raft/pull/8 presented a collection of benchmark results captured from integrating async storage writes with rafttoy. When integrated into CockroachDB, we see similar improvements to average and tail latency. However, it doesn't provide the throughput improvements at the top end because log appends and state machine application have not yet been extracted into separate goroutine pools, which would facilitate increased opportunity for batching. TODO: add images ---- Release note (performance improvement): The Raft proposal pipeline has been optimized to reduce interference between Raft proposals. This improves average and tail write latency at high concurrency.

…previous LastIndex This commit removes logic in `apply.Task.AckCommittedEntriesBeforeApplication` which was ensuring that we don't acknowledge committed Raft log entries before they were durable in a node's local Raft log. This is now ensured inside of the etcd/raft library when AsyncStorageWrites is enabled, as the comment added here describes.

This commit adds a testing facility to randomly disable non-blocking log syncs in tests. This ensures that we properly handle the case where non-blocking log sync is disabled while some log syncs are still in-progress.

This commit adds a testing facility to shuffle local raft messages before delivering them to the raft state machine. These are not required to be in order, so we shuffle in tests ensure that re-ordering is handled properly.

Simple rename to make the semantics of these fields more clear and harder to misinterpret.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kv: integrate raft async storage writes #94165

kv: integrate raft async storage writes #94165

Commits on Feb 2, 2023