kvserver: introduce blocking vfs.FS; add some tests

This introduces a `vfs.FS` implementation that can be toggled to block
on any write activity.

Add two tests:

1. check that even though a durable commit is inflight, we can still
   write non-durably. I.e. at the pebble level there is no immediate
   reason why we couldn't keep *applying* the raft log even though log
   *appends* are stuck.
2. end-to-end test where we set up a three node cluster, send unique
   writes and stall the leaseholder's disk, hoping for the writes to
   keep going through. This fails, as is expected, because raft does
   not even put up entries for application if they haven't been durably
   appended to the leader's log (which they can't be due to the stall
   we're injecting).

If we can get 2) to pass, we can have reason to believe that we can
address cockroachdb#88699 (though we'd need to verify with "real" disk stalls
and higher write volumes/stall durations and would certainly need to
configure pebble to allow much more data to accumulate in memory).

The additional benefit test 2) provides is that it prints who is
blocked. At the time of writing, it's encouraging - we see just
the LogWriter blocked on a call to `Write` (via `flushPending`).o

The next step would be to hack in the ability to apply entries
that aren't durable on the leader and to see what we get.

Touches cockroachdb#88699.

Release note: None

pkg/base/BUILD.bazel

@@ -40,6 +40,7 @@ go_library(
         "@com_github_cockroachdb_errors//:errors",
         "@com_github_cockroachdb_errors//oserror",
         "@com_github_cockroachdb_pebble//:pebble",
+        "@com_github_cockroachdb_pebble//vfs",
         "@com_github_cockroachdb_redact//:redact",
         "@com_github_dustin_go_humanize//:go-humanize",
         "@com_github_spf13_pflag//:pflag",

pkg/base/store_spec.go

@@ -29,6 +29,7 @@ import (
 	"github.com/cockroachdb/errors"
 	"github.com/cockroachdb/errors/oserror"
 	"github.com/cockroachdb/pebble"
+	"github.com/cockroachdb/pebble/vfs"
 	"github.com/cockroachdb/redact"
 	humanize "github.com/dustin/go-humanize"
 	"github.com/spf13/pflag"
@@ -248,7 +249,9 @@ type StoreSpec struct {
 	Size        SizeSpec
 	BallastSize *SizeSpec
 	InMemory    bool
-	Attributes  roachpb.Attributes
+	// Only used if InMem.
+	FS         vfs.FS // TODO(tbg): find a better way to inject this hook
+	Attributes roachpb.Attributes
 	// StickyInMemoryEngineID is a unique identifier associated with a given
 	// store which will remain in memory even after the default Engine close
 	// until it has been explicitly cleaned up by CleanupStickyInMemEngine[s]

pkg/kv/kvserver/BUILD.bazel

@@ -255,6 +255,7 @@ go_test(
         "client_split_burst_test.go",
         "client_split_test.go",
         "client_store_test.go",
+        "client_stuck_disk_test.go",
         "client_tenant_test.go",
         "client_test.go",
         "closed_timestamp_test.go",