kv: get to zero unskipped tests for v21.1

[tbg]

We should get all tests unskipped during the 21.1 stability period.

According to my sleuthing, we have the following skipped unit tests:

git grep -F 'skip.WithIssue' -- $(find ./pkg -name 'kv*' -type d) ./pkg/server/ ./pkg/util/tracing/
pkg/kv/kvclient/kvcoord/dist_sender_server_test.go:     skip.WithIssue(t, 51795, "TODO(andreimatei): This last assertion in this test was broken by #33150. "+
pkg/kv/kvclient/kvcoord/txn_coord_sender_server_test.go:        skip.WithIssue(t, 59373, "Needs rewrite - uses tracing in illegal manner")
pkg/kv/kvserver/client_raft_test.go:    skip.WithIssue(t, 59382, "flaky test")
pkg/kv/kvserver/client_raft_test.go:    skip.WithIssue(t, 59209, "flaky test")
pkg/kv/kvserver/client_split_test.go:   skip.WithIssue(t, 59091, "flaky test")
pkg/kv/kvserver/closed_timestamp_test.go:       skip.WithIssue(t, 59448, "flaky test")
pkg/kv/kvserver/gossip_test.go: skip.WithIssue(t, 50024)
pkg/kv/kvserver/raft_log_queue_test.go: skip.WithIssue(t, 38584)

Note that we could also use this Github search:

https://github.com/cockroachdb/cockroach/issues?q=is%3Aopen+is%3Aissue+project%3Acockroachdb%2Fcockroach%2F22+label%3Askipped-test

which may highlight issues outside of our code trees but still owned by us. Either way, both searches should ultimately come back empty.

and the following roachtests:

$ bin/roachtest list tag:owner-kv | grep -v "does not match" | grep skipped
acceptance/bank/zerosum-splits [kv] (skipped: https://github.com/cockroachdb/cockroach/issues/33683 (runs into  various errors during its rebalances, see IsExpectedRelocateError))
acceptance/decommission-self [kv] (skipped: https://github.com/cockroachdb/cockroach/issues/56718)
election-after-restart [kv] (skipped: https://github.com/cockroachdb/cockroach/issues/54246)
network/tpcc/nodes=4 [kv] (skipped: https://github.com/cockroachdb/cockroach/issues/49901#issuecomment-640666646)
queue/nodes=1 [kv] (skipped: https://github.com/cockroachdb/cockroach/issues/17229)
scaledata/job-coordinator/nodes=3 [kv] (skipped: skipping flaky scaledata/job-coordinator test)
scaledata/job-coordinator/nodes=6 [kv] (skipped: skipping flaky scaledata/job-coordinator test)

This might look not too bad, until you open the Github search version of this and find that there are 161 (!) issues to be looked into: https://github.com/cockroachdb/cockroach/issues?page=1&q=is%3Aopen+is%3Aissue+project%3Acockroachdb%2Fcockroach%2F22+label%3AO-roachtest

[nvanbenschoten]

We made a push in the v21.2 stability period to unskip all tests. The only ones remaining at the end were:

• rpc: TestGRPCKeepaliveFailureFailsInflightRPCs is skipped #51800
• storage: TestNewTruncateDecision failed under stress [skipped] #38584
• roachtest: jepsen subcritical-skews tests a skipped due to ntp rate limiting #35599

None of these are immediately actionable and are all tracked elsewhere. So we can close this tracking issue.