kv: disable timestamp cache + current clock assertion

Closes #60580.
Closes #60736.
Closes #60779.
Closes #61060.

This was added in 218a5a3. The check was more of a sanity check that we have and
always have had an understand of the timestamps that can enter the timestamp
cache. The fact that it's failing is a clear indication that there were issues
in past releases, because a lease transfer used to only be safe if the outgoing
leaseholder's clock was above the time of any read in its timestamp cache. We
now ship a snapshot of the timestamp cache on lease transfers, so that invariant
is less important.

I'd still like to get to the bottom of this, but I'll do so on my own branch,
off of master where it's causing disruption.

Release justification: avoid assertion failures

pkg/kv/kvserver/replica_tscache.go

@@ -52,7 +52,14 @@ func (r *Replica) addToTSCacheChecked(
 	// directly instead of using the local HLC clock as a proxy for its high
 	// water-mark, but it serves as a good proxy for proper handling of HLC
 	// clock updates and, by extension, observed timestamps.
-	if !ts.Synthetic && st.Now.ToTimestamp().Less(ts) {
+	//
+	// TODO(nvanbenschoten): this is currently disabled because we seem to
+	// regularly hit it on master. Now that we ship a snapshot of the timestamp
+	// cache on lease transfers instead of just the current clock time, the
+	// property this is asserting is no longer quite as important, so we can
+	// disable the check. However, it would still be nice to track down how we
+	// can hit this.
+	if !ts.Synthetic && st.Now.ToTimestamp().Less(ts) && false {
 		log.Fatalf(ctx, "Unsafe timestamp cache update! Cannot add timestamp %s to timestamp "+
 			"cache after evaluating %v (resp=%v; err=%v) with local hlc clock at timestamp %s. "+
 			"Non-synthetic timestamps should always lag the local hlc clock.", ts, ba, br, pErr, st.Now)