spanconfig: visualize differences with gossiped SystemConfigSpan

[irfansharif]

spanconfig: visualize differences with gossiped SystemConfigSpan

We introduce a test-only package that compares the span configs
infrastructure against the gossiped SystemConfigSpan it was designed to
replace. The datadriven test spits out a visual "diff" of the two
subsystems, indicating exactly where they differ (or not!) in terms of
implied range boundaries and the configurations to apply. It does so by
having both systems run side-by-side and scanning their contents. These
range boundaries and configurations have second-order effects (various
actions are executed on behalf of ranges by internal queues), but we
don't bother with any of that in the test.

Of particular note is the behavior of each system in the presence of
secondary tenants that create arbitrary schemas and zone configs. This
after all is what motivated the new infrastructure. Here's an example,
as a diff, that shows how the system config span is unable to split
within an active tenant's boundary unlike the new thing:

configs version=current offset=37 limit=5
----
...
/Tenant/10                                 range default
/Tenant/11/Table/3                         range default
/Tenant/11/Table/4                         range default
/Tenant/11/Table/5                         range default
/Tenant/11/Table/6                         range default
...

configs version=legacy offset=43
----
...
/Tenant/10                                 range default
/Tenant/11                                 range default

diff offset=31 limit=8
----
--- gossiped system config span (legacy)
+++ span config infrastructure (current)
...
@@ -43,4 +37,37 @@
 /Table/47                                  range system
 /Tenant/10                                 range default
-/Tenant/11                                 range default
+/Tenant/11/Table/3                         range default
+/Tenant/11/Table/4                         range default
+/Tenant/11/Table/5                         range default
+/Tenant/11/Table/6                         range default
...

This commit also gets rid of the ShadowReader. We were using it
previously to manually diagnose differences between the two subsystems.
With the improved testing, we can get rid of the painter's tape.

---

spanconfig: rm benign differences with SystemConfigSpan

In the previous commit we demonstrated visually the differences between
the gossip-backed system config span and the new infrastructure. This
commit gets rid of the benign differences. We could also just not do
anything here, but it's only a minor change.

Release note: None

---

Only the last two commits are of interest here, the earlier ones are from #71994.

[cockroach-teamcity]

This change is 

[irfansharif]

(Gentle bump here as well, it's a test only change + plus an entirely optional commit I'm happy to drop. It led to the realization in #73749 which we'll try and get to pre-22.1.)

[ajwerner]

Can you rebase this now that #71994 has merged? I think it but it'll be easy to take a last pass after the PR stands alone.

Reviewed 5 of 28 files at r1, 1 of 8 files at r2, 11 of 19 files at r5, 7 of 8 files at r6.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @ajwerner, @arulajmani, @irfansharif, and @nvanbenschoten)

---

pkg/ccl/spanconfigccl/spanconfigcomparedccl/datadriven_test.go, line 168 at r6 (raw file):

Quoted 15 lines of code…

					// Now that all tenants have reconciled their SQL changes,
					// read the current time and wait for the kvsubscriber to
					// advance past it. This lets us observe the effects of all:
					// (i)  reconciliation processes;
					// (ii) tenant initializations (where seed span configs are
					//      installed).
					now := systemTenant.Clock().Now()
					testutils.SucceedsSoon(t, func() error {
						lastUpdated := kvSubscriber.LastUpdated()
						if lastUpdated.Less(now) {
							return errors.Newf("kvsubscriber last updated timestamp (%s) lagging barrier timestamp (%s)",
								lastUpdated.GoTime(), now.GoTime())
						}
						return nil
					})

does this SuceedsSoon need to be nested in the other one?

[irfansharif]

Rebased.

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @ajwerner, @arulajmani, and @nvanbenschoten)

---

pkg/ccl/spanconfigccl/spanconfigcomparedccl/datadriven_test.go, line 168 at r6 (raw file):

Previously, ajwerner wrote…

					// Now that all tenants have reconciled their SQL changes,
					// read the current time and wait for the kvsubscriber to
					// advance past it. This lets us observe the effects of all:
					// (i)  reconciliation processes;
					// (ii) tenant initializations (where seed span configs are
					//      installed).
					now := systemTenant.Clock().Now()
					testutils.SucceedsSoon(t, func() error {
						lastUpdated := kvSubscriber.LastUpdated()
						if lastUpdated.Less(now) {
							return errors.Newf("kvsubscriber last updated timestamp (%s) lagging barrier timestamp (%s)",
								lastUpdated.GoTime(), now.GoTime())
						}
						return nil
					})

does this SuceedsSoon need to be nested in the other one?

It doesn't, un-nested.

[ajwerner]

Reviewed 5 of 19 files at r5, 5 of 59 files at r7, 8 of 54 files at r8, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @ajwerner, @arulajmani, and @nvanbenschoten)

[ajwerner]

Reviewed 5 of 19 files at r5, 5 of 59 files at r7, 8 of 54 files at r8, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @ajwerner, @arulajmani, and @nvanbenschoten)

[irfansharif]

bors r+

[irfansharif]

bors r-

[craig]

Canceled.

[irfansharif]

bors r+

[craig]

Build succeeded:

• GitHub CI (Cockroach)