spanconfig: ensure splits on tenant boundaries within KV

[irfansharif]

Is your feature request related to a problem? Please describe.

With the span configs infrastructure, it's currently possible for us to end up with ranges that straddle tenant boundaries (even if in a somewhat benign way). In this infrastructure we determine split boundaries by looking only at the start keys of system.span_configurations. A tenant could then augment their reconciliation process to install configs starting somewhat later in their addressable keyspace. This could induce KV to consider a range boundary that starts at the previous tenant's keyspace (albeit the tail end of it) and ending within the current tenant's one -- that's no good.

Describe the solution you'd like

We could do two things:

(i) Have each tenant install a span config that demarcates the end of its keyspace (in our example the previous tenant would defensively prevent its data straddling through the tenant ID boundary);
(ii) Have KV enforce these hard boundaries in keyspans that straddle tenant keyspaces.

Doing (ii) feels saner, and we already do something similar when seeding system.span_configurations for newly created tenants. We could have secondary tenants still govern span configurations over their keyspace, but we'd always split on the tenant boundary. For modified tenants, the config we'd apply over that range would be the fallback one KV already uses for missing spans. For unmodified tenants, it could either be the config for:

(i) its system.descriptor (very first table in its keyspace), or
(ii) whatever the tenant's RANGE DEFAULT is.

Describe alternatives you've considered

See above.

Additional context

See #71994 and #73746 for more details.

Jira issue: CRDB-11727

Epic CRDB-23344

[ajstorm]

@ecwall @arulajmani is this addressed with 9d4e27a?

[arulajmani]

Yeah, this can be closed now.