spanconfig: harden infrastructure for v22.1

[irfansharif]

#67679 wrapped up the major feature work for span configs (rfc: #66348). This issue tracks the miscellaneous TODOs/bugs we left along the way; we should try and address them pre-22.1. They're categorized by size+necessity and ordered unscientifically by when we should do them.

Small:

• (necessary) spanconfigjob: exempt requests to checkpoint from billing #72440
• (necessary) spanconfig: carve empty ranges for pseudo-table IDs #74171
• (necessary) spanconfig: set stage for migration #74266
• (necessary) spanconfig: mark reconciliation job as idle #76384
• (necessary) sql,spanconfig: temp indexes for index backfill should share a configuration with the final index #76608
• (necessary) spanconfig: AUTO SPAN CONFIG job does not handle duplicate after being restored #70173
• spanconfig: seed existing tenants and create new ones with RANGE TENANT  #79039
• Add a test variant of spanconfig: verify migration for rangefeed enablement #76466 for secondary tenants
• spanconfig: add mechanism to alter the buffer limit used by the KVSubscriber #77539
• Expose the number of KVSubscriber restarts due to internal errors as a metric
• Expose metrics on the KVAccessor; number of invocations, number of spans, how long they take (even if just a counter of total duration)
• Expose per-store metrics on how many replicas each tenant has

Medium:

• (necessary) *: future proof tests for span configs #74265
• (necessary) *: enable span configs by default #73876
• (necessary) spanconfig: address fallout after enabling-by-default #75639
• (necessary) zoneconfig,sql/catalog: ensure dynamic system tables support rangefeeds #73045
• spanconfig: paginate server-side for {Get,Update}SpanConfigs #77505
• (necessary) kvserver: limit the number of kv spans configurable by a tenant #70555
• (necessary) spanconfig: ensure safety without mutual exclusion in the reconciliation job #73789
• kv: collapse adjacent span configs to reduce number of necessary splits #72389
  • config,kvserver: don't unconditionally split on table boundaries for the host tenant #66063
  • sql: support manual splits on table boundaries for secondary tenants #65903
• spanconfig: checkpoint the reconciliation job and retry eagerly when possible #73694
• spanconfig: ensure splits on tenant boundaries within KV #73749

Epic CRDB-10563

Jira issue: CRDB-11795

[ajwerner]

@irfansharif can we track #73045 somewhere if not here? It's really in the critical path and I need to know when and how it's getting done.

[irfansharif]

Added, this list as good as any to track it.

[irfansharif]

Adding #76608 to the list, something we hadn't thought of in the context of index-backfills.

[irfansharif]

This thing is done. There are specific issues or PRs for all the remaining nice-to-haves. Two shorter term things we should probably do:

• spanconfig: add component metrics #80147; track additional metrics we could export, and
• spanconfig: add helpful crdb_internal builtins #80148; add a few builtins that might come in handy during escalations.