Merge #69371 #69679

69371: sql: introduce large_full_scan_rows session variable r=yuzefovich a=michae2

This commit adds a new session variable (as well as the corresponding
cluster setting) `large_full_scan_rows` that determines the limit on
the table size for which `disallow_full_table_scans` considers the full
scans as "large" and rejects such queries. Note that when stats aren't
available, all scans are considered "large". This adds more granularity
to `disallow_full_table_scans` guardrail so that it would reject only
"large" scans. Note that the internally-issued queries are still
unaffected. The default value for the new variable is 1000 and setting
it to 0 will revert to the previous behavior where all full table/index
scans are rejected.

Release justification: low-risk improvement to the existing
functionality.

Release note (ops change): New session variable `large_full_scan_rows`
(as well as the corresponding cluster setting
`sql.defaults.large_full_scan_rows`) are introduced. This setting
determines which tables are considered "large" for the purposes of
enabling `disallow_full_table_scans` feature to reject full table/index
scans only of "large" table. The default value for the new setting is
1000, and in order to reject all full table/index scans (the previous
behavior) one can set the new setting to 0. Internally-issued queries
aren't affected, and the new setting has no impact when
`disallow_full_table_scans` feature is not enabled.

69679: migrations: persist progress during intents migration r=dt a=dt

Release note: none.
Release justification: improve new functionality.

Co-authored-by: Michael Erickson <michae2@cockroachlabs.com>
Co-authored-by: David Taylor <tinystatemachine@gmail.com>

docs/generated/settings/settings-for-tenants.txt

@@ -99,6 +99,7 @@ sql.defaults.insert_fast_path.enabled	boolean	true	default value for enable_inse
 sql.defaults.interleaved_tables.enabled	boolean	false	allows creation of interleaved tables or indexes
 sql.defaults.intervalstyle	enumeration	postgres	default value for IntervalStyle session setting [postgres = 0, iso_8601 = 1, sql_standard = 2]
 sql.defaults.intervalstyle.enabled	boolean	false	default value for intervalstyle_enabled session setting
+sql.defaults.large_full_scan_rows	float	1000	default value for large_full_scan_rows session setting which determines the maximum table size allowed for a full scan when disallow_full_table_scans is set to true
 sql.defaults.locality_optimized_partitioned_index_scan.enabled	boolean	true	default value for locality_optimized_partitioned_index_scan session setting; enables searching for rows in the current region before searching remote regions
 sql.defaults.lock_timeout	duration	0s	default value for the lock_timeout; default value for the lock_timeout session setting; controls the duration a query is permitted to wait while attempting to acquire a lock on a key or while blocking on an existing lock in order to perform a non-locking read on a key; if set to 0, there is no timeout
 sql.defaults.on_update_rehome_row.enabled	boolean	true	default value for on_update_rehome_row; enables ON UPDATE rehome_row() expressions to trigger on updates

docs/generated/settings/settings.html

@@ -104,6 +104,7 @@
 <tr><td><code>sql.defaults.interleaved_tables.enabled</code></td><td>boolean</td><td><code>false</code></td><td>allows creation of interleaved tables or indexes</td></tr>
 <tr><td><code>sql.defaults.intervalstyle</code></td><td>enumeration</td><td><code>postgres</code></td><td>default value for IntervalStyle session setting [postgres = 0, iso_8601 = 1, sql_standard = 2]</td></tr>
 <tr><td><code>sql.defaults.intervalstyle.enabled</code></td><td>boolean</td><td><code>false</code></td><td>default value for intervalstyle_enabled session setting</td></tr>
+<tr><td><code>sql.defaults.large_full_scan_rows</code></td><td>float</td><td><code>1000</code></td><td>default value for large_full_scan_rows session setting which determines the maximum table size allowed for a full scan when disallow_full_table_scans is set to true</td></tr>
 <tr><td><code>sql.defaults.locality_optimized_partitioned_index_scan.enabled</code></td><td>boolean</td><td><code>true</code></td><td>default value for locality_optimized_partitioned_index_scan session setting; enables searching for rows in the current region before searching remote regions</td></tr>
 <tr><td><code>sql.defaults.lock_timeout</code></td><td>duration</td><td><code>0s</code></td><td>default value for the lock_timeout; default value for the lock_timeout session setting; controls the duration a query is permitted to wait while attempting to acquire a lock on a key or while blocking on an existing lock in order to perform a non-locking read on a key; if set to 0, there is no timeout</td></tr>
 <tr><td><code>sql.defaults.on_update_rehome_row.enabled</code></td><td>boolean</td><td><code>true</code></td><td>default value for on_update_rehome_row; enables ON UPDATE rehome_row() expressions to trigger on updates</td></tr>

pkg/jobs/jobspb/jobs.proto

@@ -711,6 +711,7 @@ message MigrationDetails {
 }
 
 message MigrationProgress {
+  bytes watermark = 1;
 }
 
 message AutoSQLStatsCompactionDetails {

pkg/migration/migrations/BUILD.bazel

@@ -54,6 +54,7 @@ go_library(
         "//pkg/util/protoutil",
         "//pkg/util/stop",
         "//pkg/util/syncutil",
+        "//pkg/util/timeutil",
         "//pkg/util/uuid",
         "@com_github_cockroachdb_errors//:errors",
         "@com_github_cockroachdb_redact//:redact",