Merge #75751 #76705 #77018 #77055 #77065 #77109 #77307 #77318 #77319 #…

…77328 #77335

75751: sql: Add DateStyle/IntervalStyle visitor r=e-mbrown a=e-mbrown

The DateStyle visitor allows for cast expressions with string
to interval and date/interval types to string cast to be rewritten.
These stable cast cause issues with DateStyle/IntervalStyle formatting so they
need to be wrapped in builtins containing their immutable version.

Release note: None
Release justification: Low risk update to new functionality

76705: backupccl: add prototype metadata.sst r=rhu713 a=rhu713

This adds writing of an additional file to the completion of BACKUP. This new file
is an sstable that contains the same metadata currently stored in the BACKUP_MANIFEST
file and statistics files, but organizes that data differently.

The current BACKUP_MANIFEST file contains a single binary-encoded protobuf message
of type BackupManifest, that in turn has several fields some of which are repeated
to contain e.g. the TableDescriptor for every table backed up, or every revision to
every table descriptor backed up. This can result in these manifests being quite large
in some cases, which is potentially concerning because as a single protobuf message,
one has to read and unmarshal the entire struct into memory to read any field(s) of it.

Organizing this metadata into an SSTable where repeated fields are instead stored as
separate messages under separate keys should instead allow reading it incrementally:
one can seek to a particular key or key prefix and then scan, acting on whatever data
is found as it is read, without loading the entire file at once (when opened using the
same seek-ing remote SST reader we use to read backup data ssts).

This initial prototype adds only the writer -- RESTORE does not rely on, or even open,
this new file at this time.

Release note: none.

77018: release: automate orchestration version update r=celiala a=rail

Previously, as a part of the release process we had to bump the
orchestration versions using `sed` with some error-prone regexes.

This patch adds `set-orchestration-version` subcommand to the release
tool. It uses templates in order to generate the orchestration files.

Release note: None

77055: sql: change index backfill merger to use batch api r=rhu713 a=rhu713

Use Batch API instead of txn.Scan() in order to limit the number of bytes per
batch response in the index backfill merger.

Fixes #76685.

Release note: None

77065: bazel: use test sharding more liberally r=rail a=rickystewart

Closes #76376.

Release note: None

77109: ccl/sqlproxyccl: add helpers related to connection migration r=JeffSwenson,andy-kimball a=jaylim-crl

#### ccl/sqlproxyccl: add helpers related to connection migration 

Informs #76000. Extracted from #76805.

This commit adds helpers related to connection migration. This includes support
for retrieving the transfer state through SHOW TRANSFER STATE, as well as
deserializing the session through crdb_internal.deserialize_session.

Release note: None

Release justification: Helpers added in this commit are needed for the
connection migration work. Connection migration is currently not being used
in production, and CockroachCloud is the only user of sqlproxy.
  
#### ccl/sqlproxyccl: fix math for defaultBufferSize in interceptors 

Previously, we incorrectly defined defaultBufferSize as 16K bytes. Note that
2 << 13 is 16K bytes. This commit fixes that behavior to match the original
intention of 8K bytes.

Release note: None

Release justification: This fixes an unintentional buglet within the sqlproxy
code that was introduced with the interceptors back then. Not having this in
means we're using double the memory for each connection within the sqlproxy.



77307: sql: add cluster setting to limit max size of serialized session r=otan,jaylim-crl a=rafiss

fixes #77302

The sql.session_transfer.max_session_size cluster setting can be used to
limit the max size of a session that is serialized using
crdb_internal.serialize_session.

No release note since this is not a public setting.

Release justification: high priority fix for new functionality.

Release note: None

77318: roachpb: extract keysbase to break some dependencies r=yuzefovich a=yuzefovich

This commit extracts a couple of things out of `roachpb` into new
`keysbase` package in order to break the dependency of `util/json` and
`sql/inverted` on `roachpb` (which is a part of the effort to clean up
the dependencies of `execgen`).

Addresses: #77234.

Release note: None

Release justification: low risk change to clean up the dependencies.

77319: sessiondatapb: move one enum definition into lex package r=yuzefovich a=yuzefovich

This commit moves the definition of `BytesEncodeFormat` enum from
`sessiondatapb` to `lex`. This is done in order to make `lex` not depend
on a lot of stuff (eventually on `roachpb`) and is a part of the effort
to clean up the dependencies of `execgen`. Note that the proto package
name is not changed, so this change is backwards-compatible.

Informs: #77234.

Release note: None

Release justification: low risk change to clean up the dependencies.

77328: roachtest: log stdout and stderr in sstable corruption test r=itsbilal a=nicktrav

To aid in debugging #77321, log the contents stdout and stderr if the
manifest dump command fails.

Release justification: Tests only.

Release note: None.

77335: kvserver: fix race that caused truncator to truncate non-alive replica r=tbg,erikgrinaker a=sumeerbhola

This was causing truncated state to be written to such a
replica, which would then get picked up as the
HardState.Commit value when a different replica was later
added back for the same range. See
#77030 (comment)
for the detailed explanation.

Also restore the default value of
kv.raft_log.loosely_coupled_truncation.enabled to true.

Fixes #77030

Release justification: Bug fix.
Release note: None

Co-authored-by: e-mbrown <ebsonari@gmail.com>
Co-authored-by: David Taylor <tinystatemachine@gmail.com>
Co-authored-by: Rui Hu <rui@cockroachlabs.com>
Co-authored-by: Rail Aliiev <rail@iqchoice.com>
Co-authored-by: Ricky Stewart <ricky@cockroachlabs.com>
Co-authored-by: Jay <jay@cockroachlabs.com>
Co-authored-by: Rafi Shamim <rafi@cockroachlabs.com>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Co-authored-by: Nick Travers <travers@cockroachlabs.com>
Co-authored-by: sumeerbhola <sumeer@cockroachlabs.com>

.github/CODEOWNERS

@@ -220,6 +220,7 @@
 /pkg/internal/team/          @cockroachdb/test-eng
 /pkg/jobs/                   @cockroachdb/cdc-prs
 /pkg/keys/                   @cockroachdb/kv-prs
+/pkg/keysbase/               @cockroachdb/kv-prs
 # Don't ping KV on updates to reserved descriptor IDs and such.
 /pkg/keys/constants.go       @cockroachdb/kv-prs-noreview
 /pkg/migration/              @cockroachdb/kv-prs-noreview @cockroachdb/sql-schema

cloud/kubernetes/bring-your-own-certs/client.yaml

@@ -1,3 +1,4 @@
+# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/bring-your-own-certs/client.yaml
 # This config file demonstrates how to connect to the CockroachDB StatefulSet
 # defined in bring-your-own-certs-statefulset.yaml that uses certificates
 # created outside of Kubernetes. See that file for why you may want to use it.

cloud/kubernetes/bring-your-own-certs/cockroachdb-statefulset.yaml

@@ -1,3 +1,4 @@
+# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/bring-your-own-certs/cockroachdb-statefulset.yaml
 # This config file defines a CockroachDB StatefulSet that uses certificates
 # created outside of Kubernetes. You may want to use it if you want to use a
 # different certificate authority from the one being used by Kubernetes or if

cloud/kubernetes/client-secure.yaml

@@ -1,3 +1,4 @@
+# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/client-secure.yaml
 apiVersion: v1
 kind: Pod
 metadata:

cloud/kubernetes/cluster-init-secure.yaml

@@ -1,3 +1,4 @@
+# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/cluster-init-secure.yaml
 apiVersion: batch/v1
 kind: Job
 metadata:

cloud/kubernetes/cluster-init.yaml

@@ -1,3 +1,4 @@
+# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/cluster-init.yaml
 apiVersion: batch/v1
 kind: Job
 metadata:

cloud/kubernetes/cockroachdb-statefulset-secure.yaml

@@ -1,3 +1,4 @@
+# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/cockroachdb-statefulset-secure.yaml
 apiVersion: v1
 kind: ServiceAccount
 metadata:

cloud/kubernetes/cockroachdb-statefulset.yaml

@@ -1,3 +1,4 @@
+# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/cockroachdb-statefulset.yaml
 apiVersion: v1
 kind: Service
 metadata:

cloud/kubernetes/multiregion/client-secure.yaml

@@ -1,3 +1,4 @@
+# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/multiregion/client-secure.yaml
 apiVersion: v1
 kind: Pod
 metadata:

cloud/kubernetes/multiregion/cluster-init-secure.yaml

@@ -1,3 +1,4 @@
+# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/multiregion/cluster-init-secure.yaml
 apiVersion: batch/v1
 kind: Job
 metadata:

cloud/kubernetes/multiregion/cockroachdb-statefulset-secure.yaml

@@ -1,3 +1,4 @@
+# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/multiregion/cockroachdb-statefulset-secure.yaml
 apiVersion: v1
 kind: ServiceAccount
 metadata:

cloud/kubernetes/multiregion/eks/cockroachdb-statefulset-secure-eks.yaml

@@ -1,3 +1,4 @@
+# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/multiregion/eks/cockroachdb-statefulset-secure-eks.yaml
 apiVersion: v1
 kind: ServiceAccount
 metadata:

cloud/kubernetes/performance/cockroachdb-daemonset-insecure.yaml

@@ -1,3 +1,4 @@
+# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/performance/cockroachdb-daemonset-insecure.yaml
 # This configuration file sets up a DaemonSet running CockroachDB in insecure
 # mode. For more information on why you might want to use a DaemonSet instead
 # of a StatefulSet, see our docs:

cloud/kubernetes/performance/cockroachdb-daemonset-secure.yaml

@@ -1,3 +1,4 @@
+# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/performance/cockroachdb-daemonset-secure.yaml
 # This configuration file sets up a secure DaemonSet running CockroachDB.
 # For more information on why you might want to use a DaemonSet instead
 # of a StatefulSet, see our docs:

cloud/kubernetes/performance/cockroachdb-statefulset-insecure.yaml

@@ -1,3 +1,4 @@
+# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/performance/cockroachdb-statefulset-insecure.yaml
 # This configuration file sets up an insecure StatefulSet running CockroachDB with
 # tweaks to make it more performant than our default configuration files. All
 # changes from the default insecure configuration have been marked with a comment

cloud/kubernetes/performance/cockroachdb-statefulset-secure.yaml

@@ -1,3 +1,4 @@
+# Generated file, DO NOT EDIT. Source: cloud/kubernetes/templates/performance/cockroachdb-statefulset-secure.yaml
 # This configuration file sets up a secure StatefulSet running CockroachDB with
 # tweaks to make it more performant than our default configuration files. All
 # changes from the default secure configuration have been marked with a comment

cloud/kubernetes/templates/bring-your-own-certs/client.yaml

@@ -0,0 +1,35 @@
+# This config file demonstrates how to connect to the CockroachDB StatefulSet
+# defined in bring-your-own-certs-statefulset.yaml that uses certificates
+# created outside of Kubernetes. See that file for why you may want to use it.
+# You should be able to adapt the core ideas to deploy your own custom
+# applications and connect them to the database similarly.
+#
+# The pod that this file defines will sleep in the cluster not using any
+# resources. After creating the pod, you can use it to open up a SQL shell to
+# the database by running:
+#
+# kubectl exec -it cockroachdb-client-secure -- ./cockroach sql --url="postgres://root@cockroachdb-public:26257/?sslmode=verify-full&sslcert=/cockroach-certs/client.root.crt&sslkey=/cockroach-certs/client.root.key&sslrootcert=/cockroach-certs/ca.crt"
+apiVersion: v1
+kind: Pod
+metadata:
+  name: cockroachdb-client-secure
+  labels:
+    app: cockroachdb-client
+spec:
+  serviceAccountName: cockroachdb
+  containers:
+  - name: cockroachdb-client
+    image: cockroachdb/cockroach:@VERSION@
+    # Keep a pod open indefinitely so kubectl exec can be used to get a shell to it
+    # and run cockroach client commands, such as cockroach sql, cockroach node status, etc.
+    command:
+    - sleep
+    - "2147483648" # 2^31
+    volumeMounts:
+    - name: client-certs
+      mountPath: /cockroach-certs
+  volumes:
+  - name: client-certs
+    secret:
+      secretName: cockroachdb.client.root
+      defaultMode: 256

cloud/kubernetes/templates/bring-your-own-certs/cockroachdb-statefulset.yaml

@@ -0,0 +1,244 @@
+# This config file defines a CockroachDB StatefulSet that uses certificates
+# created outside of Kubernetes. You may want to use it if you want to use a
+# different certificate authority from the one being used by Kubernetes or if
+# your Kubernetes cluster doesn't fully support certificate-signing requests
+# (e.g. as of July 2018, EKS doesn't work properly).
+#
+# To use this config file, first set up your certificates and load them into
+# your Kubernetes cluster as Secrets using the commands below:
+#
+# mkdir certs
+# mkdir my-safe-directory
+# cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key
+# cockroach cert create-client root --certs-dir=certs --ca-key=my-safe-directory/ca.key
+# kubectl create secret generic cockroachdb.client.root --from-file=certs
+# cockroach cert create-node --certs-dir=certs --ca-key=my-safe-directory/ca.key localhost 127.0.0.1 cockroachdb-public cockroachdb-public.default cockroachdb-public.default.svc.cluster.local *.cockroachdb *.cockroachdb.default *.cockroachdb.default.svc.cluster.local
+# kubectl create secret generic cockroachdb.node --from-file=certs
+# kubectl create -f bring-your-own-certs-statefulset.yaml
+# kubectl exec -it cockroachdb-0 -- /cockroach/cockroach init --certs-dir=/cockroach/cockroach-certs
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: cockroachdb
+  labels:
+    app: cockroachdb
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: cockroachdb
+  labels:
+    app: cockroachdb
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - secrets
+  verbs:
+  - get
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: cockroachdb
+  labels:
+    app: cockroachdb
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: cockroachdb
+subjects:
+- kind: ServiceAccount
+  name: cockroachdb
+  namespace: default
+---
+apiVersion: v1
+kind: Service
+metadata:
+  # This service is meant to be used by clients of the database. It exposes a ClusterIP that will
+  # automatically load balance connections to the different database pods.
+  name: cockroachdb-public
+  labels:
+    app: cockroachdb
+spec:
+  ports:
+  # The main port, served by gRPC, serves Postgres-flavor SQL, internode
+  # traffic and the cli.
+  - port: 26257
+    targetPort: 26257
+    name: grpc
+  # The secondary port serves the UI as well as health and debug endpoints.
+  - port: 8080
+    targetPort: 8080
+    name: http
+  selector:
+    app: cockroachdb
+---
+apiVersion: v1
+kind: Service
+metadata:
+  # This service only exists to create DNS entries for each pod in the stateful
+  # set such that they can resolve each other's IP addresses. It does not
+  # create a load-balanced ClusterIP and should not be used directly by clients
+  # in most circumstances.
+  name: cockroachdb
+  labels:
+    app: cockroachdb
+  annotations:
+    # Use this annotation in addition to the actual publishNotReadyAddresses
+    # field below because the annotation will stop being respected soon but the
+    # field is broken in some versions of Kubernetes:
+    # https://github.com/kubernetes/kubernetes/issues/58662
+    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
+    # Enable automatic monitoring of all instances when Prometheus is running in the cluster.
+    prometheus.io/scrape: "true"
+    prometheus.io/path: "_status/vars"
+    prometheus.io/port: "8080"
+spec:
+  ports:
+  - port: 26257
+    targetPort: 26257
+    name: grpc
+  - port: 8080
+    targetPort: 8080
+    name: http
+  # We want all pods in the StatefulSet to have their addresses published for
+  # the sake of the other CockroachDB pods even before they're ready, since they
+  # have to be able to talk to each other in order to become ready.
+  publishNotReadyAddresses: true
+  clusterIP: None
+  selector:
+    app: cockroachdb
+---
+apiVersion: policy/v1beta1
+kind: PodDisruptionBudget
+metadata:
+  name: cockroachdb-budget
+  labels:
+    app: cockroachdb
+spec:
+  selector:
+    matchLabels:
+      app: cockroachdb
+  maxUnavailable: 1
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: cockroachdb
+spec:
+  serviceName: "cockroachdb"
+  replicas: 3
+  selector:
+    matchLabels:
+      app: cockroachdb
+  template:
+    metadata:
+      labels:
+        app: cockroachdb
+    spec:
+      serviceAccountName: cockroachdb
+      affinity:
+        podAntiAffinity:
+          preferredDuringSchedulingIgnoredDuringExecution:
+          - weight: 100
+            podAffinityTerm:
+              labelSelector:
+                matchExpressions:
+                - key: app
+                  operator: In
+                  values:
+                  - cockroachdb
+              topologyKey: kubernetes.io/hostname
+      containers:
+      - name: cockroachdb
+        image: cockroachdb/cockroach:@VERSION@
+        imagePullPolicy: IfNotPresent
+        # TODO: Change these to appropriate values for the hardware that you're running. You can see
+        # the resources that can be allocated on each of your Kubernetes nodes by running:
+        #   kubectl describe nodes
+        # Note that requests and limits should have identical values.
+        resources:
+          requests:
+            cpu: "2"
+            memory: "8Gi"
+          limits:
+            cpu: "2"
+            memory: "8Gi" 
+        ports:
+        - containerPort: 26257
+          name: grpc
+        - containerPort: 8080
+          name: http
+# We recommend that you do not configure a liveness probe on a production environment, as this can impact the availability of production databases.
+#      livenessProbe:
+#        httpGet:
+#          path: "/health"
+#          port: http
+#          scheme: HTTPS
+#        initialDelaySeconds: 30
+#        periodSeconds: 5
+        readinessProbe:
+          httpGet:
+            path: "/health?ready=1"
+            port: http
+            scheme: HTTPS
+          initialDelaySeconds: 10
+          periodSeconds: 5
+          failureThreshold: 2
+        volumeMounts:
+        - name: datadir
+          mountPath: /cockroach/cockroach-data
+        - name: certs
+          mountPath: /cockroach/cockroach-certs
+        env:
+        - name: COCKROACH_CHANNEL
+          value: kubernetes-secure
+        - name: GOMAXPROCS
+          valueFrom:
+            resourceFieldRef:
+              resource: limits.cpu
+              divisor: "1"
+        - name: MEMORY_LIMIT_MIB
+          valueFrom:
+            resourceFieldRef:
+              resource: limits.memory
+              divisor: "1Mi"
+        command:
+          - "/bin/bash"
+          - "-ecx"
+          # The use of qualified `hostname -f` is crucial:
+          # Other nodes aren't able to look up the unqualified hostname.
+          - exec
+            /cockroach/cockroach
+            start
+            --logtostderr
+            --certs-dir /cockroach/cockroach-certs
+            --advertise-host $(hostname -f)
+            --http-addr 0.0.0.0
+            --join cockroachdb-0.cockroachdb,cockroachdb-1.cockroachdb,cockroachdb-2.cockroachdb
+            --cache $(expr $MEMORY_LIMIT_MIB / 4)MiB
+            --max-sql-memory $(expr $MEMORY_LIMIT_MIB / 4)MiB
+      # No pre-stop hook is required, a SIGTERM plus some time is all that's
+      # needed for graceful shutdown of a node.
+      terminationGracePeriodSeconds: 60
+      volumes:
+      - name: datadir
+        persistentVolumeClaim:
+          claimName: datadir
+      - name: certs
+        secret:
+          secretName: cockroachdb.node
+          defaultMode: 256
+  podManagementPolicy: Parallel
+  updateStrategy:
+    type: RollingUpdate
+  volumeClaimTemplates:
+  - metadata:
+      name: datadir
+    spec:
+      accessModes:
+        - "ReadWriteOnce"
+      resources:
+        requests:
+          storage: 100Gi