[BACKPORT pg15-cherrypicks] all: Bulk port from master - 54

Summary:
 de8c392 [#22923] YSQL: In EXPLAIN command, fix relids of prefix keys when under a subquery distinct index scan.
 a137121 [#22349] yugabyted: xCluster Integration with yugabyted.
 040071d note about Jumpcloud support (#23193)
 d1572d9 [PLAT-13410] Allow YCQL index table selection
 Excluded: 47f40bb [#22144, #22145] YSQL, QueryDiagnostics: Infrastructure for QueryDiagnostics
 c51e6c9 [PLAT-14519] Master failover must not fail when some tservers are down
 c319a56 [#23100] YSQL: drop column does not complete in docdb when alter schema rpc is missed
 26eaeaa [#23071] yugabyted: Add tests for yugabyted cluster creation.
 160f24e [PLAT-14162] Support bootstrapping for editing list of databases in DR replicatoin
 a65795e [doc][yba] Ports and encryption (#23152)
 2ca9deb [#23079] DocDB: Small cleanups to stay closer to pg15-upgrade branch
 cba0254 [PLAT-14655] Fix ephemeral upload directory on container installs
 6e99ea0 [PLAT-14546] Ansible task to replace CentOS 7 repo URLs
 a9f675e [PLAT-14570] Deploy YBC build based on the architecture of destination node
 d56446f [docs] Updated OS tables (#23156)
 019ab00 [#23055] docdb: Add in-progress restore/deletion checks to DeleteSnapshot and RestoreSnapshot.
 9aed20a [PLAT-14248][PLAT-14249] - feat : Enable/Disable Enhanced PG Compatibility
 79888cf [#20482] YSQL : Allow updating follower reads in a transaction block
 a946721 [#23122] YSQL: Add pg_partman to build
 e5aea11 [PLAT-11188] Handle a node instance without node name set properly in health checks
 04cf65d [#13549] YSQL: Disable ACCESS EXCLUSIVE LOCK in pg_partman
 ed1809c [#23123] YSQL: Disable other partition methods except native in pg_partman
 bc55a46 [#23125] YSQL: Disable pg_partman background worker
 e1ef208 [#23124] YSQL: Disable advisory locks usage in pg_partman

- explain.c:
  - yb_fix_indexpr_mutator: master
    de8c392 adds this function.  Change
    varnoold to varnosyn in accordance with upstream PG
    9ce77d75c5ab094637cc4a446296dc3be6e3c221.

Test Plan: Jenkins: rebase: pg15-cherrypicks, urgent

Reviewers: jason, tfoucher

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D36576

docs/content/preview/reference/configuration/operating-systems.md

@@ -22,7 +22,6 @@ Unless otherwise noted, operating systems are supported by all supported version
 | AlmaLinux 8      | {{<icon/yes>}} | {{<icon/yes>}} | Recommended for production<br>Recommended development platform<br>Default for YugabyteDB Anywhere-deployed nodes |
 | AlmaLinux 9      | {{<icon/yes>}} | {{<icon/yes>}} |       |
 | Amazon Linux 2   | {{<icon/no>}} | {{<icon/no>}} | Supported in v2.18.0 and later<br>Deprecated in v2.20 |
-| Amazon Linux 2 (ARM, CIS-hardened) |  | {{<icon/yes>}} | Supported in v2.20.x. Database performance in this environment varies both due to CIS-hardening and ARM. For more information, contact {{% support-general %}}.|
 | CentOS 7          | {{<icon/no>}}  |                | Deprecated in v2.20 |
 | Oracle Linux 7   | {{<icon/no>}}  |                | Deprecated in v2.20 |
 | Oracle Linux 8   | {{<icon/yes>}} |                | |

docs/content/preview/yugabyte-cloud/release-notes.md

@@ -21,6 +21,10 @@ On this page:
 
 ### July 11, 2024
 
+**New features**
+
+- Support for JumpCloud Single Sign-On [federated authentication](../managed-security/managed-authentication/), to provide single sign-on access for your account users.
+
 **Database**
 
 - Ability to choose the latest [stable release](../../faq/yugabytedb-managed-faq/#what-version-of-yugabytedb-does-my-cluster-run-on) when creating a dedicated or Sandbox cluster using the new Early Access track. Early Access has more frequent updates for faster access to the most recent stable version of YugabyteDB. Currently features YugabyteDB v2024.1.0.0.

docs/content/preview/yugabyte-platform/prepare/networking.md

@@ -22,9 +22,9 @@ The following ports need to be open. (The default port numbers can be customized
 
 | From | To | Requirements |
 | :--- | :--- | :--- |
-| DB&nbsp;nodes | DB&nbsp;nodes | Open the following ports for communication between nodes in clusters. They do not need to be exposed to your application.<ul><li>7000 - YB-Master HTTP endpoint</li><li>7100 - YB-Master RPC</li><li>9000 - YB-TServer HTTP endpoint</li><li>9100 - YB-TServer RPC</li><li>18018 - YB Controller</li></ul> |
-| YBA&nbsp;node | DB nodes | Open the following ports on database cluster nodes so that YugabyteDB Anywhere can provision them.<ul><li>22 - SSH</li><li>5433 - YSQL server</li><li>7000/7100 - YB-Master HTTP/RPC endpoint</li><li>9000/9100 - YB-TServer HTTP/RPC endpoint</li><li>9042 - YCQL server</li><li>9070 - Node agent</li><li>9300 - Prometheus Node Exporter</li><li>12000 - YCQL API</li><li>13000 - YSQL API</li><li>18018 - YB Controller</li></ul>SSH is not required after initial setup and configuration, but is recommended for subsequent troubleshooting. If you disallow SSH entirely, you must manually set up each DB node (see [Provisioning on-premises nodes](../server-nodes-software/software-on-prem-manual/)). |
-| Application | DB nodes | Open the following ports on database cluster nodes so that applications can connect via APIs.<ul><li>5433 - YSQL server</li><li>9042 - YCQL server</li></ul> |
+| DB&nbsp;nodes | DB&nbsp;nodes | Open the following ports for communication between nodes in clusters. They do not need to be exposed to your application. For universes with [Node-to-Node encryption in transit](../../security/enable-encryption-in-transit/), communication over these ports is encrypted.<ul><li>7000 - YB-Master HTTP(S)</li><li>7100 - YB-Master RPC</li><li>9000 - YB-TServer HTTP(S)</li><li>9100 - YB-TServer RPC</li><li>18018 - YB Controller RPC</li></ul> |
+| YBA&nbsp;node | DB nodes | Open the following ports on database cluster nodes so that YugabyteDB Anywhere can provision them.<ul><li>22 - SSH</li><li>5433 - YSQL server</li><li>7000/7100 - YB-Master HTTP/RPC</li><li>9000/9100 - YB-TServer HTTP/RPC</li><li>9042 - YCQL server</li><li>9070 - Node agent RPC</li><li>9300 - Prometheus Node Exporter HTTP</li><li>12000 - YCQL API</li><li>13000 - YSQL API</li><li>18018 - YB Controller RPC</li></ul>SSH is not required after initial setup and configuration, but is recommended for subsequent troubleshooting. If you disallow SSH entirely, you must manually set up each DB node (see [Provisioning on-premises nodes](../server-nodes-software/software-on-prem-manual/)). |
+| Application | DB nodes | Open the following ports on database cluster nodes so that applications can connect via APIs. For universes with [Client-to-Node encryption in transit](../../security/enable-encryption-in-transit/), communication over these ports is encrypted. Universes can also be configured with database [authorization](../../security/authorization-platform/) and [authentication](../../security/authentication/) to manage access.<ul><li>5433 - YSQL server</li><li>9042 - YCQL server</li></ul> |
 | DB nodes | YBA&nbsp;node | Open the following port on the YugabyteDB Anywhere node so that node agents can communicate.<ul><li>443 - HTTPS</li></ul> |
 | Operator | YBA&nbsp;node | Open the following ports on the YugabyteDB Anywhere node so that administrators can access the YBA UI and monitor the system and node metrics. These ports are also used by standby YBA instances in [high availability](../../administer-yugabyte-platform/high-availability/) setups.<ul><li>443 - HTTPS</li><li>9090 - Served by Prometheus, for metrics</li></ul>Port 5432 serves a local PostgreSQL instance, and is not exposed outside of localhost. |
 | DB nodes<br>YBA node | Storage | Database clusters must be allowed to contact backup storage (such as AWS S3, GCP GCS, Azure blob).<ul><li>443 - HTTPS</li></ul> |
@@ -33,6 +33,22 @@ The following ports need to be open. (The default port numbers can be customized
 
 When two database clusters are connected via [xCluster replication](../../create-deployments/async-replication-platform/), you need to ensure that the yb-master and yb-tserver RPC ports (default 7100 and 9100 respectively) are open in both directions between all nodes in both clusters.
 
+### Overriding default port assignments
+
+When [deploying a universe](../../create-deployments/create-universe-multi-zone/), you can customize the following ports:
+
+- YB-Master HTTP(S)
+- YB-Master RPC
+- YB-TServer HTTP(S)
+- YB-TServer RPC
+- YSQL server
+- YSQL API
+- YCQL server
+- YCQL API
+- Prometheus Node Exporter HTTP
+
+If you intend to customize these port numbers, replace the default port assignments with the values identifying the port that each process should use. Any value from `1024` to `65535` is valid, as long as this value does not conflict with anything else running on nodes to be provisioned.
+
 ## Provider-specific requirements
 
 <ul class="nav nav-tabs-alt nav-tabs-yb custom-tabs">

docs/content/stable/reference/configuration/operating-systems.md

@@ -22,7 +22,6 @@ Unless otherwise noted, operating systems are supported by all supported version
 | AlmaLinux 8      | {{<icon/yes>}} | {{<icon/yes>}} | Recommended for production<br>Recommended development platform<br>Default for YBA-deployed nodes |
 | AlmaLinux 9      | {{<icon/yes>}} | {{<icon/yes>}} |       |
 | Amazon Linux 2   | {{<icon/no>}}  | {{<icon/no>}}  | Supported in v2.18.0 and later<br>Deprecated in v2.20 |
-| Amazon Linux 2<br>(ARM, CIS-hardened) |  | {{<icon/yes>}} | Supported in v2.20.x. Database performance in this environment varies both due to CIS-hardening and ARM. For more information, contact {{% support-general %}}.|
 | CentOS 7          | {{<icon/no>}}  |                | Deprecated in v2.20 |
 | Oracle Linux 7   | {{<icon/no>}}  |                | Deprecated in v2.20 |
 | Oracle Linux 8   | {{<icon/yes>}} |                | |

docs/content/stable/yugabyte-platform/prepare/networking.md

@@ -12,7 +12,7 @@ menu:
 type: docs
 ---
 
-YugabyteDB Anywhere needs to be able to access nodes that will be used to create universes, and the nodes that make up universes need to be accessible to each other and to applications.
+YugabyteDB Anywhere (YBA) needs to be able to access nodes that will be used to create universes, and the nodes that make up universes need to be accessible to each other and to applications.
 
 ## Global port requirements
 
@@ -22,9 +22,9 @@ The following ports need to be open. (The default port numbers can be customized
 
 | From | To | Requirements |
 | :--- | :--- | :--- |
-| DB&nbsp;nodes | DB&nbsp;nodes | Open the following ports for communication between nodes in clusters. They do not need to be exposed to your application.<ul><li>7000 - YB-Master HTTP endpoint</li><li>7100 - YB-Master RPC</li><li>9000 - YB-TServer HTTP endpoint</li><li>9100 - YB-TServer RPC</li><li>18018 - YB Controller</li></ul> |
-| YBA&nbsp;node | DB nodes | Open the following ports on database cluster nodes so that YugabyteDB Anywhere can provision them.<ul><li>22 - SSH</li><li>5433 - YSQL server</li><li>7000/7100 - YB-Master HTTP/RPC endpoint</li><li>9000/9100 - YB-TServer HTTP/RPC endpoint</li><li>9042 - YCQL server</li><li>9070 - Node agent</li><li>9300 - Prometheus Node Exporter</li><li>12000 - YCQL API</li><li>13000 - YSQL API</li><li>18018 - YB Controller</li></ul>SSH is not required after initial setup and configuration, but is recommended for subsequent troubleshooting. If you disallow SSH entirely, you must manually set up each DB node (see [Provisioning on-premises nodes](../server-nodes-software/software-on-prem-manual/)). |
-| Application | DB nodes | Open the following ports on database cluster nodes so that applications can connect via APIs.<ul><li>5433 - YSQL server</li><li>9042 - YCQL server</li></ul> |
+| DB&nbsp;nodes | DB&nbsp;nodes | Open the following ports for communication between nodes in clusters. They do not need to be exposed to your application. For universes with [Node-to-Node encryption in transit](../../security/enable-encryption-in-transit/), communication over these ports is encrypted.<ul><li>7000 - YB-Master HTTP(S)</li><li>7100 - YB-Master RPC</li><li>9000 - YB-TServer HTTP(S)</li><li>9100 - YB-TServer RPC</li><li>18018 - YB Controller RPC</li></ul> |
+| YBA&nbsp;node | DB nodes | Open the following ports on database cluster nodes so that YugabyteDB Anywhere can provision them.<ul><li>22 - SSH</li><li>5433 - YSQL server</li><li>7000/7100 - YB-Master HTTP/RPC</li><li>9000/9100 - YB-TServer HTTP/RPC</li><li>9042 - YCQL server</li><li>9070 - Node agent RPC</li><li>9300 - Prometheus Node Exporter HTTP</li><li>12000 - YCQL API</li><li>13000 - YSQL API</li><li>18018 - YB Controller RPC</li></ul>SSH is not required after initial setup and configuration, but is recommended for subsequent troubleshooting. If you disallow SSH entirely, you must manually set up each DB node (see [Provisioning on-premises nodes](../server-nodes-software/software-on-prem-manual/)). |
+| Application | DB nodes | Open the following ports on database cluster nodes so that applications can connect via APIs. For universes with [Client-to-Node encryption in transit](../../security/enable-encryption-in-transit/), communication over these ports is encrypted. Universes can also be configured with database [authorization](../../security/authorization-platform/) and [authentication](../../security/authentication/) to manage access.<ul><li>5433 - YSQL server</li><li>9042 - YCQL server</li></ul> |
 | DB nodes | YBA&nbsp;node | Open the following port on the YugabyteDB Anywhere node so that node agents can communicate.<ul><li>443 - HTTPS</li></ul> |
 | Operator | YBA&nbsp;node | Open the following ports on the YugabyteDB Anywhere node so that administrators can access the YBA UI and monitor the system and node metrics. These ports are also used by standby YBA instances in [high availability](../../administer-yugabyte-platform/high-availability/) setups.<ul><li>443 - HTTPS</li><li>9090 - Served by Prometheus, for metrics</li></ul>Port 5432 serves a local PostgreSQL instance, and is not exposed outside of localhost. |
 | DB nodes<br>YBA node | Storage | Database clusters must be allowed to contact backup storage (such as AWS S3, GCP GCS, Azure blob).<ul><li>443 - HTTPS</li></ul> |
@@ -33,6 +33,22 @@ The following ports need to be open. (The default port numbers can be customized
 
 When two database clusters are connected via [xCluster replication](../../create-deployments/async-replication-platform/), you need to ensure that the yb-master and yb-tserver RPC ports (default 7100 and 9100 respectively) are open in both directions between all nodes in both clusters.
 
+### Overriding default port assignments
+
+When [deploying a universe](../../create-deployments/create-universe-multi-zone/), you can customize the following ports:
+
+- YB-Master HTTP(S)
+- YB-Master RPC
+- YB-TServer HTTP(S)
+- YB-TServer RPC
+- YSQL server
+- YSQL API
+- YCQL server
+- YCQL API
+- Prometheus Node Exporter HTTP
+
+If you intend to customize these port numbers, replace the default port assignments with the values identifying the port that each process should use. Any value from `1024` to `65535` is valid, as long as this value does not conflict with anything else running on nodes to be provisioned.
+
 ## Provider-specific requirements
 
 <ul class="nav nav-tabs-alt nav-tabs-yb custom-tabs">

docs/content/v2.20/reference/configuration/operating-systems.md

@@ -22,7 +22,7 @@ Unless otherwise noted, operating systems are supported by all supported version
 | AlmaLinux 8      | {{<icon/yes>}} | {{<icon/yes>}} | Recommended for production<br>Recommended development platform<br>Default for YBA-deployed nodes |
 | AlmaLinux 9      | {{<icon/yes>}} | {{<icon/yes>}} |       |
 | Amazon Linux 2   | {{<icon/partial>}} | {{<icon/partial>}} |Supported in v2.18.0 and later<br>Deprecated in v2.20; not supported in v2.21 and [upcoming release series](/preview/releases/#upcoming-release-series) |
-| Amazon Linux 2<br>(ARM, CIS-hardened) |  | {{<icon/yes>}} | Supported in v2.20.x. Database performance in this environment varies both due to CIS-hardening and ARM. For more information, contact {{% support-general %}}.|
+| Amazon Linux 2<br>(ARM, CIS-hardened) |  | {{<icon/yes>}} | Supported only in v2.20.x. Database performance in this environment varies both due to CIS-hardening and ARM. For more information, contact {{% support-general %}}.|
 | CentOS 7          | {{<icon/partial>}} |                | Deprecated in v2.20; not supported in v2.21 and [upcoming release series](/preview/releases/#upcoming-release-series) |
 | Oracle 7         | {{<icon/yes>}} |                | |
 | Oracle 8         | {{<icon/yes>}} |                | |

java/yb-pgsql/src/test/java/org/yb/pgsql/TestPgFollowerReads.java

@@ -57,27 +57,19 @@ private Long getCountForTable(String metricName, String tableName) throws Except
   @Test
   public void testSetIsolationLevelsWithReadFromFollowersSessionVariable() throws Exception {
     try (Statement statement = connection.createStatement()) {
-      // If follower reads are disabled, we should be allowed to set any staleness.
       // Enabling follower reads should fail if staleness is less than 2 * max_clock_skew.
       statement.execute("SET yb_read_from_followers = false");
-      statement.execute("SET yb_follower_read_staleness_ms = " + (2 * kMaxClockSkewMs - 1));
-      runInvalidQuery(statement, "SET yb_read_from_followers = true",
+      runInvalidQuery(statement, "SET yb_follower_read_staleness_ms = " + (2 * kMaxClockSkewMs - 1),
                       "ERROR: cannot enable yb_read_from_followers with a staleness of less than "
                       + "2 * (max_clock_skew");
-      statement.execute("SET yb_follower_read_staleness_ms = " + kMaxClockSkewMs / 2);
-      runInvalidQuery(statement, "SET yb_read_from_followers = true",
+      runInvalidQuery(statement, "SET yb_follower_read_staleness_ms = " + kMaxClockSkewMs / 2,
                       "ERROR: cannot enable yb_read_from_followers with a staleness of less than "
                       + "2 * (max_clock_skew");
-      statement.execute("SET yb_follower_read_staleness_ms = " + 0);
-      runInvalidQuery(statement, "SET yb_read_from_followers = true",
+      runInvalidQuery(statement, "SET yb_follower_read_staleness_ms = " + 0,
                       "ERROR: cannot enable yb_read_from_followers with a staleness of less than "
                       + "2 * (max_clock_skew");
-
       statement.execute("SET yb_follower_read_staleness_ms = " + (2 * kMaxClockSkewMs + 1));
-      statement.execute("SET yb_read_from_followers = true");
 
-      // If follower reads are enabled, we should be allowed to set staleness to any value over
-      // 2 * max_clock_skew, which is 500ms. Any value smaller than that should fail.
       statement.execute("SET yb_read_from_followers = true");
       statement.execute("SET yb_follower_read_staleness_ms = " + 2 * kMaxClockSkewMs);
       runInvalidQuery(statement, "SET yb_follower_read_staleness_ms = " + (2 * kMaxClockSkewMs - 1),

java/yb-yugabyted/pom.xml

@@ -59,5 +59,11 @@
       <type>test-jar</type>
       <scope>test</scope>
     </dependency>
+    <dependency>
+      <groupId>org.json</groupId>
+      <artifactId>json</artifactId>
+      <version>20180130</version>
+      <scope>test</scope>
+    </dependency>
   </dependencies>
 </project>

java/yb-yugabyted/src/test/java/org/yb/minicluster/BaseMiniYugabytedClusterTest.java

@@ -31,6 +31,10 @@ public class BaseMiniYugabytedClusterTest extends BaseYBTest {
      */
     protected static MiniYugabytedCluster miniYugabytedCluster;
 
+    protected static MiniYugabytedClusterParameters clusterParameters;
+
+    protected static List<MiniYugabytedNodeConfigurations> clusterConfigurations;
+
     /**
      * Comma separate describing the master addresses and ports.
      */
@@ -60,7 +64,7 @@ public void setUpBefore() throws Exception {
             clusterNeedsRecreation = false;
         }
         if (miniYugabytedCluster == null) {
-            createMiniCluster(1);
+            createMiniCluster(clusterParameters.numNodes);
         } else if (shouldRestartMiniClusterBetweenTests()) {
             LOG.info("Restarting the MiniCluster");
             miniYugabytedCluster.restart();
@@ -79,18 +83,16 @@ protected final void createMiniYugabytedCluster(int numNodes) throws Exception {
 
         final int replicationFactor = getReplicationFactor();
 
-        MiniYugabytedClusterParameters clusterParameters = new MiniYugabytedClusterParameters();
-
         // miniYugabytedCluster = new MiniYugabytedCluster(clusterParameters,
         // getMasterFlags(), getMasterFlags(), getTServerFlags(),
         // null, getMasterFlags(), clientHost, certFile,
         // clientCertFile, clientKeyFile);
 
         // TO-DO: replace all nulls with the actual fields.
-        miniYugabytedCluster = new MiniYugabytedCluster(clusterParameters,
-                null, null, null,
-                null, null, null, null,
-                null, null, null);
+        miniYugabytedCluster = new MiniYugabytedCluster(
+            clusterParameters,
+            clusterConfigurations
+        );
 
         masterAddresses = miniYugabytedCluster.getMasterAddresses();
         masterHostPorts = miniYugabytedCluster.getMasterHostPorts();