From 6fbcf10f0ebf2ecb598e1c77e3594887f924d673 Mon Sep 17 00:00:00 2001 From: Jesse Seldess Date: Wed, 3 Oct 2018 15:21:17 -0400 Subject: [PATCH 1/3] Document replication zones for secondary indexes Fixes #3146. --- v2.1/architecture/distribution-layer.md | 9 +- v2.1/configure-replication-zones.md | 131 +++++++++++++++++++----- v2.1/demo-automatic-cloud-migration.md | 2 +- 3 files changed, 111 insertions(+), 31 deletions(-) diff --git a/v2.1/architecture/distribution-layer.md b/v2.1/architecture/distribution-layer.md index b8d37628629..5eb2c10d267 100644 --- a/v2.1/architecture/distribution-layer.md +++ b/v2.1/architecture/distribution-layer.md @@ -10,10 +10,9 @@ The distribution layer of CockroachDB's architecture provides a unified view of If you haven't already, we recommend reading the [Architecture Overview](overview.html). {{site.data.alerts.end}} - ## Overview -To make all data in your cluster accessible from any node, CockroachDB stores data in a monolithic sorted map of key-value pairs. This keyspace describes all of the data in your cluster, as well as its location, and is divided into what we call "ranges", contiguous chunks of the keyspace, so that every key can always be found in a single range. +To make all data in your cluster accessible from any node, CockroachDB stores data in a monolithic sorted map of key-value pairs. This key-space describes all of the data in your cluster, as well as its location, and is divided into what we call "ranges", contiguous chunks of the key-space, so that every key can always be found in a single range. CockroachDB implements a sorted map to enable: @@ -41,9 +40,11 @@ Each node caches values of the `meta2` range it has accessed before, which optim After the node's meta ranges is the KV data your cluster stores. -This data is broken up into 64MiB sections of contiguous key-space known as ranges. This size represents a sweet spot for us between a size that's small enough to move quickly between nodes, but large enough to store a meaningfully contiguous set of data whose keys are more likely to be accessed together. These ranges are then shuffled around your cluster to ensure survivability. +Each table and its secondary indexes initially map to a single range, where each key-value pair in the range represents a single row in the table (also called the primary index because the table is sorted by the primary key) or a single row in a secondary index. As soon as a range reaches 64 MiB in size, it splits into two ranges. This process continues as a table and its indexes continue growing. Once a table is split across multiple ranges, it's likely that the table and secondary indexes will be stored in separate ranges. However, a range can still contain data for both the table and a secondary index. + +The default 64MiB range size represents a sweet spot for us between a size that's small enough to move quickly between nodes, but large enough to store a meaningfully contiguous set of data whose keys are more likely to be accessed together. These ranges are then shuffled around your cluster to ensure survivability. -These ranges are replicated (in the aptly named replication layer), and have the addresses of each replica stored in the `meta2` range. +These table ranges are replicated (in the aptly named replication layer), and have the addresses of each replica stored in the `meta2` range. ### Using the monolithic sorted map diff --git a/v2.1/configure-replication-zones.md b/v2.1/configure-replication-zones.md index 19cb77a0520..b9cf9a55519 100644 --- a/v2.1/configure-replication-zones.md +++ b/v2.1/configure-replication-zones.md @@ -5,44 +5,52 @@ keywords: ttl, time to live, availability zone toc: true --- -In CockroachDB, you use **replication zones** to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. Initially, there are some special pre-configured replication zones for internal system data along with a default replication zone that applies to the rest of the cluster. You can adjust these pre-configured zones as well as add zones for individual databases, tables, and rows ([enterprise-only](enterprise-licensing.html)) as needed. For example, you might use the default zone to replicate most data in a cluster normally within a single datacenter, while creating a specific zone to more highly replicate a certain database or table across multiple datacenters and geographies. +In CockroachDB, you use **replication zones** to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. Initially, there are some special pre-configured replication zones for internal system data along with a default replication zone that applies to the rest of the cluster. You can adjust these pre-configured zones as well as add zones for individual databases, tables and secondary indexes, and rows ([enterprise-only](enterprise-licensing.html)) as needed. For example, you might use the default zone to replicate most data in a cluster normally within a single datacenter, while creating a specific zone to more highly replicate a certain database or table across multiple datacenters and geographies. This page explains how replication zones work and how to use the `cockroach zone` [command](cockroach-commands.html) to configure them. -{{site.data.alerts.callout_info}}Currently, only the root user can configure replication zones.{{site.data.alerts.end}} +{{site.data.alerts.callout_info}} +Currently, only the `root` user can configure replication zones. +{{site.data.alerts.end}} -## Overview +## Replication zone levels -### Replication zone levels +### For table data -There are four replication zone levels for **table data** in a cluster, listed from least to most granular: +There are five replication zone levels for [**table data**](architecture/distribution-layer.html#table-data) in a cluster, listed from least to most granular: Level | Description ------|------------ Cluster | CockroachDB comes with a pre-configured `.default` replication zone that applies to all table data in the cluster not constrained by a database, table, or row-specific replication zone. This zone can be adjusted but not removed. See [View the Default Replication Zone](#view-the-default-replication-zone) and [Edit the Default Replication Zone](#edit-the-default-replication-zone) for more details. Database | You can add replication zones for specific databases. See [Create a Replication Zone for a Database](#create-a-replication-zone-for-a-database) for more details. -Table | You can add replication zones for specific tables. See [Create a Replication Zone for a Table](#create-a-replication-zone-for-a-table) for more details.

CockroachDB comes with a pre-configured replication zone for one internal table, `system.jobs`, which stores metadata about long-running jobs such as schema changes and backups. Historical queries are never run against this table and the rows in it are updated frequently, so the pre-configured zone gives this table a lower-than-default `ttlseconds`. -Row ([Enterprise-only](enterprise-licensing.html)) | You can add replication zones for specific rows in a table by [defining table partitions](partitioning.html). See [Create a Replication Zone for a Table Partition](#create-a-replication-zone-for-a-table-partition) for more details. +Table | You can add replication zones for specific tables. See [Create a Replication Zone for a Table](#create-a-replication-zone-for-a-table). +Index ([Enterprise-only](enterprise-licensing.html)) | The [secondary indexes](indexes.html) on a table will automatically use the replication zone for the table. However, with an enterprise license, you can add distinct replication zones for secondary indexes. See [Create a Replication Zone for a Secondary Index](#create-a-replication-zone-for-a-secondary-index) for more details. +Row ([Enterprise-only](enterprise-licensing.html)) | You can add replication zones for specific rows in a table or secondary index by [defining table partitions](partitioning.html). See [Create a Replication Zone for a Table Partition](#create-a-replication-zone-for-a-table-partition-new-in-v2-0) for more details. + +### For system data -In addition, CockroachDB stores internal **system data** in what are called system ranges. There are two replication zone levels for this internal system data, listed from least to most granular: +In addition, CockroachDB stores internal [**system data**](architecture/distribution-layer.html#monolithic-sorted-map-structure) in what are called system ranges. There are two replication zone levels for this internal system data, listed from least to most granular: Level | Description ------|------------ Cluster | The `.default` replication zone mentioned above also applies to all system ranges not constrained by a more specific replication zone. -System Range | CockroachDB comes with pre-configured replication zones for the "meta" and "liveness" system ranges. If necessary, you can add replication zones for the "timeseries" range and other "system" ranges as well. See [Create a Replication Zone for a System Range](#create-a-replication-zone-for-a-system-range) for more details. +System Range | CockroachDB comes with pre-configured replication zones for the "meta" and "liveness" system ranges. If necessary, you can add replication zones for the "timeseries" range and other "system" ranges as well. See [Create a Replication Zone for a System Range](#create-a-replication-zone-for-a-system-range) for more details.

CockroachDB also comes with a pre-configured replication zone for one internal table, `system.jobs`, which stores metadata about long-running jobs such as schema changes and backups. Historical queries are never run against this table and the rows in it are updated frequently, so the pre-configured zone gives this table a lower-than-default `ttlseconds`. + +### Level priorities When replicating data, whether table or system, CockroachDB always uses the most granular replication zone available. For example, for a piece of user data: 1. If there's a replication zone for the row, CockroachDB uses it. -2. If there's no applicable row replication zone, CockroachDB uses the table replication zone. -3. If there's no applicable table replication zone, CockroachDB uses the database replication zone. -4. If there's no applicable database replication zone, CockroachDB uses the `.default` cluster-wide replication zone. +2. If there's no applicable row replication zone and the row is from a secondary index, CockroachDB uses the secondary index replication zone. +3. If the row isn't from a secondary index or there is no applicable secondary index replication zone, CockroachDB uses the table replication zone. +4. If there's no applicable table replication zone, CockroachDB uses the database replication zone. +5. If there's no applicable database replication zone, CockroachDB uses the `.default` cluster-wide replication zone. {{site.data.alerts.callout_danger}} {% include {{page.version.version}}/known-limitations/system-range-replication.md %} {{site.data.alerts.end}} -### Replication zone format +## Replication zone format A replication zone is specified in [YAML](https://en.wikipedia.org/wiki/YAML) format and looks like this: @@ -63,13 +71,13 @@ Field | Description `num_replicas` | The number of replicas in the zone.

**Default:** `3` `constraints` | A JSON object or array of required and/or prohibited constraints influencing the location of replicas. See [Types of Constraints](#types-of-constraints) and [Scope of Constraints](#scope-of-constraints) for more details.

To prevent hard-to-detect typos, constraints placed on [store attributes and node localities](#descriptive-attributes-assigned-to-nodes) must match the values passed to at least one node in the cluster. If not, an error is signalled.

**Default:** No constraints, with CockroachDB locating each replica on a unique node and attempting to spread replicas evenly across localities. -### Replication constraints +## Replication constraints The location of replicas, both when they are first added and when they are rebalanced to maintain cluster equilibrium, is based on the interplay between descriptive attributes assigned to nodes and constraints set in zone configurations. {{site.data.alerts.callout_success}}For demonstrations of how to set node attributes and replication constraints in different scenarios, see Scenario-based Examples below.{{site.data.alerts.end}} -#### Descriptive attributes assigned to nodes +### Descriptive attributes assigned to nodes When starting a node with the [`cockroach start`](start-a-node.html) command, you can assign the following types of descriptive attributes: @@ -79,7 +87,7 @@ Attribute Type | Description **Node Capability** | Using the `--attrs` flag, you can specify node capability, which might include specialized hardware or number of cores, for example:

`--attrs=ram:64gb` **Store Type/Capability** | Using the `attrs` field of the `--store` flag, you can specify disk type or capability, for example:

`--store=path=/mnt/ssd01,attrs=ssd`
`--store=path=/mnt/hda1,attrs=hdd:7200rpm` -#### Types of constraints +### Types of constraints The node-level and store-level descriptive attributes mentioned above can be used as the following types of constraints in replication zones to influence the location of replicas. However, note the following general guidance: @@ -91,7 +99,7 @@ Constraint Type | Description | Syntax **Required** | When placing replicas, the cluster will consider only nodes/stores with matching attributes or localities. When there are no matching nodes/stores, new replicas will not be added. | `+ssd` **Prohibited** | When placing replicas, the cluster will ignore nodes/stores with matching attributes or localities. When there are no alternate nodes/stores, new replicas will not be added. | `-ssd` -#### Scope of constraints +### Scope of constraints Constraints can be specified such that they apply to all replicas in a zone or such that different constraints apply to different replicas, meaning you can effectively pick the exact location of each replica. @@ -100,7 +108,7 @@ Constraint Scope | Description | Syntax **All Replicas** | Constraints specified using JSON array syntax apply to all replicas in every range that's part of the replication zone. | `constraints: [+ssd, -region=west]` **Per-Replica** | Multiple lists of constraints can be provided in a JSON object, mapping each list of constraints to an integer number of replicas in each range that the constraints should apply to.

The total number of replicas constrained cannot be greater than the total number of replicas for the zone (`num_replicas`). However, if the total number of replicas constrained is less than the total number of replicas for the zone, the non-constrained replicas will be allowed on any nodes/stores. | `constraints: {"+ssd,-region=west": 2, "+region=east": 1}` -### Node/replica recommendations +## Node/replica recommendations See [Cluster Topography](recommended-production-settings.html#cluster-topology) recommendations for production deployments. @@ -128,6 +136,12 @@ $ cockroach zone get # View the replication zone for a table: $ cockroach zone get +# View the replication zone for an index: +$ cockroach zone get + +# View the replication zone for a table or index partition: +$ cockroach zone get + # Edit the default replication zone for the cluster: $ cockroach zone set .default --file= @@ -137,7 +151,10 @@ $ cockroach zone set --file= # Create/edit the replication zone for a table: $ cockroach zone set --file= -# Create/edit the replication zone for a table partition: +# Create/edit the replication zone for an index: +$ cockroach zone set --file= + +# Create/edit the replication zone for a table or index partition: $ cockroach zone set --file= # Remove the replication zone for a database: @@ -146,6 +163,12 @@ $ cockroach zone rm # Remove the replication zone for a table: $ cockroach zone rm +# Remove the replication zone for an index: +$ cockroach zone rm + +# Remove the replication zone for a table or index partition: +$ cockroach zone set --file= + # View help: $ cockroach zone --help $ cockroach zone ls --help @@ -344,15 +367,67 @@ $ echo 'num_replicas: 7' | cockroach zone set db1.t1 \ -f - ~~~ -### Create a replication zone for a table partition +### Create a replication zone for a secondary index + +{{site.data.alerts.callout_info}} +This is an [enterprise-only](enterprise-licensing.html) feature. +{{site.data.alerts.end}} + +The [secondary indexes](indexes.html) on a table will automatically use the replication zone for the table. However, with an enterprise license, you can add distinct replication zones for secondary indexes. + +To control replication for a specific secondary index, create a YAML file defining only the values you want to change (other values will not be affected), and use the `cockroach zone set -f ` command with appropriate flags: -{{site.data.alerts.callout_info}}This is an enterprise-only feature.{{site.data.alerts.end}} +{{site.data.alerts.callout_success}} +To get the name of a secondary index, which you need for the `cockroach zone set` command, use the [`SHOW INDEX`](show-index.html) or [`SHOW CREATE TABLE`](show-create.html) statements. +{{site.data.alerts.end}} + +{% include copy-clipboard.html %} +~~~ shell +$ cat index_zone.yaml +~~~ + +~~~ +num_replicas: 7 +~~~ + +{% include copy-clipboard.html %} +~~~ shell +$ cockroach zone set db1.table@idx1 \ +--insecure \ +--host= \ +-f index_zone.yaml +~~~ + +~~~ +range_min_bytes: 1048576 +range_max_bytes: 67108864 +gc: + ttlseconds: 86400 +num_replicas: 7 +constraints: [] +~~~ + +Alternately, you can pass the YAML content via the standard input: + +{% include copy-clipboard.html %} +~~~ shell +$ echo 'num_replicas: 7' | cockroach zone set db1.table@idx1 \ +--insecure \ +--host= \ +-f - +~~~ + +### Create a replication zone for a table or secondary index partition + +{{site.data.alerts.callout_info}} +This is an [enterprise-only](enterprise-licensing.html) feature. +{{site.data.alerts.end}} To [control replication for table partitions](partitioning.html#replication-zones), create a YAML file defining only the values you want to change (other values will not be affected), and use the `cockroach zone set -f ` command with appropriate flags: {% include copy-clipboard.html %} ~~~ shell -$ cat > australia.zone.yml +$ cat > australia_zone.yml ~~~ ~~~ shell @@ -366,9 +441,13 @@ Apply zone configurations to corresponding partitions: $ cockroach zone set roachlearn.students_by_list.australia \ --insecure \ --host= \ --f australia.zone.yml +-f australia_zone.yml ~~~ +{{site.data.alerts.callout_success}} +Since the syntax is the same for defining a replication zone for a table or index partition (`database.table.partition`), give partitions names that communicate what they are partitioning, e.g., `australia_table` vs `australia_idx1`. +{{site.data.alerts.end}} + ### Create a replication zone for a system range In addition to the databases and tables that are visible via the SQL interface, CockroachDB stores internal data in what are called system ranges. CockroachDB comes with pre-configured replication zones for some of these ranges: @@ -644,13 +723,13 @@ There's no need to make zone configuration changes; by default, the cluster is c The required constraint will force application 2's data to be replicated only within the `us-2` datacenter. -### Stricter replication for a specific table +### Stricter replication for a table and its secondary indexes **Scenario:** - You have 7 nodes, 5 with SSD drives and 2 with HDD drives. - You want data replicated 3 times by default. -- Speed and availability are important for a specific table that is queried very frequently, however, so you want the data in that table to be replicated 5 times, preferably on nodes with SSD drives. +- Speed and availability are important for a specific table and its indexes, which are queried very frequently, however, so you want the data in the table and secondary indexes to be replicated 5 times, preferably on nodes with SSD drives. **Approach:** @@ -711,7 +790,7 @@ There's no need to make zone configuration changes; by default, the cluster is c constraints: [+ssd] ~~~ - Data in the table will be replicated 5 times, and the required constraint will place data in the table on nodes with `ssd` drives. + The secondary indexes on the table will use the table's replication zone, so all data for the table will be replicated 5 times, and the required constraint will place the data on nodes with `ssd` drives. ### Tweaking the replication of system ranges diff --git a/v2.1/demo-automatic-cloud-migration.md b/v2.1/demo-automatic-cloud-migration.md index e646e78161f..1e60389871c 100644 --- a/v2.1/demo-automatic-cloud-migration.md +++ b/v2.1/demo-automatic-cloud-migration.md @@ -249,5 +249,5 @@ You may also want to learn other ways to control the location and number of repl - [Even Replication Across Datacenters](configure-replication-zones.html#even-replication-across-datacenters) - [Multiple Applications Writing to Different Databases](configure-replication-zones.html#multiple-applications-writing-to-different-databases) -- [Stricter Replication for a Specific Table](configure-replication-zones.html#stricter-replication-for-a-specific-table) +- [Stricter Replication for a Table and Its Indexes](configure-replication-zones.html#stricter-replication-for-a-table-and-its-secondary-indexes) - [Tweaking the Replication of System Ranges](configure-replication-zones.html#tweaking-the-replication-of-system-ranges) From bc6adfde187298c5c36992d386aa4f4ba515545d Mon Sep 17 00:00:00 2001 From: Jesse Seldess Date: Thu, 4 Oct 2018 16:52:25 -0400 Subject: [PATCH 2/3] Apply changes to 2.0 --- v2.0/architecture/distribution-layer.md | 4 +- v2.0/configure-replication-zones.md | 128 +++++++++++++++++++----- 2 files changed, 108 insertions(+), 24 deletions(-) diff --git a/v2.0/architecture/distribution-layer.md b/v2.0/architecture/distribution-layer.md index ff64a0723e1..91f4a4ca766 100644 --- a/v2.0/architecture/distribution-layer.md +++ b/v2.0/architecture/distribution-layer.md @@ -39,7 +39,9 @@ Each node caches values of the `meta2` range it has accessed before, which optim After the node's meta ranges is the KV data your cluster stores. -This data is broken up into 64MiB sections of contiguous key-space known as ranges. This size represents a sweet spot for us between a size that's small enough to move quickly between nodes, but large enough to store a meaningfully contiguous set of data whose keys are more likely to be accessed together. These ranges are then shuffled around your cluster to ensure survivability. +Each table and its secondary indexes initially map to a single range, where each key-value pair in the range represents a single row in the table (also called the primary index because the table is sorted by the primary key) or a single row in a secondary index. As soon as a range reaches 64 MiB in size, it splits into two ranges. This process continues as a table and its indexes continue growing. Once a table is split across multiple ranges, it's likely that the table and secondary indexes will be stored in separate ranges. However, a range can still contain data for both the table and a secondary index. + +The default 64MiB range size represents a sweet spot for us between a size that's small enough to move quickly between nodes, but large enough to store a meaningfully contiguous set of data whose keys are more likely to be accessed together. These ranges are then shuffled around your cluster to ensure survivability. These ranges are replicated (in the aptly named Replication Layer), and have the addresses of each replica stored in the `meta2` range. diff --git a/v2.0/configure-replication-zones.md b/v2.0/configure-replication-zones.md index 02c8677b183..40635adc35d 100644 --- a/v2.0/configure-replication-zones.md +++ b/v2.0/configure-replication-zones.md @@ -5,44 +5,52 @@ keywords: ttl, time to live, availability zone toc: true --- -In CockroachDB, you use **replication zones** to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. Initially, there are some special pre-configured replication zones for internal system data along with a default replication zone that applies to the rest of the cluster. You can adjust these pre-configured zones as well as add zones for individual databases, tables, and rows ([enterprise-only](enterprise-licensing.html)) as needed. For example, you might use the default zone to replicate most data in a cluster normally within a single datacenter, while creating a specific zone to more highly replicate a certain database or table across multiple datacenters and geographies. +In CockroachDB, you use **replication zones** to control the number and location of replicas for specific sets of data, both when replicas are first added and when they are rebalanced to maintain cluster equilibrium. Initially, there are some special pre-configured replication zones for internal system data along with a default replication zone that applies to the rest of the cluster. You can adjust these pre-configured zones as well as add zones for individual databases, tables and secondary indexes, and rows ([enterprise-only](enterprise-licensing.html)) as needed. For example, you might use the default zone to replicate most data in a cluster normally within a single datacenter, while creating a specific zone to more highly replicate a certain database or table across multiple datacenters and geographies. This page explains how replication zones work and how to use the `cockroach zone` [command](cockroach-commands.html) to configure them. -{{site.data.alerts.callout_info}}Currently, only the root user can configure replication zones.{{site.data.alerts.end}} +{{site.data.alerts.callout_info}} +Currently, only the `root` user can configure replication zones. +{{site.data.alerts.end}} -## Overview +## Replication zone levels -### Replication Zones Levels +### For table data -There are four replication zone levels for **table data** in a cluster, listed from least to most granular: +There are five replication zone levels for [**table data**](architecture/distribution-layer.html#table-data) in a cluster, listed from least to most granular: Level | Description ------|------------ Cluster | CockroachDB comes with a pre-configured `.default` replication zone that applies to all table data in the cluster not constrained by a database, table, or row-specific replication zone. This zone can be adjusted but not removed. See [View the Default Replication Zone](#view-the-default-replication-zone) and [Edit the Default Replication Zone](#edit-the-default-replication-zone) for more details. Database | You can add replication zones for specific databases. See [Create a Replication Zone for a Database](#create-a-replication-zone-for-a-database) for more details. -Table | You can add replication zones for specific tables. See [Create a Replication Zone for a Table](#create-a-replication-zone-for-a-table) for more details.

CockroachDB comes with a pre-configured replication zone for one internal table, `system.jobs`, which stores metadata about long-running jobs such as schema changes and backups. Historical queries are never run against this table and the rows in it are updated frequently, so the pre-configured zone gives this table a lower-than-default `ttlseconds`. -Row ([Enterprise-only](enterprise-licensing.html)) | You can add replication zones for specific rows in a table by [defining table partitions](partitioning.html). See [Create a Replication Zone for a Table Partition](#create-a-replication-zone-for-a-table-partition-new-in-v2-0) for more details. +Table | You can add replication zones for specific tables. See [Create a Replication Zone for a Table](#create-a-replication-zone-for-a-table). +Index ([Enterprise-only](enterprise-licensing.html)) | The [secondary indexes](indexes.html) on a table will automatically use the replication zone for the table. However, with an enterprise license, you can add distinct replication zones for secondary indexes. See [Create a Replication Zone for a Secondary Index](#create-a-replication-zone-for-a-secondary-index) for more details. +Row ([Enterprise-only](enterprise-licensing.html)) | You can add replication zones for specific rows in a table or secondary index by [defining table partitions](partitioning.html). See [Create a Replication Zone for a Table Partition](#create-a-replication-zone-for-a-table-partition-new-in-v2-0) for more details. + +### For system data -In addition, CockroachDB stores internal **system data** in what are called system ranges. There are two replication zone levels for this internal system data, listed from least to most granular: +In addition, CockroachDB stores internal [**system data**](architecture/distribution-layer.html#monolithic-sorted-map-structure) in what are called system ranges. There are two replication zone levels for this internal system data, listed from least to most granular: Level | Description ------|------------ Cluster | The `.default` replication zone mentioned above also applies to all system ranges not constrained by a more specific replication zone. -System Range | CockroachDB comes with pre-configured replication zones for the "meta" and "liveness" system ranges. If necessary, you can add replication zones for the "timeseries" range and other "system" ranges as well. See [Create a Replication Zone for a System Range](#create-a-replication-zone-for-a-system-range) for more details. +System Range | CockroachDB comes with pre-configured replication zones for the "meta" and "liveness" system ranges. If necessary, you can add replication zones for the "timeseries" range and other "system" ranges as well. See [Create a Replication Zone for a System Range](#create-a-replication-zone-for-a-system-range) for more details.

CockroachDB also comes with a pre-configured replication zone for one internal table, `system.jobs`, which stores metadata about long-running jobs such as schema changes and backups. Historical queries are never run against this table and the rows in it are updated frequently, so the pre-configured zone gives this table a lower-than-default `ttlseconds`. + +### Level priorities When replicating data, whether table or system, CockroachDB always uses the most granular replication zone available. For example, for a piece of user data: 1. If there's a replication zone for the row, CockroachDB uses it. -2. If there's no applicable row replication zone, CockroachDB uses the table replication zone. -3. If there's no applicable table replication zone, CockroachDB uses the database replication zone. -4. If there's no applicable database replication zone, CockroachDB uses the `.default` cluster-wide replication zone. +2. If there's no applicable row replication zone and the row is from a secondary index, CockroachDB uses the secondary index replication zone. +3. If the row isn't from a secondary index or there is no applicable secondary index replication zone, CockroachDB uses the table replication zone. +4. If there's no applicable table replication zone, CockroachDB uses the database replication zone. +5. If there's no applicable database replication zone, CockroachDB uses the `.default` cluster-wide replication zone. {{site.data.alerts.callout_danger}} {% include {{page.version.version}}/known-limitations/system-range-replication.md %} {{site.data.alerts.end}} -### Replication Zone Format +## Replication Zone Format A replication zone is specified in [YAML](https://en.wikipedia.org/wiki/YAML) format and looks like this: @@ -63,13 +71,13 @@ Field | Description `num_replicas` | The number of replicas in the zone.

**Default:** `3` `constraints` | A JSON object or array of required and/or prohibited constraints influencing the location of replicas. See [Types of Constraints](#types-of-constraints) and [Scope of Constraints](#scope-of-constraints) for more details.

**Default:** No constraints, with CockroachDB locating each replica on a unique node and attempting to spread replicas evenly across localities. -### Replication Constraints +## Replication Constraints The location of replicas, both when they are first added and when they are rebalanced to maintain cluster equilibrium, is based on the interplay between descriptive attributes assigned to nodes and constraints set in zone configurations. {{site.data.alerts.callout_success}}For demonstrations of how to set node attributes and replication constraints in different scenarios, see Scenario-based Examples below.{{site.data.alerts.end}} -#### Descriptive Attributes Assigned to Nodes +### Descriptive Attributes Assigned to Nodes When starting a node with the [`cockroach start`](start-a-node.html) command, you can assign the following types of descriptive attributes: @@ -79,7 +87,7 @@ Attribute Type | Description **Node Capability** | Using the `--attrs` flag, you can specify node capability, which might include specialized hardware or number of cores, for example:

`--attrs=ram:64gb` **Store Type/Capability** | Using the `attrs` field of the `--store` flag, you can specify disk type or capability, for example:

`--store=path=/mnt/ssd01,attrs=ssd`
`--store=path=/mnt/hda1,attrs=hdd:7200rpm` -#### Types of Constraints +### Types of Constraints The node-level and store-level descriptive attributes mentioned above can be used as the following types of constraints in replication zones to influence the location of replicas. However, note the following general guidance: @@ -91,7 +99,7 @@ Constraint Type | Description | Syntax **Required** | When placing replicas, the cluster will consider only nodes/stores with matching attributes or localities. When there are no matching nodes/stores, new replicas will not be added. | `+ssd` **Prohibited** | When placing replicas, the cluster will ignore nodes/stores with matching attributes or localities. When there are no alternate nodes/stores, new replicas will not be added. | `-ssd` -#### Scope of Constraints +### Scope of Constraints Constraints can be specified such that they apply to all replicas in a zone or such that different constraints apply to different replicas, meaning you can effectively pick the exact location of each replica. @@ -100,7 +108,7 @@ Constraint Scope | Description | Syntax **All Replicas** | Constraints specified using JSON array syntax apply to all replicas in every range that's part of the replication zone. | `constraints: [+ssd, -region=west]` **Per-Replica** | Multiple lists of constraints can be provided in a JSON object, mapping each list of constraints to an integer number of replicas in each range that the constraints should apply to.

The total number of replicas constrained cannot be greater than the total number of replicas for the zone (`num_replicas`). However, if the total number of replicas constrained is less than the total number of replicas for the zone, the non-constrained replicas will be allowed on any nodes/stores. | `constraints: {"+ssd,-region=west": 2, "+region=east": 1}` -### Node/Replica Recommendations +## Node/Replica Recommendations See [Cluster Topography](recommended-production-settings.html#cluster-topology) recommendations for production deployments. @@ -128,6 +136,12 @@ $ cockroach zone get # View the replication zone for a table: $ cockroach zone get +# View the replication zone for an index: +$ cockroach zone get + +# View the replication zone for a table or index partition: +$ cockroach zone get + # Edit the default replication zone for the cluster: $ cockroach zone set .default --file= @@ -137,7 +151,10 @@ $ cockroach zone set --file= # Create/edit the replication zone for a table: $ cockroach zone set --file= -# Create/edit the replication zone for a table partition: +# Create/edit the replication zone for an index: +$ cockroach zone set --file= + +# Create/edit the replication zone for a table or index partition: $ cockroach zone set --file= # Remove the replication zone for a database: @@ -146,6 +163,12 @@ $ cockroach zone rm # Remove the replication zone for a table: $ cockroach zone rm +# Remove the replication zone for an index: +$ cockroach zone rm + +# Remove the replication zone for a table or index partition: +$ cockroach zone set --file= + # View help: $ cockroach zone --help $ cockroach zone ls --help @@ -327,15 +350,67 @@ Alternately, you can pass the YAML content via the standard input: $ echo 'num_replicas: 7' | cockroach zone set db1.t1 --insecure -f - ~~~ -### Create a Replication Zone for a Table Partition New in v2.0 +### Create a Replication Zone for a Secondary Index + +{{site.data.alerts.callout_info}} +This is an [enterprise-only](enterprise-licensing.html) feature. +{{site.data.alerts.end}} + +The [secondary indexes](indexes.html) on a table will automatically use the replication zone for the table. However, with an enterprise license, you can add distinct replication zones for secondary indexes. + +To control replication for a specific secondary index, create a YAML file defining only the values you want to change (other values will not be affected), and use the `cockroach zone set -f ` command with appropriate flags: + +{{site.data.alerts.callout_success}} +To get the name of a secondary index, which you need for the `cockroach zone set` command, use the [`SHOW INDEX`](show-index.html) or [`SHOW CREATE TABLE`](show-create.html) statements. +{{site.data.alerts.end}} + +{% include copy-clipboard.html %} +~~~ shell +$ cat index_zone.yaml +~~~ + +~~~ +num_replicas: 7 +~~~ + +{% include copy-clipboard.html %} +~~~ shell +$ cockroach zone set db1.table@idx1 \ +--insecure \ +--host= \ +-f index_zone.yaml +~~~ -{{site.data.alerts.callout_info}}This is an enterprise-only feature.{{site.data.alerts.end}} +~~~ +range_min_bytes: 1048576 +range_max_bytes: 67108864 +gc: + ttlseconds: 86400 +num_replicas: 7 +constraints: [] +~~~ + +Alternately, you can pass the YAML content via the standard input: + +{% include copy-clipboard.html %} +~~~ shell +$ echo 'num_replicas: 7' | cockroach zone set db1.table@idx1 \ +--insecure \ +--host= \ +-f - +~~~ + +### Create a Replication Zone for a Table or Secondary Index Partition New in v2.0 + +{{site.data.alerts.callout_info}} +This is an [enterprise-only](enterprise-licensing.html) feature. +{{site.data.alerts.end}} To [control replication for table partitions](partitioning.html#replication-zones), create a YAML file defining only the values you want to change (other values will not be affected), and use the `cockroach zone set -f ` command with appropriate flags: {% include copy-clipboard.html %} ~~~ shell -$ cat > australia.zone.yml +$ cat > australia_zone.yml ~~~ ~~~ shell @@ -346,9 +421,16 @@ Apply zone configurations to corresponding partitions: {% include copy-clipboard.html %} ~~~ shell -$ cockroach zone set roachlearn.students_by_list.australia --insecure -f australia.zone.yml +$ cockroach zone set roachlearn.students_by_list.australia \ +--insecure \ +--host= \ +-f australia_zone.yml ~~~ +{{site.data.alerts.callout_success}} +Since the syntax is the same for defining a replication zone for a table or index partition (`database.table.partition`), give partitions names that communicate what they are partitioning, e.g., `australia_table` vs `australia_idx1`. +{{site.data.alerts.end}} + ### Create a Replication Zone for a System Range In addition to the databases and tables that are visible via the SQL interface, CockroachDB stores internal data in what are called system ranges. CockroachDB comes with pre-configured replication zones for some of these ranges: From 517aa4424056c3014ae599bf3d92f448d4ae16c8 Mon Sep 17 00:00:00 2001 From: Jesse Seldess Date: Thu, 4 Oct 2018 17:06:48 -0400 Subject: [PATCH 3/3] Fix links --- v2.0/configure-replication-zones.md | 12 ++++++------ v2.0/create-table.md | 2 +- v2.0/performance-tuning.md | 3 +-- v2.0/recommended-production-settings.md | 2 +- v2.1/configure-replication-zones.md | 2 +- v2.1/create-table.md | 2 +- v2.1/recommended-production-settings.md | 2 +- 7 files changed, 12 insertions(+), 13 deletions(-) diff --git a/v2.0/configure-replication-zones.md b/v2.0/configure-replication-zones.md index 40635adc35d..9121e3e6680 100644 --- a/v2.0/configure-replication-zones.md +++ b/v2.0/configure-replication-zones.md @@ -13,9 +13,9 @@ This page explains how replication zones work and how to use the `cockroach zone Currently, only the `root` user can configure replication zones. {{site.data.alerts.end}} -## Replication zone levels +## Replication Zone Levels -### For table data +### For Table Data There are five replication zone levels for [**table data**](architecture/distribution-layer.html#table-data) in a cluster, listed from least to most granular: @@ -25,9 +25,9 @@ Cluster | CockroachDB comes with a pre-configured `.default` replication zone th Database | You can add replication zones for specific databases. See [Create a Replication Zone for a Database](#create-a-replication-zone-for-a-database) for more details. Table | You can add replication zones for specific tables. See [Create a Replication Zone for a Table](#create-a-replication-zone-for-a-table). Index ([Enterprise-only](enterprise-licensing.html)) | The [secondary indexes](indexes.html) on a table will automatically use the replication zone for the table. However, with an enterprise license, you can add distinct replication zones for secondary indexes. See [Create a Replication Zone for a Secondary Index](#create-a-replication-zone-for-a-secondary-index) for more details. -Row ([Enterprise-only](enterprise-licensing.html)) | You can add replication zones for specific rows in a table or secondary index by [defining table partitions](partitioning.html). See [Create a Replication Zone for a Table Partition](#create-a-replication-zone-for-a-table-partition-new-in-v2-0) for more details. +Row ([Enterprise-only](enterprise-licensing.html)) | You can add replication zones for specific rows in a table or secondary index by [defining table partitions](partitioning.html). See [Create a Replication Zone for a Table Partition](#create-a-replication-zone-for-a-table-or-secondary-index-partition-new-in-v2-0) for more details. -### For system data +### For System Data In addition, CockroachDB stores internal [**system data**](architecture/distribution-layer.html#monolithic-sorted-map-structure) in what are called system ranges. There are two replication zone levels for this internal system data, listed from least to most granular: @@ -36,7 +36,7 @@ Level | Description Cluster | The `.default` replication zone mentioned above also applies to all system ranges not constrained by a more specific replication zone. System Range | CockroachDB comes with pre-configured replication zones for the "meta" and "liveness" system ranges. If necessary, you can add replication zones for the "timeseries" range and other "system" ranges as well. See [Create a Replication Zone for a System Range](#create-a-replication-zone-for-a-system-range) for more details.

CockroachDB also comes with a pre-configured replication zone for one internal table, `system.jobs`, which stores metadata about long-running jobs such as schema changes and backups. Historical queries are never run against this table and the rows in it are updated frequently, so the pre-configured zone gives this table a lower-than-default `ttlseconds`. -### Level priorities +### Level Priorities When replicating data, whether table or system, CockroachDB always uses the most granular replication zone available. For example, for a piece of user data: @@ -361,7 +361,7 @@ The [secondary indexes](indexes.html) on a table will automatically use the repl To control replication for a specific secondary index, create a YAML file defining only the values you want to change (other values will not be affected), and use the `cockroach zone set -f ` command with appropriate flags: {{site.data.alerts.callout_success}} -To get the name of a secondary index, which you need for the `cockroach zone set` command, use the [`SHOW INDEX`](show-index.html) or [`SHOW CREATE TABLE`](show-create.html) statements. +To get the name of a secondary index, which you need for the `cockroach zone set` command, use the [`SHOW INDEX`](show-index.html) or [`SHOW CREATE TABLE`](show-create-table.html) statements. {{site.data.alerts.end}} {% include copy-clipboard.html %} diff --git a/v2.0/create-table.md b/v2.0/create-table.md index 81e6588875a..39126f4e422 100644 --- a/v2.0/create-table.md +++ b/v2.0/create-table.md @@ -88,7 +88,7 @@ By default, tables are created in the default replication zone but can be placed ## Row-Level Replication New in v2.0 -CockroachDB allows [enterprise users](enterprise-licensing.html) to [define table partitions](partitioning.html), thus providing row-level control of how and where the data is stored. See [Create a Replication Zone for a Table Partition](configure-replication-zones.html#create-a-replication-zone-for-a-table-partition-new-in-v2-0) for more information. +CockroachDB allows [enterprise users](enterprise-licensing.html) to [define table partitions](partitioning.html), thus providing row-level control of how and where the data is stored. See [Create a Replication Zone for a Table Partition](configure-replication-zones.html#create-a-replication-zone-for-a-table-or-secondary-index-partition-new-in-v2-0) for more information. {{site.data.alerts.callout_info}}The primary key required for partitioning is different from the conventional primary key. To define the primary key for partitioning, prefix the unique identifier(s) in the primary key with all columns you want to partition and subpartition the table on, in the order in which you want to nest your subpartitions. See Partition using Primary Key for more details.{{site.data.alerts.end}} diff --git a/v2.0/performance-tuning.md b/v2.0/performance-tuning.md index 8cb81d8f54f..fa3c9739acc 100644 --- a/v2.0/performance-tuning.md +++ b/v2.0/performance-tuning.md @@ -2,7 +2,6 @@ title: Performance Tuning summary: Essential techniques for getting fast reads and writes in a single- and multi-region CockroachDB deployment. toc: true - --- This tutorial shows you essential techniques for getting fast reads and writes in CockroachDB, starting with a single-region deployment and expanding into multiple regions. @@ -1777,7 +1776,7 @@ For this service, the most effective technique for improving read and write late The `rides` table contains 1 million rows, so dropping this index will take a few minutes. {{site.data.alerts.end}} -7. Now [create replication zones](configure-replication-zones.html#create-a-replication-zone-for-a-table-partition-new-in-v2-0) to require city data to be stored on specific nodes based on node locality. +7. Now [create replication zones](configure-replication-zones.html#create-a-replication-zone-for-a-table-or-secondary-index-partition-new-in-v2-0) to require city data to be stored on specific nodes based on node locality. City | Locality -----|--------- diff --git a/v2.0/recommended-production-settings.md b/v2.0/recommended-production-settings.md index e1ea035a6a1..da37a574c92 100644 --- a/v2.0/recommended-production-settings.md +++ b/v2.0/recommended-production-settings.md @@ -56,7 +56,7 @@ Term | Definition - For best resilience: - Use many smaller nodes instead of fewer larger ones. Recovery from a failed node is faster when data is spread across more nodes. - - Use [zone configs](configure-replication-zones.html) to increase the replication factor from 3 (the default) to 5. This is especially recommended if you are using local disks rather than a cloud providers' network-attached disks that are often replicated underneath the covers, because local disks have a greater risk of failure. You can do this for the [entire cluster](configure-replication-zones.html#edit-the-default-replication-zone) or for specific [databases](configure-replication-zones.html#create-a-replication-zone-for-a-database), [tables](configure-replication-zones.html#create-a-replication-zone-for-a-table), or [rows](configure-replication-zones.html#create-a-replication-zone-for-a-table-partition-new-in-v2-0) (enterprise-only). + - Use [zone configs](configure-replication-zones.html) to increase the replication factor from 3 (the default) to 5. This is especially recommended if you are using local disks rather than a cloud providers' network-attached disks that are often replicated underneath the covers, because local disks have a greater risk of failure. You can do this for the [entire cluster](configure-replication-zones.html#edit-the-default-replication-zone) or for specific [databases](configure-replication-zones.html#create-a-replication-zone-for-a-database), [tables](configure-replication-zones.html#create-a-replication-zone-for-a-table), or [rows](configure-replication-zones.html#create-a-replication-zone-for-a-table-or-secondary-index-partition-new-in-v2-0) (enterprise-only). {{site.data.alerts.callout_danger}} {% include {{page.version.version}}/known-limitations/system-range-replication.md %} {{site.data.alerts.end}} diff --git a/v2.1/configure-replication-zones.md b/v2.1/configure-replication-zones.md index b9cf9a55519..7e153f7a793 100644 --- a/v2.1/configure-replication-zones.md +++ b/v2.1/configure-replication-zones.md @@ -25,7 +25,7 @@ Cluster | CockroachDB comes with a pre-configured `.default` replication zone th Database | You can add replication zones for specific databases. See [Create a Replication Zone for a Database](#create-a-replication-zone-for-a-database) for more details. Table | You can add replication zones for specific tables. See [Create a Replication Zone for a Table](#create-a-replication-zone-for-a-table). Index ([Enterprise-only](enterprise-licensing.html)) | The [secondary indexes](indexes.html) on a table will automatically use the replication zone for the table. However, with an enterprise license, you can add distinct replication zones for secondary indexes. See [Create a Replication Zone for a Secondary Index](#create-a-replication-zone-for-a-secondary-index) for more details. -Row ([Enterprise-only](enterprise-licensing.html)) | You can add replication zones for specific rows in a table or secondary index by [defining table partitions](partitioning.html). See [Create a Replication Zone for a Table Partition](#create-a-replication-zone-for-a-table-partition-new-in-v2-0) for more details. +Row ([Enterprise-only](enterprise-licensing.html)) | You can add replication zones for specific rows in a table or secondary index by [defining table partitions](partitioning.html). See [Create a Replication Zone for a Table Partition](#create-a-replication-zone-for-a-table-or-secondary-index-partition) for more details. ### For system data diff --git a/v2.1/create-table.md b/v2.1/create-table.md index 350a7f4a136..9e6b5b824f5 100644 --- a/v2.1/create-table.md +++ b/v2.1/create-table.md @@ -74,7 +74,7 @@ By default, tables are created in the default replication zone but can be placed ## Row-level replication -CockroachDB allows [enterprise users](enterprise-licensing.html) to [define table partitions](partitioning.html), thus providing row-level control of how and where the data is stored. See [Create a Replication Zone for a Table Partition](configure-replication-zones.html#create-a-replication-zone-for-a-table-partition) for more information. +CockroachDB allows [enterprise users](enterprise-licensing.html) to [define table partitions](partitioning.html), thus providing row-level control of how and where the data is stored. See [Create a Replication Zone for a Table Partition](configure-replication-zones.html#create-a-replication-zone-for-a-table-or-secondary-index-partition) for more information. {{site.data.alerts.callout_info}}The primary key required for partitioning is different from the conventional primary key. To define the primary key for partitioning, prefix the unique identifier(s) in the primary key with all columns you want to partition and subpartition the table on, in the order in which you want to nest your subpartitions. See Partition using Primary Key for more details.{{site.data.alerts.end}} diff --git a/v2.1/recommended-production-settings.md b/v2.1/recommended-production-settings.md index 2ad04f8307a..18bf4b6b05e 100644 --- a/v2.1/recommended-production-settings.md +++ b/v2.1/recommended-production-settings.md @@ -59,7 +59,7 @@ For added context about CockroachDB's fault tolerance and automated repair capab - For best resilience: - Use many smaller nodes instead of fewer larger ones. Recovery from a failed node is faster when data is spread across more nodes. - - Use [zone configs](configure-replication-zones.html) to increase the replication factor from 3 (the default) to 5. This is especially recommended if you are using local disks rather than a cloud providers' network-attached disks that are often replicated underneath the covers, because local disks have a greater risk of failure. You can do this for the [entire cluster](configure-replication-zones.html#edit-the-default-replication-zone) or for specific [databases](configure-replication-zones.html#create-a-replication-zone-for-a-database), [tables](configure-replication-zones.html#create-a-replication-zone-for-a-table), or [rows](configure-replication-zones.html#create-a-replication-zone-for-a-table-partition) (enterprise-only). + - Use [zone configs](configure-replication-zones.html) to increase the replication factor from 3 (the default) to 5. This is especially recommended if you are using local disks rather than a cloud providers' network-attached disks that are often replicated underneath the covers, because local disks have a greater risk of failure. You can do this for the [entire cluster](configure-replication-zones.html#edit-the-default-replication-zone) or for specific [databases](configure-replication-zones.html#create-a-replication-zone-for-a-database), [tables](configure-replication-zones.html#create-a-replication-zone-for-a-table), or [rows](configure-replication-zones.html#create-a-replication-zone-for-a-table-or-secondary-index-partition) (enterprise-only). {{site.data.alerts.callout_danger}} {% include {{page.version.version}}/known-limitations/system-range-replication.md %} {{site.data.alerts.end}}