Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

distsql: does not work with multi-tenancy #47900

Closed
tbg opened this issue Apr 22, 2020 · 0 comments
Closed

distsql: does not work with multi-tenancy #47900

tbg opened this issue Apr 22, 2020 · 0 comments
Labels
A-multitenancy Related to multi-tenancy T-sql-queries SQL Queries Team X-anchored-telemetry The issue number is anchored by telemetry references.

Comments

@tbg
Copy link
Member

tbg commented Apr 22, 2020 •
edited by cockroach-jira-scripts
Loading

See here for background

Multi-tenancy Phase 2 will ship with DistSQL disabled, which makes a lot of sense since at most one SQL tenant server will be running on behalf of a given tenant at any time (i.e. there's nobody else to distribute to).

Sometime down the road, we may want to lift that restriction. This needs a few main ingredients:

  • access to leaseholder cache and range descriptor cache is not necessarily given
  • gossip not available (note: used for version compat here as well as other things)
  • rpc: inter-tenant connections #47899

Jira issue: CRDB-4378
@tbg tbg added the A-multitenancy Related to multi-tenancy label Apr 22, 2020
@tbg tbg mentioned this issue Apr 22, 2020
Closed
tbg added a commit to tbg/cockroach that referenced this issue May 5, 2020
@tbg
sql: downgrade some gossip.Deprecated() calls
cfe7722 
Deprecated() calls are blocking phase 2, but the ones
made optional here were not.

Touches cockroachdb#47900.
Touches cockroachdb#47925.

Release note: None
tbg added a commit to tbg/cockroach that referenced this issue May 7, 2020
@tbg
sql: downgrade some gossip.Deprecated() calls
cb06dc9 
Deprecated() calls are blocking phase 2, but the ones
made optional here were not.

Touches cockroachdb#47900.
Touches cockroachdb#47925.

Release note: None
@knz knz added the X-anchored-telemetry The issue number is anchored by telemetry references. label Sep 11, 2020
@knz knz mentioned this issue Sep 11, 2020
Merged
craig bot pushed a commit that referenced this issue Sep 14, 2020
@knz
Merge #54256
40f03d9 
54256: sql: make multi-tenancy errors report to telemetry r=asubiotto a=knz

Fixes #48164. 

Issues referenced, I also added the X-anchored-telemetry label to them on github (we keep those issues open on github until their reference in the code is removed):

#54254 
#54255 
#54250 
#54251 
#54252 
#49854 
#48274
#47150
#47971
#47970
#47900 
#47925 


Co-authored-by: Raphael 'kena' Poss <knz@thaumogen.net>
@asubiotto asubiotto mentioned this issue Nov 16, 2020
Closed
@otan otan mentioned this issue May 14, 2021
Closed
@jlinder jlinder added the T-sql-queries SQL Queries Team label Jun 16, 2021
rharding6373 added a commit to rharding6373/cockroach that referenced this issue Feb 16, 2022
@rharding6373
distsql: adds support for distributed bulk ops in multi-tenant
87e64e6 
Previously, distsql did not support distributed queries in multi-tenant
environments. Queries were always planned using a single SQL pod, even
if using multiple pods could yield better parallelism.

This change adds an interface inside distsql that allows different
implementations for functions that differ between single- and
multi-tenant. Currently, it only has `SetupAllNodesPlanning`, which for
single-tenant remains the same, but in multi-tenant returns all
available SQL pods at the time it is invoked.

This PR also fixes some places where SQL pods need to communicate with
other SQL pods using the PodNodeDialer instead of the NodeDialer that
handles SQL <-> KV communication in multi-tenant mode.

Fixes cockroachdb#74548

References cockroachdb#47900

Release note (sql change): Adds support for distributed import queries in
multi-tenant environments, which allows import queries to have improved
parallelism by utilizing all available SQL pods in the tenant.

review
rharding6373 added a commit to rharding6373/cockroach that referenced this issue Feb 17, 2022
@rharding6373
distsql: adds support for distributed bulk ops in multi-tenant
e01cc77 
Previously, distsql did not support distributed queries in multi-tenant
environments. Queries were always planned using a single SQL pod, even
if using multiple pods could yield better parallelism.

This change adds an interface inside distsql that allows different
implementations for functions that differ between single- and
multi-tenant. Currently, it only has `SetupAllNodesPlanning`, which for
single-tenant remains the same, but in multi-tenant returns all
available SQL pods at the time it is invoked.

This PR also fixes some places where SQL pods need to communicate with
other SQL pods using the PodNodeDialer instead of the NodeDialer that
handles SQL <-> KV communication in multi-tenant mode.

Fixes cockroachdb#74548

References cockroachdb#47900

Release note (sql change): Adds support for distributed import queries in
multi-tenant environments, which allows import queries to have improved
parallelism by utilizing all available SQL pods in the tenant.

review
@rharding6373 rharding6373 mentioned this issue Feb 23, 2022
Merged
rharding6373 added a commit to rharding6373/cockroach that referenced this issue Feb 23, 2022
@rharding6373
distsql: adds support for distributed bulk ops in multi-tenant
c43f7a2 
Previously, distsql did not support distributed queries in multi-tenant
environments. Queries were always planned using a single SQL pod, even
if using multiple pods could yield better parallelism.

This change adds an interface inside distsql that allows different
implementations for functions that differ between single- and
multi-tenant. Currently, it only has `SetupAllNodesPlanning`, which for
single-tenant remains the same, but in multi-tenant returns all
available SQL pods at the time it is invoked.

This PR also fixes some places where SQL pods need to communicate with
other SQL pods using the PodNodeDialer instead of the NodeDialer that
handles SQL <-> KV communication in multi-tenant mode.

Fixes cockroachdb#74548

References cockroachdb#47900

Release note (sql change): Adds support for distributed import queries in
multi-tenant environments, which allows import queries to have improved
parallelism by utilizing all available SQL pods in the tenant.
rharding6373 added a commit to rharding6373/cockroach that referenced this issue Feb 24, 2022
@rharding6373
distsql: adds support for distributed bulk ops in multi-tenant
c4a6f7a 
Previously, distsql did not support distributed queries in multi-tenant
environments. Queries were always planned using a single SQL pod, even
if using multiple pods could yield better parallelism.

This change adds an interface inside distsql that allows different
implementations for functions that differ between single- and
multi-tenant. Currently, it only has `SetupAllNodesPlanning`, which for
single-tenant remains the same, but in multi-tenant returns all
available SQL pods at the time it is invoked.

This PR also fixes some places where SQL pods need to communicate with
other SQL pods using the PodNodeDialer instead of the NodeDialer that
handles SQL <-> KV communication in multi-tenant mode.

Fixes cockroachdb#74548

References cockroachdb#47900

Release note (sql change): Adds support for distributed import queries in
multi-tenant environments, which allows import queries to have improved
parallelism by utilizing all available SQL pods in the tenant.
rharding6373 added a commit to rharding6373/cockroach that referenced this issue Feb 24, 2022
@rharding6373
distsql: adds support for distributed bulk ops in multi-tenant
ba98615 
Previously, distsql did not support distributed queries in multi-tenant
environments. Queries were always planned using a single SQL pod, even
if using multiple pods could yield better parallelism.

This change adds an interface inside distsql that allows different
implementations for functions that differ between single- and
multi-tenant. Currently, it only has `SetupAllNodesPlanning`, which for
single-tenant remains the same, but in multi-tenant returns all
available SQL pods at the time it is invoked.

This PR also fixes some places where SQL pods need to communicate with
other SQL pods using the PodNodeDialer instead of the NodeDialer that
handles SQL <-> KV communication in multi-tenant mode.

Fixes cockroachdb#74548

References cockroachdb#47900

Release note (sql change): Adds support for distributed import queries in
multi-tenant environments, which allows import queries to have improved
parallelism by utilizing all available SQL pods in the tenant.
rharding6373 added a commit to rharding6373/cockroach that referenced this issue Feb 25, 2022
@rharding6373
distsql: adds support for distributed bulk ops in multi-tenant
371dd11 
Previously, distsql did not support distributed queries in multi-tenant
environments. Queries were always planned using a single SQL pod, even
if using multiple pods could yield better parallelism.

This change adds an interface inside distsql that allows different
implementations for functions that differ between single- and
multi-tenant. Currently, it only has `SetupAllNodesPlanning`, which for
single-tenant remains the same, but in multi-tenant returns all
available SQL pods at the time it is invoked.

This PR also fixes some places where SQL pods need to communicate with
other SQL pods using the PodNodeDialer instead of the NodeDialer that
handles SQL <-> KV communication in multi-tenant mode.

Fixes cockroachdb#74548

References cockroachdb#47900

Release note (sql change): Adds support for distributed import queries in
multi-tenant environments, which allows import queries to have improved
parallelism by utilizing all available SQL pods in the tenant.
rharding6373 added a commit to rharding6373/cockroach that referenced this issue Feb 25, 2022
@rharding6373
distsql: adds support for distributed bulk ops in multi-tenant
677a7d5 
Previously, distsql did not support distributed queries in multi-tenant
environments. Queries were always planned using a single SQL pod, even
if using multiple pods could yield better parallelism.

This change adds an interface inside distsql that allows different
implementations for functions that differ between single- and
multi-tenant. Currently, it only has `SetupAllNodesPlanning`, which for
single-tenant remains the same, but in multi-tenant returns all
available SQL pods at the time it is invoked.

This PR also fixes some places where SQL pods need to communicate with
other SQL pods using the PodNodeDialer instead of the NodeDialer that
handles SQL <-> KV communication in multi-tenant mode.

Fixes cockroachdb#74548

References cockroachdb#47900

Release note (sql change): Adds support for distributed import queries in
multi-tenant environments, which allows import queries to have improved
parallelism by utilizing all available SQL pods in the tenant.
craig bot pushed a commit that referenced this issue Feb 25, 2022
@rharding6373 @rickystewart @rafiss
Merge #76566 #76958 #77025 #77044
c3ff362 
76566: distsql: adds support for distributed bulk ops in multi-tenant r=rharding6373 a=rharding6373

distsql: adds support for distributed bulk ops in multi-tenant
    
Previously, distsql did not support distributed queries in multi-tenant
environments. Queries were always planned using a single SQL pod, even
if using multiple pods could yield better parallelism.
    
This change adds an interface inside distsql that allows different
implementations for functions that differ between single- and
multi-tenant. Currently, it only has `SetupAllNodesPlanning`, which for
single-tenant remains the same, but in multi-tenant returns all
available SQL pods at the time it is invoked.
    
This PR also fixes some places where SQL pods need to communicate with
other SQL pods using the PodNodeDialer instead of the NodeDialer that
handles SQL <-> KV communication in multi-tenant mode.

Fixes #74548

References #47900
    
Release note (sql change): Adds support for distributed import queries in
multi-tenant environments, which allows import queries to have improved
parallelism by utilizing all available SQL pods in the tenant.

importccl, backupccl: add import and restore tenant tests with multiple SQL pods

These tests validate that imports and restores can be
distributed across multiple SQL pods in a single tenant.

Release note: None




76958: ci: a couple tweaks to bazel CI r=rail a=rickystewart

* Run `go mod tidy` in Bazel CI.
* When running unit tests, write the profile file to `/artifacts` so it
  can be inspected later if necessary.

Release note: None

77025: sql: fix error handling for AOST clause r=RichardJCai a=rafiss

fixes #76979
fixes #76978
fixes #76977
fixes #76975
fixes #76787

The AOST clause logic would previously error for any bounded staleness
query that was used in a prepared statement. This is because prepared
statements use an implicit txn, so the same AOST clause has to be
checked multiple times in the txn. A recent commit refactored
the error handling and caused this bug.

Release note: None

77044: schemachanger: bump size of test r=rail a=rickystewart

Release note: None

Co-authored-by: rharding6373 <rharding6373@users.noreply.github.com>
Co-authored-by: Ricky Stewart <ricky@cockroachlabs.com>
Co-authored-by: Rafi Shamim <rafi@cockroachlabs.com>
RajivTS pushed a commit to RajivTS/cockroach that referenced this issue Mar 6, 2022
@rharding6373 @RajivTS
distsql: adds support for distributed bulk ops in multi-tenant
d00aa1d 
Previously, distsql did not support distributed queries in multi-tenant
environments. Queries were always planned using a single SQL pod, even
if using multiple pods could yield better parallelism.

This change adds an interface inside distsql that allows different
implementations for functions that differ between single- and
multi-tenant. Currently, it only has `SetupAllNodesPlanning`, which for
single-tenant remains the same, but in multi-tenant returns all
available SQL pods at the time it is invoked.

This PR also fixes some places where SQL pods need to communicate with
other SQL pods using the PodNodeDialer instead of the NodeDialer that
handles SQL <-> KV communication in multi-tenant mode.

Fixes cockroachdb#74548

References cockroachdb#47900

Release note (sql change): Adds support for distributed import queries in
multi-tenant environments, which allows import queries to have improved
parallelism by utilizing all available SQL pods in the tenant.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-multitenancy Related to multi-tenancy T-sql-queries SQL Queries Team X-anchored-telemetry The issue number is anchored by telemetry references.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jlinder @knz @tbg @asubiotto