-
Notifications
You must be signed in to change notification settings - Fork 3.8k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
server,kvaccessor: record span configs during tenant creation/gc
For newly created tenants, we want to ensure hard splits on tenant boundaries. The source of truth for split points in the span configs subsystem is the contents of system.span_configurations. To ensure hard splits, we insert a single key record at the tenant prefix. In a future commit we'll introduce the spanconfig.Reconciler process, which runs per tenant and governs all config entries under each tenant's purview. This has implications for this initial record we're talking about (this record might get cleaned up for e.g.); we'll explore it in tests for the Reconciler itself. Creating a single key record is easy enough -- we could've written directly to system.span_configurations. When a tenant is GC-ed however, we need to clear out all tenant state transactionally. To that end we plumb in a txn-scoped KVAccessor into the planner where crdb_internal.destroy_tenant is executed. This lets us easily delete all abandoned tenant span config records. Note: We get rid of spanconfig.experimental_kvaccessor.enabled. Access to spanconfigs infrastructure is already sufficiently gated through the env var. Now that crdb_internal.create_tenant attempts to write through the KVAccessor, it's cumbersome to have to enable the setting manually in every multi-tenant test (increasingly the default) enabling some part of the span configs infrastructure. --- This commit also needs a migration -- for existing clusters with secondary tenants, when upgrading we need to install this initial record at the tenant prefix for all extant tenants (and make sure to continue doing so for subsequent newly created tenants). This is to preserve the hard-split-on-tenant-boundary invariant we wish to provide. It's possible for an upgraded multi-tenant cluster to have dormant sql pods that have never reconciled. If KV switches over to the span configs subsystem, splitting only on the contents of system.span_configurations, we'll fail to split on all tenant boundaries. To this end we introduce clusterversion.SeedTenantSpanConfigs, which allows us to seed span config data for secondary tenants. The associated migration seeds entries for existing tenants. Release note: None
- Loading branch information
1 parent
884088d
commit 29fadd6
Showing
28 changed files
with
589 additions
and
129 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
189 changes: 189 additions & 0 deletions
189
pkg/ccl/migrationccl/migrationsccl/seed_tenant_span_configs_external_test.go
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,189 @@ | ||
| // Copyright 2021 The Cockroach Authors. | ||
| // | ||
| // Licensed as a CockroachDB Enterprise file under the Cockroach Community | ||
| // License (the "License"); you may not use this file except in compliance with | ||
| // the License. You may obtain a copy of the License at | ||
| // | ||
| // https://github.com/cockroachdb/cockroach/blob/master/licenses/CCL.txt | ||
|
|
||
| package migrationsccl_test | ||
|
|
||
| import ( | ||
| "context" | ||
| "testing" | ||
|
|
||
| "github.com/cockroachdb/cockroach/pkg/base" | ||
| _ "github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl" | ||
| "github.com/cockroachdb/cockroach/pkg/clusterversion" | ||
| "github.com/cockroachdb/cockroach/pkg/keys" | ||
| "github.com/cockroachdb/cockroach/pkg/roachpb" | ||
| "github.com/cockroachdb/cockroach/pkg/server" | ||
| "github.com/cockroachdb/cockroach/pkg/spanconfig" | ||
| "github.com/cockroachdb/cockroach/pkg/testutils/sqlutils" | ||
| "github.com/cockroachdb/cockroach/pkg/testutils/testcluster" | ||
| "github.com/cockroachdb/cockroach/pkg/util/leaktest" | ||
| "github.com/cockroachdb/cockroach/pkg/util/log" | ||
| "github.com/stretchr/testify/require" | ||
| ) | ||
|
|
||
| // TestPreSeedSpanConfigsWrittenWhenActive tests that seed span configs are | ||
| // written to for fresh tenants if the cluster version that introduced it is | ||
| // active. | ||
| func TestPreSeedSpanConfigsWrittenWhenActive(t *testing.T) { | ||
| defer leaktest.AfterTest(t)() | ||
| defer log.Scope(t).Close(t) | ||
|
|
||
| ctx := context.Background() | ||
| tc := testcluster.StartTestCluster(t, 1, base.TestClusterArgs{ | ||
| ServerArgs: base.TestServerArgs{ | ||
| EnableSpanConfigs: true, // we use spanconfig.KVAccessor to check if its contents are as we'd expect | ||
| Knobs: base.TestingKnobs{ | ||
| Server: &server.TestingKnobs{ | ||
| DisableAutomaticVersionUpgrade: 1, | ||
| BinaryVersionOverride: clusterversion.ByKey( | ||
| clusterversion.PreSeedTenantSpanConfigs, | ||
| ), | ||
| }, | ||
| }, | ||
| }, | ||
| }) | ||
|
|
||
| defer tc.Stopper().Stop(ctx) | ||
| ts := tc.Server(0) | ||
|
|
||
| tenantID := roachpb.MakeTenantID(10) | ||
| _, err := ts.StartTenant(ctx, base.TestTenantArgs{TenantID: tenantID}) | ||
| require.NoError(t, err) | ||
|
|
||
| scKVAccessor := ts.SpanConfigKVAccessor().(spanconfig.KVAccessor) | ||
| tenantPrefix := keys.MakeTenantPrefix(tenantID) | ||
| tenantSpan := roachpb.Span{Key: tenantPrefix, EndKey: tenantPrefix.PrefixEnd()} | ||
| tenantSeedSpan := roachpb.Span{Key: tenantPrefix, EndKey: tenantPrefix.Next()} | ||
|
|
||
| { | ||
| entries, err := scKVAccessor.GetSpanConfigEntriesFor(ctx, []roachpb.Span{ | ||
| tenantSpan, | ||
| }) | ||
| require.NoError(t, err) | ||
| require.Len(t, entries, 1) | ||
| require.Equal(t, entries[0].Span, tenantSeedSpan) | ||
| } | ||
| } | ||
|
|
||
| // TestSeedTenantSpanConfigs tests that the migration installs relevant seed | ||
| // span configs for existing secondary tenants. | ||
| func TestSeedTenantSpanConfigs(t *testing.T) { | ||
| defer leaktest.AfterTest(t)() | ||
| defer log.Scope(t).Close(t) | ||
|
|
||
| ctx := context.Background() | ||
| tc := testcluster.StartTestCluster(t, 1, base.TestClusterArgs{ | ||
| ServerArgs: base.TestServerArgs{ | ||
| EnableSpanConfigs: true, // we use spanconfig.KVAccessor to check if its contents are as we'd expect | ||
| Knobs: base.TestingKnobs{ | ||
| Server: &server.TestingKnobs{ | ||
| DisableAutomaticVersionUpgrade: 1, | ||
| BinaryVersionOverride: clusterversion.ByKey( | ||
| clusterversion.PreSeedTenantSpanConfigs - 1, | ||
| ), | ||
| }, | ||
| }, | ||
| }, | ||
| }) | ||
|
|
||
| defer tc.Stopper().Stop(ctx) | ||
| ts := tc.Server(0) | ||
| tdb := sqlutils.MakeSQLRunner(tc.ServerConn(0)) | ||
| scKVAccessor := ts.SpanConfigKVAccessor().(spanconfig.KVAccessor) | ||
|
|
||
| tenantID := roachpb.MakeTenantID(10) | ||
| tenantPrefix := keys.MakeTenantPrefix(tenantID) | ||
| tenantSpan := roachpb.Span{Key: tenantPrefix, EndKey: tenantPrefix.PrefixEnd()} | ||
| tenantSeedSpan := roachpb.Span{Key: tenantPrefix, EndKey: tenantPrefix.Next()} | ||
| { | ||
| _, err := ts.StartTenant(ctx, base.TestTenantArgs{TenantID: tenantID}) | ||
| require.NoError(t, err) | ||
| } | ||
|
|
||
| { // Ensure that no span config entries are to be found | ||
| entries, err := scKVAccessor.GetSpanConfigEntriesFor(ctx, []roachpb.Span{ | ||
| tenantSpan, | ||
| }) | ||
| require.NoError(t, err) | ||
| require.Empty(t, entries) | ||
| } | ||
|
|
||
| tdb.Exec(t, | ||
| "SET CLUSTER SETTING version = $1", | ||
| clusterversion.ByKey(clusterversion.SeedTenantSpanConfigs).String(), | ||
| ) | ||
|
|
||
| { // Ensure that the tenant now has a span config entry. | ||
| entries, err := scKVAccessor.GetSpanConfigEntriesFor(ctx, []roachpb.Span{ | ||
| tenantSpan, | ||
| }) | ||
| require.NoError(t, err) | ||
| require.Len(t, entries, 1) | ||
| require.Equal(t, entries[0].Span, tenantSeedSpan) | ||
| } | ||
| } | ||
|
|
||
| // TestSeedTenantSpanConfigsWithExistingEntry tests that the migration ignores | ||
| // tenants with existing span config entries. | ||
| func TestSeedTenantSpanConfigsWithExistingEntry(t *testing.T) { | ||
| defer leaktest.AfterTest(t)() | ||
| defer log.Scope(t).Close(t) | ||
|
|
||
| ctx := context.Background() | ||
| tc := testcluster.StartTestCluster(t, 1, base.TestClusterArgs{ | ||
| ServerArgs: base.TestServerArgs{ | ||
| EnableSpanConfigs: true, // we use spanconfig.KVAccessor to check if its contents are as we'd expect | ||
| Knobs: base.TestingKnobs{ | ||
| Server: &server.TestingKnobs{ | ||
| DisableAutomaticVersionUpgrade: 1, | ||
| BinaryVersionOverride: clusterversion.ByKey( | ||
| clusterversion.PreSeedTenantSpanConfigs, | ||
| ), | ||
| }, | ||
| }, | ||
| }, | ||
| }) | ||
|
|
||
| defer tc.Stopper().Stop(ctx) | ||
| ts := tc.Server(0) | ||
| tdb := sqlutils.MakeSQLRunner(tc.ServerConn(0)) | ||
| scKVAccessor := ts.SpanConfigKVAccessor().(spanconfig.KVAccessor) | ||
|
|
||
| tenantID := roachpb.MakeTenantID(10) | ||
| tenantPrefix := keys.MakeTenantPrefix(tenantID) | ||
| tenantSpan := roachpb.Span{Key: tenantPrefix, EndKey: tenantPrefix.PrefixEnd()} | ||
| tenantSeedSpan := roachpb.Span{Key: tenantPrefix, EndKey: tenantPrefix.Next()} | ||
| { | ||
| _, err := ts.StartTenant(ctx, base.TestTenantArgs{TenantID: tenantID}) | ||
| require.NoError(t, err) | ||
| } | ||
|
|
||
| { // Ensure that the tenant already has a span config entry. | ||
| entries, err := scKVAccessor.GetSpanConfigEntriesFor(ctx, []roachpb.Span{ | ||
| tenantSpan, | ||
| }) | ||
| require.NoError(t, err) | ||
| require.Len(t, entries, 1) | ||
| require.Equal(t, entries[0].Span, tenantSeedSpan) | ||
| } | ||
|
|
||
| // Ensure the cluster version bump goes through successfully. | ||
| tdb.Exec(t, | ||
| "SET CLUSTER SETTING version = $1", | ||
| clusterversion.ByKey(clusterversion.SeedTenantSpanConfigs).String(), | ||
| ) | ||
|
|
||
| { // Ensure that the tenant's span config entry stay as it was. | ||
| entries, err := scKVAccessor.GetSpanConfigEntriesFor(ctx, []roachpb.Span{ | ||
| tenantSpan, | ||
| }) | ||
| require.NoError(t, err) | ||
| require.Len(t, entries, 1) | ||
| require.Equal(t, entries[0].Span, tenantSeedSpan) | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.