Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental: Multi-tenant import support in Vitess #15503

Merged
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
1b8029f
Add multi_tenant attribute to keyspace
rohit-nayak-ps Mar 13, 2024
4ee9820
Update internal KeyspaceSchema object and add unit test for it
rohit-nayak-ps Mar 13, 2024
07a795e
Fix json serialization/deserialization adding a test for it
rohit-nayak-ps Mar 13, 2024
aa959db
Change multi tenant attribute to use an object with column name/type.…
rohit-nayak-ps Mar 13, 2024
f5b5a9b
Add tenant id to Create. Pass it to the server
rohit-nayak-ps Mar 13, 2024
8d0f27c
Create json/struct for vrep options and update related sql/func. Upda…
rohit-nayak-ps Mar 13, 2024
a619db6
Initial e2e test
rohit-nayak-ps Mar 13, 2024
d58dcb4
Add logic to append tenant clause filter to forward workflow
rohit-nayak-ps Mar 13, 2024
20f22ea
Update options and filter on reverse streams
rohit-nayak-ps Mar 13, 2024
b30edf0
Add e2e test to CI
rohit-nayak-ps Mar 14, 2024
8000c5b
Add KeyspaceRoutingRules to the vschema and related topo/vschema file…
rohit-nayak-ps Mar 14, 2024
10b811c
Add keyspace routing rules for a multi-tenant migration
rohit-nayak-ps Mar 16, 2024
4f7e07d
Don't point target keyspace in routing rules since we can't have targ…
rohit-nayak-ps Mar 16, 2024
890a6a9
Remove reverse switching tests since we cannot really do it in multi-…
rohit-nayak-ps Mar 16, 2024
7960093
Add default to options column in VReplicationExec insert generator
rohit-nayak-ps Mar 17, 2024
097e900
Fix unit tests in go/vt/binlog
rohit-nayak-ps Mar 17, 2024
931f9b6
Fix failing unit tests in go/vt/vtctl/workflow
rohit-nayak-ps Mar 17, 2024
2a357bb
Fix TestInsertGenerator
rohit-nayak-ps Mar 17, 2024
9036db2
Fix Materializer tests
rohit-nayak-ps Mar 17, 2024
ce4e897
Fix Reshard unit tests
rohit-nayak-ps Mar 17, 2024
1fd7766
Fix unit tests in /go/vt/vttablet/tabletmanager
rohit-nayak-ps Mar 17, 2024
a2fb454
Fix failing tests in tabletmanager/vreplication
rohit-nayak-ps Mar 17, 2024
78cdbc6
Revert temporary hacks
rohit-nayak-ps Mar 17, 2024
670ef6d
Revert to warn on no config
rohit-nayak-ps Mar 17, 2024
d42d6f9
Start a simple multi tenant test. To be expanded
rohit-nayak-ps Mar 17, 2024
4f599c4
Fix TestUpdateVSchema
rohit-nayak-ps Mar 17, 2024
f4d78e0
Use latest versions of vtctld/vtctl in Query Serving (Queries) - Upgr…
rohit-nayak-ps Mar 18, 2024
b2c6e1c
Fix static code check failures. Attempt to fix upgrade/downgrade test
rohit-nayak-ps Mar 18, 2024
6b18d92
Add Get/ApplyKeyspaceRoutingRules to vtctldclient
rohit-nayak-ps Mar 18, 2024
1aec51c
Add multi tenant ci workflow
rohit-nayak-ps Mar 18, 2024
c4066e9
Fix keyspace routing rules duplicate and modify help text
rohit-nayak-ps Mar 18, 2024
1248a1d
Change attribute from VReplicationWorkflowOptions to WorkflowOptions
rohit-nayak-ps Mar 18, 2024
b680c4b
Minor refactors
rohit-nayak-ps Mar 18, 2024
e698083
Validate tenant id provided matches specified type in vschema
rohit-nayak-ps Mar 18, 2024
863ab46
Fix failing tests
rohit-nayak-ps Mar 18, 2024
4a4c48d
Enhance simple multi tenant test to validate routing better. Remove r…
rohit-nayak-ps Mar 19, 2024
59521fd
Fix incorrect insert into target keyspace before switching traffic
rohit-nayak-ps Mar 19, 2024
562b61e
Minor refactor. Cleanup comments/commented code
rohit-nayak-ps Mar 24, 2024
f54b2a9
Address review comments
rohit-nayak-ps Mar 24, 2024
4123421
Address more review comments
rohit-nayak-ps Mar 24, 2024
d154b66
Remove timeout for workflow to see if it will fix static check error …
rohit-nayak-ps Mar 24, 2024
a18387e
Rebase and make ci workflows again
rohit-nayak-ps Mar 24, 2024
f70f7eb
Add mise config file to .gitignore
rohit-nayak-ps Mar 24, 2024
2eb44e6
Fix regression in reverse workflow validation.
rohit-nayak-ps Mar 24, 2024
7850fea
Make Proto
rohit-nayak-ps Mar 24, 2024
6758d01
Update e2e tests: don't specify multi_tenant_spec in source vschema
rohit-nayak-ps Mar 27, 2024
077a5fa
Address review comments
rohit-nayak-ps Mar 31, 2024
a437a7f
Address more review comments
rohit-nayak-ps Mar 31, 2024
082d6d4
Re-enable wait for throttler config to be applied on all shards durin…
rohit-nayak-ps Mar 31, 2024
94b0291
Use query.Type instead of string equivalent in vschema to define data…
rohit-nayak-ps Apr 1, 2024
38d34e9
Use INT64 string instead of enum in test vschema
rohit-nayak-ps Apr 1, 2024
36027f3
Address review comments
rohit-nayak-ps Apr 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion go/test/endtoend/vreplication/multi_tenant_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ const (
{
"multi_tenant_spec": {
"tenant_id_column_name": "tenant_id",
"tenant_id_column_type": "int64"
"tenant_id_column_type": 265
rohit-nayak-ps marked this conversation as resolved.
Show resolved Hide resolved
},
"tables": {
"t1": {}
Expand Down
274 changes: 138 additions & 136 deletions go/vt/proto/vschema/vschema.pb.go

Large diffs are not rendered by default.

32 changes: 8 additions & 24 deletions go/vt/proto/vschema/vschema_vtproto.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 7 additions & 5 deletions go/vt/vtctl/workflow/utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ import (
"strings"
"sync"

querypb "vitess.io/vitess/go/vt/proto/query"

"vitess.io/vitess/go/vt/vtgate/vindexes"

"google.golang.org/protobuf/encoding/prototext"
Expand Down Expand Up @@ -806,13 +808,13 @@ func getTenantClause(vrOptions *vtctldatapb.WorkflowOptions,

var tenantId string
switch tenantColumnType {
case "int", "int64":
case querypb.Type_INT64:
_, err := strconv.Atoi(vrOptions.TenantId)
if err != nil {
return nil, fmt.Errorf("tenant id is not a valid int: %s", vrOptions.TenantId)
}
tenantId = vrOptions.TenantId
case "string":
case querypb.Type_VARCHAR:
tenantId = fmt.Sprintf("'%s'", vrOptions.TenantId)
default:
return nil, fmt.Errorf("unsupported tenant column type: %s", tenantColumnType)
Expand Down Expand Up @@ -847,14 +849,14 @@ func updateKeyspaceRoutingRule(ctx context.Context, ts *topo.Server, routes map[
return nil
}

func validateTenantId(dataType, value string) error {
func validateTenantId(dataType querypb.Type, value string) error {
switch dataType {
case "int", "int64":
case querypb.Type_INT64:
_, err := strconv.Atoi(value)
if err != nil {
return fmt.Errorf("value %s is not a valid int", value)
}
case "string":
case querypb.Type_VARCHAR:
// no validation needed
default:
return fmt.Errorf("unsupported data type: %s", dataType)
Expand Down
28 changes: 14 additions & 14 deletions go/vt/vtgate/vindexes/vschema_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -259,23 +259,23 @@ func TestMultiTenantAttribute(t *testing.T) {
wanted *vschemapb.MultiTenantSpec
}{
{
name: "Default Value",
name: "Not Set",
}, {
name: "MultiTenantSpec Set to True",
name: "Empty MultiTenantSpec",
multiTenantSpec: &vschemapb.MultiTenantSpec{
TenantIdColumnName: "",
},
wanted: &vschemapb.MultiTenantSpec{},
}, {
name: "String",
multiTenantSpec: &vschemapb.MultiTenantSpec{
TenantIdColumnName: "tenant_id",
TenantIdColumnType: "int64",
TenantIdColumnType: querypb.Type_VARCHAR,
},
wanted: &vschemapb.MultiTenantSpec{
TenantIdColumnName: "tenant_id",
TenantIdColumnType: "int64",
TenantIdColumnType: querypb.Type_VARCHAR,
},
}, {
name: "MultiTenantSpec Set to True",
multiTenantSpec: &vschemapb.MultiTenantSpec{
TenantIdColumnName: "",
},
wanted: &vschemapb.MultiTenantSpec{},
},
}
for _, test := range tests {
Expand Down Expand Up @@ -2660,7 +2660,7 @@ func TestVSchemaPBJSON(t *testing.T) {
"foreignKeyMode": "unmanaged",
"multi_tenant_spec": {
"tenant_id_column_name": "tenant_id",
"tenant_id_column_type": "int64"
"tenant_id_column_type": 265
},
"tables": {
"t1": {
Expand Down Expand Up @@ -2694,7 +2694,7 @@ func TestVSchemaPBJSON(t *testing.T) {
ForeignKeyMode: vschemapb.Keyspace_unmanaged,
MultiTenantSpec: &vschemapb.MultiTenantSpec{
TenantIdColumnName: "tenant_id",
TenantIdColumnType: "int64",
TenantIdColumnType: querypb.Type_INT64,
},
Tables: map[string]*vschemapb.Table{
"t1": {
Expand Down Expand Up @@ -2742,7 +2742,7 @@ func TestVSchemaJSON(t *testing.T) {
ForeignKeyMode: vschemapb.Keyspace_managed,
MultiTenantSpec: &vschemapb.MultiTenantSpec{
TenantIdColumnName: "tenant_id",
TenantIdColumnType: "int64",
TenantIdColumnType: querypb.Type_INT64,
},
Keyspace: &Keyspace{
Name: "k1",
Expand Down Expand Up @@ -2840,7 +2840,7 @@ func TestVSchemaJSON(t *testing.T) {
},
"multi_tenant_spec": {
"tenant_id_column_name": "tenant_id",
"tenant_id_column_type": "int64"
"tenant_id_column_type": 265
}
}
}`
Expand Down
2 changes: 1 addition & 1 deletion proto/vschema.proto
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ message MultiTenantSpec {
// tenant_column is the name of the column that specifies the tenant id.
string tenant_id_column_name = 1;
// tenant_column_type is the type of the column that specifies the tenant id.
string tenant_id_column_type = 2;
query.Type tenant_id_column_type = 2;
}

// Vindex is the vindex info for a Keyspace.
Expand Down
4 changes: 2 additions & 2 deletions web/vtadmin/src/proto/vtadmin.d.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading