Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Experimental: Multi-tenant import support in Vitess #15503

Merged
merged 52 commits into from
Apr 1, 2024
Merged

Experimental: Multi-tenant import support in Vitess #15503

merged 52 commits into from
Apr 1, 2024
+12,562 −5,635

Commits on Mar 31, 2024

  1. Add multi_tenant attribute to keyspace 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    1b8029f View commit details
    Browse the repository at this point in the history

  2. Update internal KeyspaceSchema object and add unit test for it 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    4ee9820 View commit details
    Browse the repository at this point in the history

  3. Fix json serialization/deserialization adding a test for it 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    07a795e View commit details
    Browse the repository at this point in the history

  4. Change multi tenant attribute to use an object with column name/type.… 

    … Update tests

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    aa959db View commit details
    Browse the repository at this point in the history

  5. Add tenant id to Create. Pass it to the server 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    f5b5a9b View commit details
    Browse the repository at this point in the history

  6. Create json/struct for vrep options and update related sql/func. Upda… 

    …te create options and also add source alias.

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    8d0f27c View commit details
    Browse the repository at this point in the history

  7. Initial e2e test 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    a619db6 View commit details
    Browse the repository at this point in the history

  8. Add logic to append tenant clause filter to forward workflow 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    d58dcb4 View commit details
    Browse the repository at this point in the history

  9. Update options and filter on reverse streams 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    20f22ea View commit details
    Browse the repository at this point in the history

  10. Add e2e test to CI 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    b30edf0 View commit details
    Browse the repository at this point in the history

  11. Add KeyspaceRoutingRules to the vschema and related topo/vschema file… 

    …s, vtctld and a unit test

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    8000c5b View commit details
    Browse the repository at this point in the history

  12. Add keyspace routing rules for a multi-tenant migration 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    10b811c View commit details
    Browse the repository at this point in the history

  13. Don't point target keyspace in routing rules since we can't have targ… 

    …ets pointing to a single tenant. Running tests without Reverse

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    4f7e07d View commit details
    Browse the repository at this point in the history

  14. Remove reverse switching tests since we cannot really do it in multi-… 

    …tenant migrations

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    890a6a9 View commit details
    Browse the repository at this point in the history

  15. Add default to options column in VReplicationExec insert generator 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    7960093 View commit details
    Browse the repository at this point in the history

  16. Fix unit tests in go/vt/binlog 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    097e900 View commit details
    Browse the repository at this point in the history

  17. Fix failing unit tests in go/vt/vtctl/workflow 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    931f9b6 View commit details
    Browse the repository at this point in the history

  18. Fix TestInsertGenerator 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    2a357bb View commit details
    Browse the repository at this point in the history

  19. Fix Materializer tests 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    9036db2 View commit details
    Browse the repository at this point in the history

  20. Fix Reshard unit tests 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    ce4e897 View commit details
    Browse the repository at this point in the history

  21. Fix unit tests in /go/vt/vttablet/tabletmanager 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    1fd7766 View commit details
    Browse the repository at this point in the history

  22. Fix failing tests in tabletmanager/vreplication 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    a2fb454 View commit details
    Browse the repository at this point in the history

  23. Revert temporary hacks 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    78cdbc6 View commit details
    Browse the repository at this point in the history

  24. Revert to warn on no config 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    670ef6d View commit details
    Browse the repository at this point in the history

  25. Start a simple multi tenant test. To be expanded 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    d42d6f9 View commit details
    Browse the repository at this point in the history

  26. Fix TestUpdateVSchema 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    4f599c4 View commit details
    Browse the repository at this point in the history

  27. Use latest versions of vtctld/vtctl in Query Serving (Queries) - Upgr… 

    …ade Downgrade Testing

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    f4d78e0 View commit details
    Browse the repository at this point in the history

  28. Fix static code check failures. Attempt to fix upgrade/downgrade test 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    b2c6e1c View commit details
    Browse the repository at this point in the history

  29. Add Get/ApplyKeyspaceRoutingRules to vtctldclient 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    6b18d92 View commit details
    Browse the repository at this point in the history

  30. Add multi tenant ci workflow 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    1aec51c View commit details
    Browse the repository at this point in the history

  31. Fix keyspace routing rules duplicate and modify help text 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    c4066e9 View commit details
    Browse the repository at this point in the history

  32. Change attribute from VReplicationWorkflowOptions to WorkflowOptions 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    1248a1d View commit details
    Browse the repository at this point in the history

  33. Minor refactors 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    b680c4b View commit details
    Browse the repository at this point in the history

  34. Validate tenant id provided matches specified type in vschema 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    e698083 View commit details
    Browse the repository at this point in the history

  35. Fix failing tests 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    863ab46 View commit details
    Browse the repository at this point in the history

  36. Enhance simple multi tenant test to validate routing better. Remove r… 

    …outing rules on complete

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    4a4c48d View commit details
    Browse the repository at this point in the history

  37. Fix incorrect insert into target keyspace before switching traffic 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    59521fd View commit details
    Browse the repository at this point in the history

  38. Minor refactor. Cleanup comments/commented code 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    562b61e View commit details
    Browse the repository at this point in the history

  39. Address review comments 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    f54b2a9 View commit details
    Browse the repository at this point in the history

  40. Address more review comments 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    4123421 View commit details
    Browse the repository at this point in the history

  41. Remove timeout for workflow to see if it will fix static check error … 

    …which is not reproduced locally

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    d154b66 View commit details
    Browse the repository at this point in the history

  42. Rebase and make ci workflows again 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    a18387e View commit details
    Browse the repository at this point in the history

  43. Add mise config file to .gitignore 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    f70f7eb View commit details
    Browse the repository at this point in the history

  44. Fix regression in reverse workflow validation. 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    2eb44e6 View commit details
    Browse the repository at this point in the history

  45. Make Proto 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    7850fea View commit details
    Browse the repository at this point in the history

  46. Update e2e tests: don't specify multi_tenant_spec in source vschema 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    6758d01 View commit details
    Browse the repository at this point in the history

  47. Address review comments 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    077a5fa View commit details
    Browse the repository at this point in the history

  48. Address more review comments 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    a437a7f View commit details
    Browse the repository at this point in the history

  49. Re-enable wait for throttler config to be applied on all shards durin… 

    …g e2e tests

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    082d6d4 View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2024

  1. Use query.Type instead of string equivalent in vschema to define data… 

    … type for tenant id column

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    94b0291 View commit details
    Browse the repository at this point in the history

  2. Use INT64 string instead of enum in test vschema 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    38d34e9 View commit details
    Browse the repository at this point in the history

  3. Address review comments 

    Signed-off-by: Rohit Nayak <rohit@planetscale.com>
    @rohit-nayak-ps
    rohit-nayak-ps committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    36027f3 View commit details
    Browse the repository at this point in the history