Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Customize final destination schemas/datasets when configuring a source #1119

Closed
ChristopheDuong opened this issue Nov 30, 2020 · 0 comments
Closed
Labels
type/enhancement New feature or request

Comments

@ChristopheDuong
Copy link
Contributor

ChristopheDuong commented Nov 30, 2020

Tell us about the problem you're trying to solve

  • Currently, the schema (or dataset for BigQuery) is chosen when creating the destination. Up until now, we expected the data to be replicated and written in that destination "folder"
  • Following the work on Change jdbc sources to discover more than standard schemas #1038 we can now replicate multiple schemas at once when setting up a single DB source.
  • However this could fail especially in multi-tenants scenarios, we could find conflicts in table naming from different schemas. The user would potentially like to keep tables from different schemas separated or merge them all together.
  • Or as noted in Prefix table names in destinations with the source name #973, different sources could also use generic table names such as users which would definitely not be merge-able with different schemas without having to prefix table names with the source type.
  • Another concern is that it is sometimes desirable to not touch table names (adding prefix) but only tweak schemas .

Describe the solution you’d like

In the UI, when selecting the tables to be replicated when configuring a Catalog, it would be helpful to override the default schema defined in the destination per stream.

A new field in the Configured Catalog could be created to handle such customizable "namespaces" and eventual table renaming.

Such feature has been discussed and requested by users such as:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant