Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Destination Databricks: connector should support schema evolution (schemaMerge) #26658

Closed
gpodevijn opened this issue May 26, 2023 · 2 comments · Fixed by #26942
Closed

Destination Databricks: connector should support schema evolution (schemaMerge) #26658

gpodevijn opened this issue May 26, 2023 · 2 comments · Fixed by #26942
Labels
area/connectors Connector related issues community connectors/destination/databricks connectors/destinations-warehouse releaseStage/alpha team/destinations Destinations team's backlog type/enhancement New feature or request

Comments

@gpodevijn
Copy link
Contributor

gpodevijn commented May 26, 2023 •
edited
Loading

What area the feature impact?

Connectors

Revelant Information

Databricks and its delta tables support schema evolution via the COPY INTO command parameter's mergeSchema=true. The connector should provide a parameter to enable or disable schema evolution.

In #26942, we added an option in the connector configuration interface to enable or disable schema evolution:
image

This option, once enabled, sets the following parameter to the COPY INTO statement used to load the data into the destination table (set to false by default): COPY_OPTIONS ('mergeSchema' = 'true')

We conducted some tests locally and it works great so far. We planned on running it in production in the coming weeks.
@gpodevijn gpodevijn added needs-triage type/enhancement New feature or request labels May 26, 2023
@igrankova igrankova changed the title The databicks destination connector should support schema evolution (schemaMerge) Destination Databicks: connector should support schema evolution (schemaMerge) Jun 1, 2023
@gpodevijn gpodevijn mentioned this issue Jun 2, 2023
Merged
@marcosmarxm marcosmarxm changed the title Destination Databicks: connector should support schema evolution (schemaMerge) Destination Databricks: connector should support schema evolution (schemaMerge) Jun 2, 2023
@evantahler
Copy link
Contributor

Nice! How's the work going internally?

@gpodevijn
Copy link
Contributor Author

gpodevijn commented Jun 6, 2023 •
edited
Loading

Thanks @evantahler !
The team reviewed the associated PR before we opened it here. We made several tests locally but it's not yet running in production (because we don't run Airbyte in production yet), but it's planned for the up coming weeks. I will update the original issue with the link to the PR.

marcosmarxm pushed a commit that referenced this issue Jun 8, 2023
@gpodevijn
✨ Destination Databricks: connector supports schema evolution (schema…
eb14b96 
…Merge) #26658 (#26942)

* https://prima-assicurazioni-spa.myjetbrains.com/youtrack/issue/SSDP-165/Enable-schema-evolution-in-the-Airbyte-Databricks-connector (#1)

* Ssdp 165/enable schema evolution in the airbyte databricks connector (#4)

* https://prima-assicurazioni-spa.myjetbrains.com/youtrack/issue/SSDP-165/Enable-schema-evolution-in-the-Airbyte-Databricks-connector

* bump minor version

* bump minor version

* Ssdp 165/add doc (#5)

* https://prima-assicurazioni-spa.myjetbrains.com/youtrack/issue/SSDP-165/Enable-schema-evolution-in-the-Airbyte-Databricks-connector

* bump minor version

* bump minor version

* Edit connector doc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/connectors Connector related issues community connectors/destination/databricks connectors/destinations-warehouse releaseStage/alpha team/destinations Destinations team's backlog type/enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

✨ Destination Databricks: connector supports schema evolution (schemaMerge) #26658 primait/airbyte
4 participants
@gpodevijn @evantahler @igrankova @octavia-squidington-iii