Enable CustomDbtNormalization on k8s #238
Conversation
The purpose of this modifications is to "re-clone" the repo inside the pod if the folder doesn't exists. It reuses the structure of the NormalizationRunner. At this points it uses a custom script to create the profile of bigquery destination. The better form is to reuse the transform-config.py script to create the dbt profile.
|
@davinchia Can this be used? |
|
How can we benefit from this update? We would love to run dbt transformation on k8s. |
|
Small bump on this, would love to have this merged ! |
Is it true? |
It's for internal usage, dbt invocation and the end of pipeline should still work |
|
|
|
Your branch is not currently up-to-date with |
|
This feature was deprecated. |
|
Your branch is not currently up-to-date with |
Solves the issue 5091 - now, only for bigquery destination.
The purpose of this modifications is to "re-clone" the repo inside the pod if the folder doesn't exists. It reuses the structure of the NormalizationRunner.
At this points it uses a custom script to create the profile of bigquery destination. The better form is to reuse the transform-config script to create the dbt profile (pls someone should help me on that).
What
Describe what the change is solving
It helps to add screenshots if it affects the frontend.
Enable CustomDbtNormalization on k8s
How
Describe the solution
Added the NormalizationRunner steps to DbtNormalizationRunner and enforced the creation of git_repo inside the normalization pod.
Recommended reading order
x.javay.javaCan this PR be safely reverted / rolled back?
If you know that your PR is backwards-compatible and can be simply reverted or rolled back, check the YES box.
Otherwise if your PR has a breaking change, like a database migration for example, check the NO box.
If unsure, leave it blank.
🚨 User Impact 🚨
Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.