Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Normalization: Upgrade MySQL to dbt 1.0.0 #11470

Merged
merged 23 commits into from
Jun 15, 2022
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
6ac01c0
Upgrade MySQL to dbt 1.0.0
alafanechere Mar 28, 2022
111580c
update changelog
alafanechere Mar 28, 2022
ca38ffa
update dbt_project.yml and packages.yml
alafanechere Mar 29, 2022
cb6b9b3
Merge branch 'master' into augustin/normalization/upgrade-mysql-adapter
alafanechere Mar 29, 2022
3b1bc29
fix entrypoint
alafanechere Mar 30, 2022
ca450c1
Merge branch 'master' into augustin/normalization/upgrade-mysql-adapter
alafanechere Mar 31, 2022
afe752a
Merge branch 'master' into augustin/normalization/upgrade-mysql-adapter
alafanechere Apr 1, 2022
0971ef7
Merge branch 'master' into augustin/normalization/upgrade-mysql-adapter
edgao Apr 5, 2022
0c0cb5d
fix concat + regenerate output
edgao Apr 7, 2022
58ee603
tmp max workers = 1
alafanechere Apr 11, 2022
ad5ead5
Merge branch 'master' into augustin/normalization/upgrade-mysql-adapter
alafanechere Apr 11, 2022
e4bb4ea
Merge branch 'master' into augustin/normalization/upgrade-mysql-adapter
alafanechere Apr 12, 2022
ed93465
Merge branch 'master' into augustin/normalization/upgrade-mysql-adapter
edgao Apr 12, 2022
4402c20
rebump
edgao Apr 12, 2022
7bf48fd
Merge branch 'master' into augustin/normalization/upgrade-mysql-adapter
alafanechere Apr 20, 2022
ab9cde8
Merge branch 'master' into augustin/normalization/upgrade-mysql-adapter
edgao Apr 20, 2022
355adfe
Merge branch 'master' into augustin/normalization/upgrade-mysql-adapter
alafanechere Apr 22, 2022
6b5b34c
explicit airbyteDocker before runnig tests?
edgao Apr 22, 2022
37cfd39
only run normalization docker?
edgao Apr 22, 2022
61ab575
Merge branch 'master' into augustin/normalization/upgrade-mysql-adapter
alafanechere May 19, 2022
ad7ad53
Merge branch 'master' into augustin/normalization/upgrade-mysql-adapter
edgao Jun 15, 2022
8a7c1d4
regenerate mysql output
edgao Jun 15, 2022
d72be95
bump version
edgao Jun 15, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -401,7 +401,7 @@ def run_check_dbt_command(normalization_image: str, command: str, cwd: str, forc
"""
Run dbt subprocess while checking and counting for "ERROR", "FAIL" or "WARNING" printed in its outputs
"""
if normalization_image.startswith("airbyte/normalization-oracle") or normalization_image.startswith("airbyte/normalization-mysql"):
if normalization_image.startswith("airbyte/normalization-oracle"):
dbtAdditionalArgs = []
else:
dbtAdditionalArgs = ["--event-buffer-size=10000"]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
# As of today, dbt-mysql doesn't support 1.0.0
# IF YOU UPGRADE DBT, make sure to also edit these files:
# 1. Remove the "normalization-mysql" entry here https://github.com/airbytehq/airbyte/pull/11267/files#diff-9a3bcae8cb5c56aa30c00548e06eade6ad771f3d4f098f6867ae9a183049dfd8R404
# 2. Check if oracle.Dockerfile is on DBT 1.0.0 yet; if it is, then revert this entire edit https://github.com/airbytehq/airbyte/pull/11267/files#diff-8880e85b2b5690accc6f15f9292a8589a6eb83564803d57c4ee74e2ee8ede09eR117-R130
FROM fishtownanalytics/dbt:0.19.0
FROM fishtownanalytics/dbt:1.0.0
COPY --from=airbyte/base-airbyte-protocol-python:0.1.1 /airbyte /airbyte

# Install SSH Tunneling dependencies
Expand All @@ -24,8 +20,7 @@ RUN pip install .

WORKDIR /airbyte/normalization_code
RUN pip install .
# Based of https://github.com/dbeatty10/dbt-mysql/tree/dev/0.19.0
RUN pip install dbt-mysql==0.19.0
RUN pip install dbt-mysql==1.0.0

WORKDIR /airbyte/normalization_code/dbt-template/
# Download external dbt dependencies
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
public class NormalizationRunnerFactory {

public static final String BASE_NORMALIZATION_IMAGE_NAME = "airbyte/normalization";
public static final String NORMALIZATION_VERSION = "0.1.73";
public static final String NORMALIZATION_VERSION = "0.1.74";

static final Map<String, ImmutablePair<String, DefaultNormalizationRunner.DestinationType>> NORMALIZATION_MAPPING =
ImmutableMap.<String, ImmutablePair<String, DefaultNormalizationRunner.DestinationType>>builder()
Expand Down
1 change: 1 addition & 0 deletions docs/understanding-airbyte/basic-normalization.md
Original file line number Diff line number Diff line change
Expand Up @@ -350,6 +350,7 @@ Therefore, in order to "upgrade" to the desired normalization version, you need

| Airbyte Version | Normalization Version | Date | Pull Request | Subject |
|:----------------| :--- | :--- | :--- | :--- |
| 0.35.60-alpha | 0.1.74 | 2022-03-28 | [\#11470](https://github.com/airbytehq/airbyte/pull/11470) | Upgrade MySQL to dbt 1.0.0 |
| 0.35.60-alpha | 0.1.73 | 2022-03-25 | [\#11267](https://github.com/airbytehq/airbyte/pull/11267) | Set `--event-buffer-size` to reduce memory usage |
| 0.35.59-alpha | 0.1.72 | 2022-03-24 | [\#11093](https://github.com/airbytehq/airbyte/pull/11093) | Added Snowflake OAuth2.0 support |
| 0.35.53-alpha | 0.1.71 | 2022-03-14 | [\#11077](https://github.com/airbytehq/airbyte/pull/11077) | Enable BigQuery to handle project ID embedded inside dataset ID |
Expand Down