Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update dbt-clickhouse version from 1.1.7 to 1.1.8 #16339

Merged
merged 18 commits into from
Sep 14, 2022

Conversation

Barsoomx
Copy link
Contributor

@Barsoomx Barsoomx commented Sep 5, 2022

What

When working with complex DBT projects, sometimes Cronjobs,Builds, and Airbyte jobs interfere with each other, and cause SELECT errors or deadlocks when tables are being recreated by one or several processes. This was addressed in dbt-clickhouse PR, updating dbt-clickhouse version in combination with ClickHouse Atomic engine and SYNC setting should help avoid those issues since normalization and table renames are performed by Airbyte with dbt-clickhouse.

How

Bump dbt-clickhouse version from 1.1.7 to 1.1.8

dbt-core is pinned at 1.1.2 with 1.2.x update pending in the coming weeks

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub by running the /publish command described here
  • After the connector is published, connector added to connector index as described here
  • Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here
Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub and connector version bumped by running the /publish command described here
Connector Generator
  • Issue acceptance criteria met
  • PR name follows PR naming conventions
  • If adding a new generator, add it to the list of scaffold modules being tested
  • The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
  • Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

@Barsoomx Barsoomx requested a review from a team as a code owner September 5, 2022 20:30
@CLAassistant
Copy link

CLAassistant commented Sep 5, 2022

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Sep 5, 2022
@marcosmarxm
Copy link
Member

marcosmarxm commented Sep 6, 2022

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/3002323859
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/3002323859
🐛 https://gradle.com/s/5g5pvvlzjl4ky

Build Failed

Test summary info:

Could not find result summary

@marcosmarxm marcosmarxm self-assigned this Sep 6, 2022
@Barsoomx
Copy link
Contributor Author

Barsoomx commented Sep 6, 2022

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.

some errors from pip because of old and unsupported dbt image(fishtown-analytics). Should I change it to xemuliam/dbt:1.1 or build an image from python-slim and see if the error persists?

@marcosmarxm
Copy link
Member

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.

some errors from pip because of old and unsupported dbt image(fishtown-analytics). Should I change it to xemuliam/dbt:1.1 or build an image from python-slim and see if the error persists?

I don't feel comfortable changing the base image. Can we try to install the clickhouse lib without forcing the dbt-core version?

@Barsoomx
Copy link
Contributor Author

Barsoomx commented Sep 9, 2022

Done. You can re-run the workflow, but i'm not too sure anything'll change

@marcosmarxm
Copy link
Member

marcosmarxm commented Sep 9, 2022

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/3022770271
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/3022770271
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         191     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1437    624    57%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         191     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1437    624    57%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/utils.py                             51      1    98%
normalization/transform_catalog/destination_name_transformer.py     166      5    97%
normalization/transform_catalog/stream_processor.py                 589     35    94%
normalization/transform_catalog/catalog_processor.py                147     12    92%
normalization/transform_catalog/dbt_macro.py                         22      3    86%
normalization/transform_catalog/table_name_registry.py              174     51    71%
normalization/transform_catalog/transform.py                         61     22    64%
normalization/transform_config/transform.py                         191     77    60%
-------------------------------------------------------------------------------------
TOTAL                                                              1437    206    86%

Build Passed

Test summary info:

	 =========================== short test summary info ============================
	 SKIPPED [1] integration_tests/test_ephemeral.py:99: ephemeral materialization isn't supported in ClickHouse yet
	 SKIPPED [1] integration_tests/test_ephemeral.py:61: Destinations DestinationType.MYSQL is not in NORMALIZATION_TEST_TARGET env variable (MYSQL is also skipped)
	 SKIPPED [1] integration_tests/test_normalization.py:143: DestinationType.CLICKHOUSE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:81: Destinations DestinationType.CLICKHOUSE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:146: DestinationType.MSSQL is disabled as it doesnt fully support schema change in incremental yet
	 SKIPPED [2] integration_tests/test_normalization.py:134: DestinationType.MYSQL does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:134: DestinationType.ORACLE does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:81: Destinations DestinationType.ORACLE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:143: DestinationType.SNOWFLAKE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:143: DestinationType.TIDB is disabled as it doesnt support schema change in incremental yet (column type changes)
	 �[32m================= �[32m�[1m26 passed�[0m, �[33m11 skipped�[0m�[32m in 3469.43s (0:57:49)�[0m�[32m ==================�[0m
	 =========================== short test summary info ============================
	 SKIPPED [1] integration_tests/test_ephemeral.py:99: ephemeral materialization isn't supported in ClickHouse yet
	 SKIPPED [1] integration_tests/test_ephemeral.py:61: Destinations DestinationType.MYSQL is not in NORMALIZATION_TEST_TARGET env variable (MYSQL is also skipped)
	 SKIPPED [1] integration_tests/test_normalization.py:143: DestinationType.CLICKHOUSE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:81: Destinations DestinationType.CLICKHOUSE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:146: DestinationType.MSSQL is disabled as it doesnt fully support schema change in incremental yet
	 SKIPPED [2] integration_tests/test_normalization.py:134: DestinationType.MYSQL does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:134: DestinationType.ORACLE does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:81: Destinations DestinationType.ORACLE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:143: DestinationType.SNOWFLAKE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:143: DestinationType.TIDB is disabled as it doesnt support schema change in incremental yet (column type changes)
	 �[32m================= �[32m�[1m26 passed�[0m, �[33m11 skipped�[0m�[32m in 3329.35s (0:55:29)�[0m�[32m ==================�[0m

@grishick
Copy link
Contributor

grishick commented Sep 10, 2022

/test connector=connectors/destination-clickhouse

🕑 connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/3026136714
✅ connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/3026136714
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         191     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1437    624    57%

Build Passed

Test summary info:

All Passed

@Barsoomx
Copy link
Contributor Author

Resolved the conflict, version bumped again. @marcosmarxm @grishick

@github-actions github-actions bot added area/platform issues related to the platform area/worker Related to worker labels Sep 13, 2022
@marcosmarxm
Copy link
Member

marcosmarxm commented Sep 13, 2022

/publish connector=bases/base-normalization

🕑 Publishing the following connectors:
bases/base-normalization
https://github.com/airbytehq/airbyte/actions/runs/3046915782


Connector Did it publish? Were definitions generated?
bases/base-normalization

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@marcosmarxm
Copy link
Member

marcosmarxm commented Sep 13, 2022

/publish connector=bases/base-normalization

🕑 Publishing the following connectors:
bases/base-normalization
https://github.com/airbytehq/airbyte/actions/runs/3047001103


Connector Did it publish? Were definitions generated?
bases/base-normalization

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@grishick
Copy link
Contributor

Looks like publish command timed out. Not sure why it took more than 4 hours though

@marcosmarxm
Copy link
Member

marcosmarxm commented Sep 13, 2022

/publish connector=bases/base-normalization

🕑 Publishing the following connectors:
bases/base-normalization
https://github.com/airbytehq/airbyte/actions/runs/3048583658


Connector Did it publish? Were definitions generated?
bases/base-normalization

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@marcosmarxm
Copy link
Member

Looks like publish command timed out. Not sure why it took more than 4 hours though

I'll try again.

@marcosmarxm marcosmarxm merged commit 17cb363 into airbytehq:master Sep 14, 2022
robbinhan pushed a commit to robbinhan/airbyte that referenced this pull request Sep 29, 2022
…bytehq#16339)

* Bump dbt-clickhouse version from 1.1.7 to 1.1.8

* pin dbt<1.2

* update doc

* pin dbt core lt 1.2

* bump normalization version

* loosen requirements

* fix md

* remove empty line

* bump normalization version in worker

* bump normalization version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
…bytehq#16339)

* Bump dbt-clickhouse version from 1.1.7 to 1.1.8

* pin dbt<1.2

* update doc

* pin dbt core lt 1.2

* bump normalization version

* loosen requirements

* fix md

* remove empty line

* bump normalization version in worker

* bump normalization version

Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/documentation Improvements or additions to documentation area/platform issues related to the platform area/worker Related to worker community normalization
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants