Skip to content

Commit

Permalink
Merge pull request #35 from fivetran/MagicBot/dbt-utils-cross-db-migr…
Browse files Browse the repository at this point in the history
…ation

Updates for dbt-utils to dbt-core cross-db macro migration
  • Loading branch information
fivetran-joemarkiewicz authored Dec 20, 2022
2 parents 0054940 + 2bb98ba commit 8275ff6
Show file tree
Hide file tree
Showing 23 changed files with 345 additions and 185 deletions.
24 changes: 24 additions & 0 deletions .buildkite/hooks/pre-command
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#!/bin/bash

set -e

# Export secrets for Docker containers.
# Restrict exposing secrets only to the steps that need them
export GCLOUD_SERVICE_KEY=$(gcloud secrets versions access latest --secret="GCLOUD_SERVICE_KEY" --project="dbt-package-testing-363917")
export CI_POSTGRES_DBT_HOST=$(gcloud secrets versions access latest --secret="CI_POSTGRES_DBT_HOST" --project="dbt-package-testing-363917")
export CI_POSTGRES_DBT_USER=$(gcloud secrets versions access latest --secret="CI_POSTGRES_DBT_USER" --project="dbt-package-testing-363917")
export CI_POSTGRES_DBT_PASS=$(gcloud secrets versions access latest --secret="CI_POSTGRES_DBT_PASS" --project="dbt-package-testing-363917")
export CI_POSTGRES_DBT_DBNAME=$(gcloud secrets versions access latest --secret="CI_POSTGRES_DBT_DBNAME" --project="dbt-package-testing-363917")
export CI_REDSHIFT_DBT_DBNAME=$(gcloud secrets versions access latest --secret="CI_REDSHIFT_DBT_DBNAME" --project="dbt-package-testing-363917")
export CI_REDSHIFT_DBT_HOST=$(gcloud secrets versions access latest --secret="CI_REDSHIFT_DBT_HOST" --project="dbt-package-testing-363917")
export CI_REDSHIFT_DBT_PASS=$(gcloud secrets versions access latest --secret="CI_REDSHIFT_DBT_PASS" --project="dbt-package-testing-363917")
export CI_REDSHIFT_DBT_USER=$(gcloud secrets versions access latest --secret="CI_REDSHIFT_DBT_USER" --project="dbt-package-testing-363917")
export CI_SNOWFLAKE_DBT_ACCOUNT=$(gcloud secrets versions access latest --secret="CI_SNOWFLAKE_DBT_ACCOUNT" --project="dbt-package-testing-363917")
export CI_SNOWFLAKE_DBT_DATABASE=$(gcloud secrets versions access latest --secret="CI_SNOWFLAKE_DBT_DATABASE" --project="dbt-package-testing-363917")
export CI_SNOWFLAKE_DBT_PASS=$(gcloud secrets versions access latest --secret="CI_SNOWFLAKE_DBT_PASS" --project="dbt-package-testing-363917")
export CI_SNOWFLAKE_DBT_ROLE=$(gcloud secrets versions access latest --secret="CI_SNOWFLAKE_DBT_ROLE" --project="dbt-package-testing-363917")
export CI_SNOWFLAKE_DBT_USER=$(gcloud secrets versions access latest --secret="CI_SNOWFLAKE_DBT_USER" --project="dbt-package-testing-363917")
export CI_SNOWFLAKE_DBT_WAREHOUSE=$(gcloud secrets versions access latest --secret="CI_SNOWFLAKE_DBT_WAREHOUSE" --project="dbt-package-testing-363917")
export CI_DATABRICKS_DBT_HOST=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_HOST" --project="dbt-package-testing-363917")
export CI_DATABRICKS_DBT_HTTP_PATH=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_HTTP_PATH" --project="dbt-package-testing-363917")
export CI_DATABRICKS_DBT_TOKEN=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_TOKEN" --project="dbt-package-testing-363917")
58 changes: 58 additions & 0 deletions .buildkite/pipeline.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
steps:
- label: ":snowflake-db: Run Tests - Snowflake"
key: "run_dbt_snowflake"
plugins:
- docker#v3.13.0:
image: "python:3.8"
shell: [ "/bin/bash", "-e", "-c" ]
environment:
- "BASH_ENV=/tmp/.bashrc"
- "CI_SNOWFLAKE_DBT_ACCOUNT"
- "CI_SNOWFLAKE_DBT_DATABASE"
- "CI_SNOWFLAKE_DBT_PASS"
- "CI_SNOWFLAKE_DBT_ROLE"
- "CI_SNOWFLAKE_DBT_USER"
- "CI_SNOWFLAKE_DBT_WAREHOUSE"
commands: |
bash .buildkite/scripts/run_models.sh snowflake
- label: ":gcloud: Run Tests - BigQuery"
key: "run_dbt_bigquery"
plugins:
- docker#v3.13.0:
image: "python:3.8"
shell: [ "/bin/bash", "-e", "-c" ]
environment:
- "BASH_ENV=/tmp/.bashrc"
- "GCLOUD_SERVICE_KEY"
commands: |
bash .buildkite/scripts/run_models.sh bigquery
- label: ":amazon-redshift: Run Tests - Redshift"
key: "run_dbt_redshift"
plugins:
- docker#v3.13.0:
image: "python:3.8"
shell: [ "/bin/bash", "-e", "-c" ]
environment:
- "BASH_ENV=/tmp/.bashrc"
- "CI_REDSHIFT_DBT_DBNAME"
- "CI_REDSHIFT_DBT_HOST"
- "CI_REDSHIFT_DBT_PASS"
- "CI_REDSHIFT_DBT_USER"
commands: |
bash .buildkite/scripts/run_models.sh redshift
- label: ":bricks: Run Tests - Databricks"
key: "run_dbt_databricks"
plugins:
- docker#v3.13.0:
image: "python:3.8"
shell: [ "/bin/bash", "-e", "-c" ]
environment:
- "BASH_ENV=/tmp/.bashrc"
- "CI_DATABRICKS_DBT_HOST"
- "CI_DATABRICKS_DBT_HTTP_PATH"
- "CI_DATABRICKS_DBT_TOKEN"
commands: |
bash .buildkite/scripts/run_models.sh databricks
23 changes: 23 additions & 0 deletions .buildkite/scripts/run_models.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/bin/bash

set -euo pipefail

apt-get update
apt-get install libsasl2-dev

python3 -m venv venv
. venv/bin/activate
pip install --upgrade pip setuptools
pip install -r integration_tests/requirements.txt
mkdir -p ~/.dbt
cp integration_tests/ci/sample.profiles.yml ~/.dbt/profiles.yml

db=$1
echo `pwd`
cd integration_tests
dbt deps
dbt seed --target "$db" --full-refresh
dbt run --target "$db" --full-refresh
dbt test --target "$db"
dbt run --vars '{github__using_repo_team: false}' --target "$db" --full-refresh
dbt test --target "$db"
64 changes: 0 additions & 64 deletions .circleci/config.yml

This file was deleted.

5 changes: 2 additions & 3 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
Pull Request
**Are you a current Fivetran customer?**
<!--- Please tell us your name, title and company -->

Expand Down Expand Up @@ -29,9 +28,9 @@ Pull Request

**How did you test the PR changes?**
<!--- Proof of testing is required in order for the PR to be approved. -->
<!--- To check a box, remove the space and insert an x in the box (eg. [x] CircleCi). -->
<!--- To check a box, remove the space and insert an x in the box (eg. [x] Buildkite). -->
<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
- [ ] CircleCi <!--- CircleCi testing is only applicable to Fivetran employees. -->
- [ ] Buildkite <!--- Buildkite testing is only applicable to Fivetran employees. -->
- [ ] Local (please provide additional testing details below)

**Select which warehouse(s) were used to test the PR**
Expand Down
41 changes: 41 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,44 @@
# dbt_github v0.6.0
[PR #35](https://github.com/fivetran/dbt_github/pull/35) includes the following breaking changes:
## 🚨 Breaking Changes 🚨:
- Dispatch update for dbt-utils to dbt-core cross-db macros migration. Specifically `{{ dbt_utils.<macro> }}` have been updated to `{{ dbt.<macro> }}` for the below macros:
- `any_value`
- `bool_or`
- `cast_bool_to_text`
- `concat`
- `date_trunc`
- `dateadd`
- `datediff`
- `escape_single_quotes`
- `except`
- `hash`
- `intersect`
- `last_day`
- `length`
- `listagg`
- `position`
- `replace`
- `right`
- `safe_cast`
- `split_part`
- `string_literal`
- `type_bigint`
- `type_float`
- `type_int`
- `type_numeric`
- `type_string`
- `type_timestamp`
- `array_append`
- `array_concat`
- `array_construct`
- For `current_timestamp` and `current_timestamp_in_utc` macros, the dispatch AND the macro names have been updated to the below, respectively:
- `dbt.current_timestamp_backcompat`
- `dbt.current_timestamp_in_utc_backcompat`
- Dependencies on `fivetran/fivetran_utils` have been upgraded, previously `[">=0.3.0", "<0.4.0"]` now `[">=0.4.0", "<0.5.0"]`.
## 🎉 Documentation and Feature Updates 🎉:
- Updated README documentation for easier navigation and dbt package setup. [#35](https://github.com/fivetran/dbt_github/pull/35)
- Added Databricks compatibility. [#38](https://github.com/fivetran/dbt_github/pull/38)

# dbt_github v0.5.1
## Fixes
- The `url_link` logic within `int_github__issue_joined` was focused on only providing the correct url for pull requests. This update includes a `case when` statement to provide the accurate url logic for both Issues and Pull Requests. ([#31](https://github.com/fivetran/dbt_github/pull/31))
Expand Down
Loading

0 comments on commit 8275ff6

Please sign in to comment.