Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration workflow update to support all-in-one adapter testing #1309

Merged
merged 13 commits into from
Aug 1, 2024
20 changes: 0 additions & 20 deletions .github/scripts/update_dbt_core_branch.sh

This file was deleted.

21 changes: 21 additions & 0 deletions .github/scripts/update_dev_dependency_branches.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/bin/bash -e
set -e


dbt_adapters_branch=$1
dbt_core_branch=$2
dbt_common_branch=$3
target_req_file="dev-requirements.txt"
core_req_sed_pattern="s|dbt-core.git.*#egg=dbt-core|dbt-core.git@${dbt_core_branch}#egg=dbt-core|g"
adapters_req_sed_pattern="s|dbt-adapters.git|dbt-adapters.git@${dbt_adapters_branch}|g"
common_req_sed_pattern="s|dbt-common.git|dbt-common.git@${dbt_common_branch}|g"
if [[ "$OSTYPE" == darwin* ]]; then
# mac ships with a different version of sed that requires a delimiter arg
sed -i "" "$adapters_req_sed_pattern" $target_req_file
sed -i "" "$core_req_sed_pattern" $target_req_file
sed -i "" "$common_req_sed_pattern" $target_req_file
else
sed -i "$adapters_req_sed_pattern" $target_req_file
sed -i "$core_req_sed_pattern" $target_req_file
sed -i "$common_req_sed_pattern" $target_req_file
fi
60 changes: 50 additions & 10 deletions .github/workflows/integration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@

name: Adapter Integration Tests

run-name: "${{ (contains(github.event_name, 'workflow_') && inputs.name) || github.event_name }}: ${{ (contains(github.event_name, 'workflow_') && inputs.adapter_branch) || github.ref_name }} by @${{ github.actor }}"

on:
# pushes to release branches
push:
Expand All @@ -34,10 +36,31 @@ on:
# manual trigger
workflow_dispatch:
inputs:
dbt-core-branch:
description: "branch of dbt-core to use in dev-requirements.txt"
name:
description: "Name to associate with run (example: 'dbt-adapters-242')"
required: false
type: string
default: "Adapter Integration Tests"
adapter_branch:
description: "The branch of this adapter repository to use"
type: string
required: false
default: "main"
dbt_adapters_branch:
description: "The branch of dbt-adapters to use"
type: string
required: false
default: "main"
dbt_core_branch:
description: "The branch of dbt-core to use"
type: string
required: false
default: "main"
dbt_common_branch:
description: "The branch of dbt-common to use"
type: string
required: false
default: "main"

# explicitly turn off permissions for `GITHUB_TOKEN`
permissions: read-all
Expand Down Expand Up @@ -67,12 +90,19 @@ jobs:
run-python-tests: ${{ steps.filter.outputs.bigquery-python }}

steps:
- name: Check out the repository (non-PR)
if: github.event_name != 'pull_request_target'
- name: Check out the repository (push)
if: github.event_name == 'push'
uses: actions/checkout@v4
with:
persist-credentials: false

- name: Check out the repository (workflow_dispatch)
if: github.event_name == 'workflow_dispatch'
uses: actions/checkout@v4
with:
persist-credentials: false
ref: ${{ inputs.adapter_branch }}

- name: Check out the repository (PR)
if: github.event_name == 'pull_request_target'
uses: actions/checkout@v4
Expand Down Expand Up @@ -143,12 +173,19 @@ jobs:

steps:
- name: Check out the repository
if: github.event_name != 'pull_request_target'
if: github.event_name != 'pull_request_target' && github.event_name != 'workflow_dispatch'
uses: actions/checkout@v4
with:
persist-credentials: false

# explicity checkout the branch for the PR,
- name: Check out the repository (workflow_dispatch)
if: github.event_name == 'workflow_dispatch'
uses: actions/checkout@v4
with:
persist-credentials: false
ref: ${{ inputs.adapter_branch }}

# explicitly checkout the branch for the PR,
# this is necessary for the `pull_request_target` event
- name: Check out the repository (PR)
if: github.event_name == 'pull_request_target'
Expand All @@ -169,11 +206,14 @@ jobs:
python -m pip --version
tox --version

- name: Update dev_requirements.txt
if: inputs.dbt-core-branch != ''
- name: Update Adapters and Core branches (update dev_requirements.txt)
if: github.event_name == 'workflow_dispatch'
run: |
pip install bumpversion
./.github/scripts/update_dbt_core_branch.sh ${{ inputs.dbt-core-branch }}
./.github/scripts/update_dev_dependency_branches.sh \
${{ inputs.dbt_adapters_branch }} \
${{ inputs.dbt_core_branch }} \
${{ inputs.dbt_common_branch }}
cat dev-requirements.txt

- name: Run tox (bigquery)
if: matrix.adapter == 'bigquery'
Expand Down