Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration workflow update to support all-in-one adapter testing #1149

Merged
merged 22 commits into from
Aug 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 0 additions & 20 deletions .github/scripts/update_dbt_core_branch.sh

This file was deleted.

21 changes: 21 additions & 0 deletions .github/scripts/update_dev_dependency_branches.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/bin/bash -e
set -e


dbt_adapters_branch=$1
dbt_core_branch=$2
dbt_common_branch=$3
target_req_file="dev-requirements.txt"
core_req_sed_pattern="s|dbt-core.git.*#egg=dbt-core|dbt-core.git@${dbt_core_branch}#egg=dbt-core|g"
adapters_req_sed_pattern="s|dbt-adapters.git|dbt-adapters.git@${dbt_adapters_branch}|g"
common_req_sed_pattern="s|dbt-common.git|dbt-common.git@${dbt_common_branch}|g"
if [[ "$OSTYPE" == darwin* ]]; then
# mac ships with a different version of sed that requires a delimiter arg
sed -i "" "$adapters_req_sed_pattern" $target_req_file
sed -i "" "$core_req_sed_pattern" $target_req_file
sed -i "" "$common_req_sed_pattern" $target_req_file
else
sed -i "$adapters_req_sed_pattern" $target_req_file
sed -i "$core_req_sed_pattern" $target_req_file
sed -i "$common_req_sed_pattern" $target_req_file
fi
53 changes: 43 additions & 10 deletions .github/workflows/integration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@

name: Adapter Integration Tests

run-name: "${{ (contains(github.event_name, 'workflow_') && inputs.name) || github.event_name }}: ${{ (contains(github.event_name, 'workflow_') && inputs.adapter_branch) || github.ref_name }} by @${{ github.actor }}"

on:
# pushes to release branches
push:
Expand All @@ -34,10 +36,31 @@ on:
# manual trigger
workflow_dispatch:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we anticipate other workflows (i.e. in dbt-core) wanting to run this workflow? Should we also add workflow_call?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Workflow_call just calls the action code, without actually executing in this repository. We might want to standardize on an action in a central repository, but I'm guessing it wouldn't be in dbt-snowflake. Workflow_dispatch works for cross-repo calls. I'm running these now with a script, but once the individual repository changes are in place I'm going to try to set up an action in dbt-adapters to call all of the individual adapter integration tests and see how that goes.

Copy link
Contributor

@mikealfare mikealfare Jul 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed w/ @gshank, workflow_dispatch is effectively workflow_call + the workflow repo's code in this scenario, which is what we want.

inputs:
mikealfare marked this conversation as resolved.
Show resolved Hide resolved
dbt-core-branch:
description: "branch of dbt-core to use in dev-requirements.txt"
name:
mikealfare marked this conversation as resolved.
Show resolved Hide resolved
description: "Name to associate with run (example: 'dbt-adapters-242')"
required: false
type: string
default: "Adapter Integration Tests"
adapter_branch:
description: "The branch of this adapter repository to use"
type: string
required: false
default: "main"
dbt_adapters_branch:
description: "The branch of dbt-adapters to use"
type: string
required: false
default: "main"
dbt_core_branch:
description: "The branch of dbt-core to use"
type: string
required: false
default: "main"
dbt_common_branch:
description: "The branch of dbt-common to use"
type: string
required: false
default: "main"

# explicitly turn off permissions for `GITHUB_TOKEN`
permissions: read-all
Expand Down Expand Up @@ -141,12 +164,19 @@ jobs:

steps:
- name: Check out the repository
if: github.event_name != 'pull_request_target'
if: github.event_name == 'push'
uses: actions/checkout@v4
with:
persist-credentials: false

- name: Check out the repository (workflow_dispatch)
if: github.event_name == 'workflow_dispatch'
uses: actions/checkout@v4
with:
persist-credentials: false
ref: ${{ inputs.adapter_branch }}

# explicity checkout the branch for the PR,
# explicitly checkout the branch for the PR,
# this is necessary for the `pull_request_target` event
- name: Check out the repository (PR)
if: github.event_name == 'pull_request_target'
Expand All @@ -160,19 +190,22 @@ jobs:
with:
python-version: ${{ matrix.python-version }}

- name: Update Adapters and Core branches (update dev_requirements.txt)
if: ${{ github.event_name == 'workflow_dispatch' }}
run: |
./.github/scripts/update_dev_dependency_branches.sh \
${{ inputs.dbt_adapters_branch }} \
${{ inputs.dbt_core_branch }} \
${{ inputs.dbt_common_branch }}
cat dev-requirements.txt

- name: Install python dependencies
run: |
python -m pip install --user --upgrade pip
python -m pip install tox
python -m pip --version
tox --version

- name: Update dev_requirements.txt
if: inputs.dbt-core-branch != ''
run: |
pip install bumpversion
./.github/scripts/update_dbt_core_branch.sh ${{ inputs.dbt-core-branch }}

- name: Run tox (snowflake)
if: matrix.adapter == 'snowflake'
env:
Expand Down