Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A/B tests with package sync + repeats #355

Merged
merged 11 commits into from
Sep 22, 2022
Merged

A/B tests with package sync + repeats #355

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
45f0661
Repeat A/B tests
crusaderky Sep 15, 2022
c778881
Merge from main
crusaderky Sep 20, 2022
c58806c
Throttle concurrent runs
crusaderky Sep 20, 2022
1544518
tweaks
crusaderky Sep 20, 2022
79f4fa1
Add upstream
crusaderky Sep 20, 2022
245917a
Merge branch 'main' into guido/AB_package_sync
crusaderky Sep 20, 2022
a0a4ac3
Env variables
crusaderky Sep 20, 2022
cce60d0
temp enable A/B
crusaderky Sep 20, 2022
603a55f
fix null hypothesis
crusaderky Sep 20, 2022
911f0c6
revert temp changes
crusaderky Sep 20, 2022
a7b1281
Merge branch 'main' into guido/AB_package_sync
crusaderky Sep 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
297 changes: 57 additions & 240 deletions .github/workflows/ab_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,101 +20,52 @@ jobs:
name: Discover A/B environments
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
- uses: actions/setup-python@v4

- name: Install Python
uses: actions/setup-python@v4
with:
python-version: '3.10'
- id: set-matrix

- name: Install dependencies
run: pip install PyYaml

- name: Generate dynamic matrix
id: set-matrix
run: echo "::set-output name=matrix::$(python ci/scripts/discover_ab_environments.py)"

outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
matrix: ${{ steps.set-matrix.outputs.matrix }}

# Everything below this point runs iff there are files matching
# AB_environments/AB_*.conda.yaml
# AB_environments/AB_*.dask.yaml
# AB_environments/AB_*.{conda,dask}.yaml
# and AB_environments/config.yaml set repeat > 0

software:
name: Setup
runs-on: ubuntu-latest
tests:
name: A/B Tests - ${{ matrix.category }} ${{ matrix.runtime-version }} ${{ matrix.os }} py${{ matrix.python-version }}
needs: discover_ab_envs
if: ${{ fromJson(needs.discover_ab_envs.outputs.matrix) }}
strategy:
fail-fast: false
matrix:
python-version: ["3.9"]
runtime-version: ${{ fromJson(needs.discover_ab_envs.outputs.matrix) }}

steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0

- name: Set up environment
uses: conda-incubator/setup-miniconda@v2
with:
miniforge-variant: Mambaforge
use-mamba: true
condarc-file: ci/condarc
python-version: ${{ matrix.python-version }}
environment-file: ci/environment.yml

- name: Build Coiled Software Environment
env:
DASK_COILED__TOKEN: ${{ secrets.COILED_BENCHMARK_BOT_TOKEN }}
run: |
PYTHON_VERSION_FORMATTED=$(echo "${{ matrix.python-version }}" | sed 's/\.//g' )
NAME_HEAD=dask-engineering/coiled-runtime-${{ github.event_name }}
NAME_TAIL=$GITHUB_RUN_ID-${{ matrix.runtime-version }}-py$PYTHON_VERSION_FORMATTED
if [[ ${{ github.event_name }} = 'pull_request' ]]
then
NAME_MID=${{ github.event.number }}
else
NAME_MID=$GITHUB_REF_TYPE-$(echo "$GITHUB_REF_NAME" | sed 's/\./-/g' )
fi
# env name can only contain lowercase ASCII letters, numbers, hyphens and underscores
COILED_SOFTWARE_NAME=$NAME_HEAD-$(echo $NAME_MID-$NAME_TAIL | tr 'A-Z' 'a-z' | sed -r 's/[^a-z0-9_-]/_/g')

cp AB_environments/${{ matrix.runtime-version }}.conda.yaml coiled_software_environment.yaml
COILED_SOFTWARE_ENV=$(python ci/scripts/dask_config_to_env.py AB_environments/${{ matrix.runtime-version }}.dask.yaml)
ENV_FILE=coiled_software_environment.yaml
cat $ENV_FILE

mamba install coiled
echo "Creating Coiled software environment for $COILED_SOFTWARE_NAME"
echo "Environment parameters: $COILED_SOFTWARE_ENV"
coiled env create --name $COILED_SOFTWARE_NAME --conda $ENV_FILE $COILED_SOFTWARE_ENV

# Put COILED_SOFTWARE_NAME into a file so it can be downloaded in subsequent workflow jobs
echo $COILED_SOFTWARE_NAME > software_name.txt

# Dummy for compatibility with tests.yml
echo false > test_upstream.txt

- name: Upload environment file
uses: actions/upload-artifact@v3
with:
name: software-environment-${{ matrix.runtime-version }}-py${{ matrix.python-version }}
path: |
coiled_software_environment.yaml
software_name.txt
test_upstream.txt

runtime:
name: Runtime - ${{ matrix.os }}, Python ${{ matrix.python-version }}, Runtime ${{ matrix.runtime-version }}
needs: [discover_ab_envs, software]
runs-on: ${{ matrix.os }}
timeout-minutes: 120
strategy:
fail-fast: false
# AWS implements limiters to how many EC2 instances you can spawn in parallel *on
# the same AWS account*. If such limit is reached, jobs will randomly fail when
# trying to create the Coiled clusters, and restarting failed jobs won't fix the
# problem.
max-parallel: 20
matrix:
os: [ubuntu-latest]
python-version: ["3.9"]
runtime-version: ${{ fromJson(needs.discover_ab_envs.outputs.matrix) }}
category: ${{ fromJson(needs.discover_ab_envs.outputs.matrix).category }}
runtime-version: ${{ fromJson(needs.discover_ab_envs.outputs.matrix).runtime }}
repeat: ${{ fromJson(needs.discover_ab_envs.outputs.matrix).repeat }}

steps:
- uses: actions/checkout@v2
- name: Checkout
uses: actions/checkout@v2
with:
fetch-depth: 0

Expand All @@ -127,16 +78,21 @@ jobs:
python-version: ${{ matrix.python-version }}
environment-file: ci/environment.yml

- name: Download software environment assets
if: matrix.runtime-version == 'latest' || startsWith(matrix.runtime-version, 'AB_')
uses: actions/download-artifact@v3
with:
name: software-environment-${{ matrix.runtime-version }}-py${{ matrix.python-version }}
- name: Create null hypothesis as a copy of baseline
if: matrix.runtime-version == 'AB_null_hypothesis'
run: |
cd AB_environments
cp AB_baseline.conda.yaml AB_null_hypothesis.conda.yaml
cp AB_baseline.dask.yaml AB_null_hypothesis.dask.yaml

- name: Install coiled-runtime
env:
COILED_RUNTIME_VERSION: ${{ matrix.runtime-version }}
run: source ci/scripts/install_coiled_runtime.sh
run: |
source ci/scripts/install_coiled_runtime.sh AB_environments/${{ matrix.runtime-version }}.conda.yaml

- name: Convert dask config into environment variables
run: python ci/scripts/dask_config_to_env.py AB_environments/${{ matrix.runtime-version }}.dask.yaml >> $GITHUB_ENV

- name: Run Coiled Runtime Tests
id: test
Expand All @@ -145,181 +101,40 @@ jobs:
AWS_ACCESS_KEY_ID: ${{ secrets.RUNTIME_CI_BOT_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.RUNTIME_CI_BOT_AWS_SECRET_ACCESS_KEY }}
COILED_RUNTIME_VERSION: ${{ matrix.runtime-version }}
DB_NAME: runtime-${{ matrix.os }}-${{ matrix.runtime-version }}-py${{ matrix.python-version }}.db
BENCHMARK: true
run: bash ci/scripts/run_tests.sh tests/runtime

- name: Upload benchmark results
uses: actions/upload-artifact@v3
if: always()
with:
name: runtime-${{ matrix.os }}-${{ matrix.runtime-version }}-py${{ matrix.python-version }}
path: runtime-${{ matrix.os }}-${{ matrix.runtime-version }}-py${{ matrix.python-version }}.db

benchmarks:
name: Benchmarks - ${{ matrix.os }}, Python ${{ matrix.python-version }}, Runtime ${{ matrix.runtime-version }}
needs: [discover_ab_envs, software]
runs-on: ${{ matrix.os }}
timeout-minutes: 120
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
python-version: ["3.9"]
runtime-version: ${{ fromJson(needs.discover_ab_envs.outputs.matrix) }}

steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0

- name: Set up environment
uses: conda-incubator/setup-miniconda@v2
with:
miniforge-variant: Mambaforge
use-mamba: true
condarc-file: ci/condarc
python-version: ${{ matrix.python-version }}
environment-file: ci/environment.yml

- name: Download software environment assets
uses: actions/download-artifact@v3
with:
name: software-environment-${{ matrix.runtime-version }}-py${{ matrix.python-version }}

- name: Install coiled-runtime
env:
COILED_RUNTIME_VERSION: ${{ matrix.runtime-version }}
run: source ci/scripts/install_coiled_runtime.sh

- name: Run benchmarking tests
id: benchmarking_tests
env:
DASK_COILED__TOKEN: ${{ secrets.COILED_BENCHMARK_BOT_TOKEN }}
AWS_ACCESS_KEY_ID: ${{ secrets.RUNTIME_CI_BOT_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.RUNTIME_CI_BOT_AWS_SECRET_ACCESS_KEY }}
COILED_RUNTIME_VERSION: ${{ matrix.runtime-version }}
DB_NAME: benchmark-${{ matrix.os }}-${{ matrix.runtime-version }}-py${{ matrix.python-version }}.db
BENCHMARK: true
run: bash ci/scripts/run_tests.sh tests/benchmarks

- name: Upload benchmark results
uses: actions/upload-artifact@v3
if: always()
with:
name: benchmark-${{ matrix.os }}-${{ matrix.runtime-version }}-py${{ matrix.python-version }}
path: benchmark-${{ matrix.os }}-${{ matrix.runtime-version }}-py${{ matrix.python-version }}.db

stability:
name: Stability - ${{ matrix.os }}, Python ${{ matrix.python-version }}, Runtime ${{ matrix.runtime-version }}
needs: [discover_ab_envs, software]
runs-on: ${{ matrix.os }}
timeout-minutes: 120
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
python-version: ["3.9"]
runtime-version: ${{ fromJson(needs.discover_ab_envs.outputs.matrix) }}

steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0

- name: Set up environment
uses: conda-incubator/setup-miniconda@v2
with:
miniforge-variant: Mambaforge
use-mamba: true
condarc-file: ci/condarc
python-version: ${{ matrix.python-version }}
environment-file: ci/environment.yml

- name: Download software environment assets
if: matrix.runtime-version == 'latest' || startsWith(matrix.runtime-version, 'AB_')
uses: actions/download-artifact@v3
with:
name: software-environment-${{ matrix.runtime-version }}-py${{ matrix.python-version }}

- name: Install coiled-runtime
env:
COILED_RUNTIME_VERSION: ${{ matrix.runtime-version }}
run: source ci/scripts/install_coiled_runtime.sh

- name: Run stability tests
id: stability_tests
env:
DASK_COILED__TOKEN: ${{ secrets.COILED_BENCHMARK_BOT_TOKEN }}
AWS_ACCESS_KEY_ID: ${{ secrets.RUNTIME_CI_BOT_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.RUNTIME_CI_BOT_AWS_SECRET_ACCESS_KEY }}
COILED_RUNTIME_VERSION: ${{ matrix.runtime-version }}
DB_NAME: stability-${{ matrix.os }}-${{ matrix.runtime-version }}-py${{ matrix.python-version }}.db
DB_NAME: ${{ matrix.category }}-${{ matrix.os }}-${{ matrix.runtime-version }}-${{ matrix.repeat }}-py${{ matrix.python-version }}.db
BENCHMARK: true
CLUSTER_DUMP: true
run: bash ci/scripts/run_tests.sh tests/stability
run: bash ci/scripts/run_tests.sh tests/${{ matrix.category }}

- name: Upload benchmark results
uses: actions/upload-artifact@v3
if: always()
with:
name: stability-${{ matrix.os }}-${{ matrix.runtime-version }}-py${{ matrix.python-version }}
path: stability-${{ matrix.os }}-${{ matrix.runtime-version }}-py${{ matrix.python-version }}.db

cleanup:
needs: [discover_ab_envs, software, runtime, benchmarks, stability]
if: always() && ${{ fromJson(needs.discover_ab_envs.outputs.matrix) }}
name: Cleanup
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.9"]
runtime-version: ${{ fromJson(needs.discover_ab_envs.outputs.matrix) }}

steps:
- uses: actions/checkout@v2

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}

- name: Install coiled
run: python -m pip install coiled

- name: Download software environment assets
uses: actions/download-artifact@v3
with:
name: software-environment-${{ matrix.runtime-version }}-py${{ matrix.python-version }}

- name: Remove Coiled software environment
env:
DASK_COILED__TOKEN: ${{ secrets.COILED_BENCHMARK_BOT_TOKEN }}
run: |
SOFTWARE_NAME=$(cat software_name.txt)
echo "Deleting $SOFTWARE_NAME"
coiled env delete $SOFTWARE_NAME
name: ${{ matrix.category }}-${{ matrix.os }}-${{ matrix.runtime-version }}-${{ matrix.repeat }}-py${{ matrix.python-version }}
path: ${{ matrix.category }}-${{ matrix.os }}-${{ matrix.runtime-version }}-${{ matrix.repeat }}-py${{ matrix.python-version }}.db

process-results:
needs: [discover_ab_envs, runtime, benchmarks, stability]
needs: [discover_ab_envs, tests]
name: Combine separate benchmark results
if: always() && ${{ fromJson(needs.discover_ab_envs.outputs.matrix) }}
if: always() && ${{ fromJson(needs.discover_ab_envs.outputs.matrix).runtime }}
runs-on: ubuntu-latest
concurrency:
# Fairly strict concurrency rule to avoid stepping on benchmark db.
# Could eventually replace with a real db in coiled, RDS, or litestream
group: process-benchmarks
cancel-in-progress: false
steps:
- uses: actions/checkout@v2
- name: Checkout
uses: actions/checkout@v2

- uses: actions/setup-python@v4
- name: Install Python
uses: actions/setup-python@v4

- name: Install dependencies
run: pip install alembic

- uses: actions/download-artifact@v3
- name: Download artifacts
uses: actions/download-artifact@v3
with:
path: benchmarks

Expand All @@ -331,23 +146,25 @@ jobs:
- name: Upload benchmark results as artifact
uses: actions/upload-artifact@v3
with:
name: benchmark.db
name: benchmark
path: benchmark.db

static-site:
needs: [discover_ab_envs, process-results]
# Always generate the site, as this can be skipped even if an indirect dependency fails (like a test run)
if: always() && ${{ fromJson(needs.discover_ab_envs.outputs.matrix) }}
if: always() && ${{ fromJson(needs.discover_ab_envs.outputs.matrix).runtime }}
name: Build static dashboards
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Checkout
uses: actions/checkout@v2
with:
fetch-depth: 0

- uses: actions/download-artifact@v3
- name: Download artifacts
uses: actions/download-artifact@v3
with:
name: benchmark.db
name: benchmark

- name: Set up environment
uses: conda-incubator/setup-miniconda@v2
Expand Down
Loading