Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: update infrastructure to use the uv tool #1813

Merged
merged 35 commits into from
Jan 29, 2025
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
7c8fada
Change project config
egparedes Jan 20, 2025
092388b
Pre-commit changes
egparedes Jan 23, 2025
92fd293
Add dev-tasks.py
egparedes Jan 23, 2025
8602cde
Add noxfile
egparedes Jan 20, 2025
017e19d
Update GitHub CI
egparedes Jan 24, 2025
a1a772a
Update documentation
egparedes Jan 24, 2025
d302b62
WIP
egparedes Jan 27, 2025
6d022a2
WIP tests
egparedes Jan 27, 2025
18914ec
config fixes
egparedes Jan 27, 2025
2c44506
Add automatic broadcasting of package marks to all its submodules
egparedes Jan 27, 2025
eb70205
Fixes
egparedes Jan 27, 2025
dd0d491
Address some reviewer comments and other fixes
egparedes Jan 27, 2025
5353aa4
Fix CSCS CI
egparedes Jan 28, 2025
5181601
More fixes for review comments
egparedes Jan 28, 2025
3ad6f97
Merge branch 'main' into migrate-to-uv
egparedes Jan 28, 2025
8589e6a
Fixes, documentation and cleanups
egparedes Jan 28, 2025
2b4d479
Doc enhancements
egparedes Jan 28, 2025
4c58c8f
Merge branch 'main' into migrate-to-uv
egparedes Jan 28, 2025
bf7d558
Fix minimum versions of 3rd party packages
egparedes Jan 28, 2025
bf5ca9f
Fix CSCS CI
egparedes Jan 28, 2025
e663a13
WIP fix CI
egparedes Jan 28, 2025
2a66e1b
Try again to assemble the nox command line in CI
egparedes Jan 28, 2025
f389bf8
Another try
egparedes Jan 28, 2025
fab37b9
And again
egparedes Jan 28, 2025
b6711f4
Fall back to previous approach with quotes
egparedes Jan 28, 2025
dfa5b07
And even more fixes
egparedes Jan 28, 2025
7b04d69
Missing regular success exit code
egparedes Jan 28, 2025
3317146
Apply Hannes' fixes for the errors in CI
egparedes Jan 29, 2025
f7daa93
Delete default .python-version file and remove atlas4py from test dep…
egparedes Jan 29, 2025
253f234
Restore .python-version
egparedes Jan 29, 2025
5bfd53b
Fix versions again
egparedes Jan 29, 2025
895cfe8
Merge branch 'main' into migrate-to-uv
egparedes Jan 29, 2025
e1ef6f6
Fix extra-index-url
egparedes Jan 29, 2025
e63cb50
Final fixes
egparedes Jan 29, 2025
8ae39f9
Change installation strategy for atlas4py
egparedes Jan 29, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions .github/workflows/code-quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,17 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
cache: 'pip'
cache-dependency-path: |
**/pyproject.toml
**/constraints.txt
**/requirements-dev.txt
- uses: pre-commit/action@v3.0.1
python-version-file: ".python-version"

- name: Install uv
uses: astral-sh/setup-uv@v5
with:
enable-cache: true
cache-dependency-glob: "uv.lock"

- name: "Run pre-commit"
uses: pre-commit/action@v3.0.1
126 changes: 54 additions & 72 deletions .github/workflows/daily-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ on:
- cron: '0 4 * * *'
workflow_dispatch:

## COMMENTED OUT: only for testing CI action changes
## COMMENTED OUT: only for testing CI action changes.
## It only works for PRs to `main` branch from branches in the upstream gt4py repo.
# pull_request:
# branches:
# - main
Expand All @@ -15,106 +16,87 @@ jobs:
daily-ci:
strategy:
matrix:
# dependencies-strategy -> The strategy that `uv lock` should use to select
# between the different compatible versions for a given package requirement
# [arg: --resolution, env: UV_RESOLUTION=]
dependencies-strategy: ["lowest-direct", "highest"]
gt4py-module: ["cartesian", "eve", "next", "storage"]
os: ["ubuntu-latest"] #, "macos-latest"]
python-version: ["3.10", "3.11"]
tox-module-factor: ["cartesian", "eve", "next", "storage"]
os: ["ubuntu-latest"]
requirements-file: ["requirements-dev.txt", "min-requirements-test.txt", "min-extra-requirements-test.txt"]
fail-fast: false

runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4

- name: Install C++ libraries
if: ${{ matrix.os == 'macos-latest' }}
shell: bash
run: |
brew install boost
run: brew install boost

- name: Install C++ libraries
if: ${{ matrix.os == 'ubuntu-latest' }}
shell: bash
run: |
sudo apt install libboost-dev
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
run: sudo apt install libboost-dev

- name: Install uv and set the python version
uses: astral-sh/setup-uv@v5
with:
enable-cache: true
cache-dependency-glob: "uv.lock"
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: |
**/pyproject.toml
**/constraints.txt
**/requirements-dev.txt
- name: Install tox
run: |
python -m pip install -c ./constraints.txt pip setuptools wheel tox
python -m pip list
- name: Update requirements
run: |
pyversion=${{ matrix.python-version }}
pyversion_no_dot=${pyversion//./}
tox run -e requirements-py${pyversion_no_dot}
# TODO(egparedes): add notification for dependencies updates
# - name: Check for updated requirements
# id: update-requirements
# continue-on-error: true
# if: ${{ matrix.python-version == '3.8' && matrix.tox-module-factor == 'cartesian' }}
# shell: bash
# run: |
# if diff -q constraints.txt CURRENT-constraints.txt; then
# echo "REQS_DIFF=''" >> $GITHUB_OUTPUT
# else
# diff --changed-group-format='%<' --unchanged-group-format='' constraints.txt CURRENT-constraints.txt | tr '\n' ' ' > constraints.txt.diff
# echo "REQS_DIFF='$(cat constraints.txt.diff)'" >> $GITHUB_OUTPUT
# fi
# echo "REQS_DIFF_TEST="FOOOOOOOO" >> $GITHUB_OUTPUT
# - name: Notify updated requirements (if any)
# if: ${{ steps.update-requirements.outputs.REQS_DIFF }}
# env:
# SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
# uses: slackapi/slack-github-action@v1.23.0
# with:
# channel-id: ${{ vars.SLACK_BOT_CHANNEL }}
# payload: |
# {
# "text": "TEXT",
# "blocks": [
# {
# "type": "section",
# "text": {
# "type": "plain_text",
# "text": "@channel: AA/${{ steps.update-requirements.outputs.REQS_DIFF }}/BB/ ${{ steps.update-requirements.outputs.REQS_DIFF_TEST }} /CC"
# }
# },
# {
# "type": "section",
# "text": {
# "type": "mrkdwn",
# "text": "@channel: AA/${{ steps.update-requirements.outputs.REQS_DIFF }}/BB/ ${{ steps.update-requirements.outputs.REQS_DIFF_TEST }} /CC"
# }
# }
# ]
# }
- name: Run tests

- name: Run CPU tests for '${{ matrix.gt4py-module }}' with '${{ matrix.dependencies-strategy }}' resolution strategy
env:
NUM_PROCESSES: auto
ENV_REQUIREMENTS_FILE: ${{ matrix.requirements-file }}
run: |
tox run --skip-missing-interpreters -m test-${{ matrix.tox-module-factor }}-cpu
UV_RESOLUTION: ${{ matrix.dependencies-strategy }}
run: uv run nox -s 'test_${{ matrix.gt4py-module }}-${{ matrix.python-version }}' -t 'cpu'

- name: Notify slack
if: ${{ failure() }}
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
uses: slackapi/slack-github-action@v1.23.0
with:
channel-id: ${{ vars.SLACK_BOT_CHANNEL }}
channel-id: ${{ vars.SLACK_BOT_CHANNEL }} # Use SLACK_BOT_CHANNEL_TEST for testing
payload: |
{
"text": "Failed tests for ${{ github.workflow }} (dependencies-strategy=${{ matrix.dependencies-strategy }}, python=${{ matrix.python-version }}, component=${{ matrix.gt4py-module }}) [https://github.com/GridTools/gt4py/actions/runs/${{ github.run_id }}].",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "Failed tests: <https://github.com/GridTools/gt4py/actions/runs/${{ github.run_id }}|${{ github.workflow }}: **${{ matrix.gt4py-module }} (CPU)** for **Python-${{ matrix.python-version }}** with **'${{ matrix.dependencies-strategy }}' uv resolution strategy**.>"
}
}
]
}

weekly-reminder:
runs-on: ubuntu-latest
steps:
- id: get_day_of_the_week
name: Get day of the week
run: echo "day_of_week=$(date +'%u')" >> $GITHUB_OUTPUT

- name: Weekly notification
if: ${{ env.DAY_OF_WEEK == 1 }}
env:
DAY_OF_WEEK: ${{ steps.get_day_of_the_week.outputs.day_of_week }}
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
uses: slackapi/slack-github-action@v1.23.0
with:
channel-id: ${{ vars.SLACK_BOT_CHANNEL }} # Use SLACK_BOT_CHANNEL_TEST for testing
payload: |
{
"text": "${{ github.workflow }}: `test-${{ matrix.tox-module-factor }}-cpu (python${{ matrix.python-version }})`>: *Failed tests!*",
"text": "Weekly reminder to check the latest runs of the GT4Py Daily CI workflow at the GitHub Actions dashboard [https://github.com/GridTools/gt4py/actions/workflows/daily-ci.yml].",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "<https://github.com/GridTools/gt4py/actions/runs/${{ github.run_id }}|${{ github.workflow }}: `test-${{ matrix.tox-module-factor }}-cpu (python${{ matrix.python-version }})`>: *Failed tests!*"
"text": "Weekly reminder to check the latest runs of the <https://github.com/GridTools/gt4py/actions/workflows/daily-ci.yml|GT4Py Daily CI> workflow at the GitHub Actions dashboard."
}
}
]
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/test-cartesian-fallback.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,11 @@ on:

jobs:
test-cartesian:
runs-on: ubuntu-latest
strategy:
matrix:
codegen-factor: [internal, dace]
os: ["ubuntu-latest"]
python-version: ["3.10", "3.11"]
tox-factor: [internal, dace]

runs-on: ${{ matrix.os }}
steps:
- run: 'echo "No build required"'
43 changes: 22 additions & 21 deletions .github/workflows/test-cartesian.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ on:
pull_request:
branches:
- main
paths-ignore: # Skip if only gt4py.next and irrelevant doc files have been updated
paths-ignore: # Skip when only gt4py.next or doc files have been updated
- "src/gt4py/next/**"
- "tests/next_tests/**"
- "examples/**"
Expand All @@ -20,35 +20,36 @@ concurrency:

jobs:
test-cartesian:
runs-on: ubuntu-latest
strategy:
matrix:
codegen-factor: [internal, dace]
os: ["ubuntu-latest"]
python-version: ["3.10", "3.11"]
tox-factor: [internal, dace]
fail-fast: false
havogt marked this conversation as resolved.
Show resolved Hide resolved

runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4

- name: Install C++ libraries
if: ${{ matrix.os == 'macos-latest' }}
shell: bash
run: |
sudo apt install libboost-dev
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
run: brew install boost

- name: Install C++ libraries
if: ${{ matrix.os == 'ubuntu-latest' }}
shell: bash
run: sudo apt install libboost-dev

- name: Install uv and set the python version
uses: astral-sh/setup-uv@v5
with:
enable-cache: true
cache-dependency-glob: "uv.lock"
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: |
**/pyproject.toml
**/constraints.txt
**/requirements-dev.txt
- name: Install python dependencies
run: |
python -m pip install -c ./constraints.txt pip setuptools wheel
python -m pip install -r ./requirements-dev.txt
- name: Test with tox

- name: Run CPU 'cartesian' tests with nox
env:
NUM_PROCESSES: auto
shell: bash
run: |
pyversion=${{ matrix.python-version }}
pyversion_no_dot=${pyversion//./}
tox run -e cartesian-py${pyversion_no_dot}-${{ matrix.tox-factor }}-cpu
run: uv run nox -s 'test_cartesian-${{ matrix.python-version }}(${{ matrix.codegen-factor }}, cpu)'
10 changes: 6 additions & 4 deletions .github/workflows/test-eve-fallback.yml
Original file line number Diff line number Diff line change
@@ -1,24 +1,26 @@
name: "Fallback: Test Eve"

on:
push:
branches:
- main
pull_request:
branches:
- main
paths-ignore: # Inverse of corresponding workflow
- "src/gt4py/eve/**"
- "tests/eve_tests/**"
- "workflows/**"
- "*.cfg"
- "*.ini"
- ".github/workflows/**"
- "*.lock"
- "*.toml"
- "*.yml"

jobs:
test-eve:
strategy:
matrix:
python-version: ["3.10", "3.11"]
os: ["ubuntu-latest"]
python-version: ["3.10", "3.11"]

runs-on: ${{ matrix.os }}
steps:
Expand Down
51 changes: 11 additions & 40 deletions .github/workflows/test-eve.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,61 +10,32 @@ on:
paths: # Run when gt4py.eve files (or package settings) are changed
- "src/gt4py/eve/**"
- "tests/eve_tests/**"
- "workflows/**"
- "*.cfg"
- "*.ini"
- ".github/workflows/**"
- "*.lock"
- "*.toml"
- "*.yml"

jobs:
test-eve:
strategy:
matrix:
python-version: ["3.10", "3.11"]
os: ["ubuntu-latest"]
python-version: ["3.10", "3.11"]
fail-fast: false

runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5

- name: Install uv and set the python version
uses: astral-sh/setup-uv@v5
with:
enable-cache: true
cache-dependency-glob: "uv.lock"
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: |
**/pyproject.toml
**/constraints.txt
**/requirements-dev.txt
- name: Install python dependencies
run: |
python -m pip install -c ./constraints.txt pip setuptools wheel
python -m pip install -r ./requirements-dev.txt
- name: Run tox tests

- name: Run 'eve' tests with nox
env:
NUM_PROCESSES: auto
shell: bash
run: |
pyversion=${{ matrix.python-version }}
pyversion_no_dot=${pyversion//./}
tox run -e eve-py${pyversion_no_dot}
# mv coverage.json coverage-py${{ matrix.python-version }}-${{ matrix.os }}.json
# - name: Upload coverage.json artifact
# uses: actions/upload-artifact@v4
# with:
# name: coverage-py${{ matrix.python-version }}-${{ matrix.os }}
# path: coverage-py${{ matrix.python-version }}-${{ matrix.os }}.json
# - name: Gather info
# run: |
# echo ${{ github.ref_type }} >> info.txt
# echo ${{ github.ref }} >> info.txt
# echo ${{ github.sha }} >> info.txt
# echo ${{ github.event.number }} >> info.txt
# echo ${{ github.event.pull_request.head.ref }} >> info.txt
# echo ${{ github.event.pull_request.head.sha }} >> info.txt
# echo ${{ github.run_id }} >> info.txt
# - name: Upload info artifact
# uses: actions/upload-artifact@v4
# with:
# name: info-py${{ matrix.python-version }}-${{ matrix.os }}
# path: info.txt
run: uv run nox -s test_eve-${{ matrix.python-version }}
Loading
Loading