Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace poetry with pip, build with hatchling, dynamically calculate minimum dependencies with uv #733

Merged
merged 39 commits into from
Sep 17, 2024
Merged
Show file tree
Hide file tree
Changes from 36 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
1b7343b
Specify dependency groups in a way that doesn't require poetry
mfisher87 Jun 30, 2024
f2e15ce
Add new poetry hooks
mfisher87 Jun 30, 2024
732abd1
Update Python compatibility in README
mfisher87 Jun 30, 2024
a31b7ae
Attempt to fix tests by installing test dependencies
mfisher87 Jun 30, 2024
5704abb
Decouple test, lint, typecheck scripts
mfisher87 Jun 30, 2024
82189e7
Remove redundant config
mfisher87 Jun 30, 2024
5b819a3
Avoid duplicate test workflow runs
mfisher87 Jun 30, 2024
21be5f2
Install docs dependencies during readthedocs build
mfisher87 Jun 30, 2024
24a5191
Migrate poetry dependency specification to standardized specification
mfisher87 Sep 4, 2024
edd6442
Replace caret dependency notation
mfisher87 Sep 7, 2024
6ac66b7
Fixup conditional dependency specifications
mfisher87 Sep 7, 2024
a39a519
Unpin kerchunk extra dependencies
mfisher87 Sep 7, 2024
4c57306
Remove poetry
mfisher87 Sep 7, 2024
cde7b85
Calculate minimum dependencies dynamically with uv
mfisher87 Sep 7, 2024
5f5986f
Encode mypy's files and python version in config
mfisher87 Sep 7, 2024
6dada1a
Replace poetry in GitHub Actions
mfisher87 Sep 7, 2024
8e8a7e7
Replace poetry in conda dev environment
mfisher87 Sep 7, 2024
8643011
Replace poetry for binder
mfisher87 Sep 7, 2024
bd3fe74
Replace poetry for readthedocs
mfisher87 Sep 7, 2024
cadcbdd
Add nox dev dependency and fixup noxfile
mfisher87 Sep 7, 2024
e194f14
Fixup composable action
mfisher87 Sep 7, 2024
c0cf028
Install test deps for mindeps test action
mfisher87 Sep 7, 2024
9818907
Upgrade minimum evrsions of some test dependencies
mfisher87 Sep 7, 2024
7c4eb2e
Add uv lockfile pre-commit check
mfisher87 Sep 7, 2024
4409b11
Add docs dependencies to conda dev env
mfisher87 Sep 11, 2024
bff4354
Restore integration tests to run on all supported Python versions
mfisher87 Sep 11, 2024
adbc43c
Move dev environment file out of CI directory
mfisher87 Sep 11, 2024
82381d7
Fix typo
mfisher87 Sep 14, 2024
87c22ea
Use modern syntax to get Actions output into GUI
mfisher87 Sep 14, 2024
c5c414e
Fix spelling
mfisher87 Sep 14, 2024
f53b451
Explicitly source instead of using dot alias
mfisher87 Sep 14, 2024
f96b5f0
Clarify supported versions of Python in README
mfisher87 Sep 14, 2024
d1b619d
Add development guide note for conda users
mfisher87 Sep 17, 2024
df578ed
Clarify purpose of Nox vs full dev env
mfisher87 Sep 17, 2024
0923beb
Replace reference to Make
mfisher87 Sep 17, 2024
54e0ae1
Merge branch 'main' into poetry-groups-to-pep621-optional-deps
mfisher87 Sep 17, 2024
a6aee82
Rename conda dev environment file
mfisher87 Sep 17, 2024
2cef7fb
Remove references to removed/moved files
mfisher87 Sep 17, 2024
de313be
Move build system to top of pyproject
mfisher87 Sep 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions .github/actions/install-pkg/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
name: Install earthaccess Python package
description: Install earthaccess Python package and testing dependencies

inputs:
python-version:
required: true

runs:
using: composite
steps:
- uses: actions/setup-python@v5
with:
python-version: ${{ inputs.python-version }}
cache: pip

- name: Display full python version
shell: bash
id: full-python-version
run: echo "{version}=$(python -c "import sys; print('-'.join(str(v) for v in sys.version_info))")" >> $GITHUB_OUTPUT

- name: Install package and test dependencies
shell: bash
run: pip install .[test]
39 changes: 5 additions & 34 deletions .github/workflows/integration-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,6 @@ concurrency:
group: integration-tests-${{ github.ref }}
cancel-in-progress: true

defaults:
run:
shell: bash -l {0}

jobs:
integration-tests:
runs-on: ubuntu-latest
Expand All @@ -31,44 +27,19 @@ jobs:

steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5

- uses: ./.github/actions/install-pkg
with:
python-version: ${{ matrix.python-version }}
- name: Get full python version
id: full-python-version
run: echo ::set-output name=version::$(python -c "import sys; print('-'.join(str(v) for v in sys.version_info))")
- name: Install poetry
uses: abatilo/actions-poetry@v3
- name: Configure poetry
run: |
poetry config virtualenvs.create true --local
poetry config virtualenvs.in-project true --local
poetry self add setuptools
- name: Set up cache
uses: actions/cache@v4
id: cache
with:
path: .venv
key: venv-${{ runner.os }}-${{ steps.full-python-version.outputs.version }}-${{ hashFiles('poetry.lock') }}
- name: Ensure cache is healthy
if: steps.cache.outputs.cache-hit == 'true'
run: poetry run pip --version >/dev/null 2>&1 || rm -rf .venv
- name: Install Dependencies
if: ${{ !env.ACT }}
run: poetry install
- name: Install Dependencies
if: ${{ env.ACT }}
# When using `act` to run the workflow locally, the `poetry install` command
# may fail due to network issues when running multiple Docker containers.
run: poetry install || poetry install || poetry install

- name: Test
env:
EARTHDATA_USERNAME: ${{ secrets.EDL_USERNAME }}
EARTHDATA_PASSWORD: ${{ secrets.EDL_PASSWORD }}
EARTHACCESS_TEST_USERNAME: ${{ secrets.EDL_USERNAME }}
EARTHACCESS_TEST_PASSWORD: ${{ secrets.EDL_PASSWORD }}
run: poetry run bash scripts/integration-test.sh
run: ./scripts/integration-test.sh

- name: Upload coverage
# Don't upload coverage when using the `act` tool to run the workflow locally
if: ${{ !env.ACT }}
Expand Down
27 changes: 23 additions & 4 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,32 @@ on:

jobs:
build:
name: Build package
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: verify version matches git tag
run: scripts/verify_tag.sh
- name: Build and publish to pypi
uses: JRubics/poetry-publish@v2.0

- uses: hynek/build-and-inspect-python-package@v2

publish:
name: Publish to PyPI
needs: ["build"]
runs-on: ubuntu-latest
environment: pypi
permissions:
id-token: write
steps:
- uses: actions/download-artifact@v4
with:
name: Packages
path: dist

- uses: pypa/gh-action-pypi-publish@release/v1
# NOTE: This is not the current best practice. Instead, we should use
# "trusted publishing":
# https://github.com/pypa/gh-action-pypi-publish?tab=readme-ov-file#trusted-publishing
with:
python_version: "3.9"
pypi_token: ${{ secrets.PYPI_TOKEN }}
password: ${{ secrets.PYPI_TOKEN }}
36 changes: 16 additions & 20 deletions .github/workflows/test-mindeps.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Test minimum dependencies
name: Unit test minimum dependencies

on:
push:
Expand All @@ -11,11 +11,6 @@ concurrency:
group: test-mindeps-${{ github.ref }}
cancel-in-progress: true

# Required shell entrypoint to have properly activated conda environments
defaults:
run:
shell: bash -l {0}

jobs:
test-mindeps:
runs-on: ubuntu-latest
Expand All @@ -25,25 +20,26 @@ jobs:
steps:
- name: Checkout source
uses: actions/checkout@v4.1.1
- name: Setup Conda Environment
uses: conda-incubator/setup-miniconda@v3.0.4

- name: Install uv
uses: astral-sh/setup-uv@v2
mfisher87 marked this conversation as resolved.
Show resolved Hide resolved
with:
version: "0.4.7"
enable-cache: true

- name: Set up Python
uses: actions/setup-python@v5
with:
miniforge-variant: Mambaforge
miniforge-version: latest
use-mamba: true
channel-priority: strict
environment-file: ci/environment-mindeps.yaml
activate-environment: test-environment
auto-activate-base: false
python-version-file: pyproject.toml

- name: Install
run: python -m pip install --no-deps -e .
- name: Install minimum-compatible dependencies
run: uv sync --resolution lowest-direct --extra test

- name: mamba list
run: mamba list
- name: Install earthaccess
run: uv pip install --no-deps .

- name: Test
run: bash scripts/test.sh
run: uv run pytest tests/unit --cov=earthaccess --cov=tests --cov-report=term-missing --capture=no --tb=native --log-cli-level=INFO

- name: Upload coverage
# Don't upload coverage when using the `act` tool to run the workflow locally
Expand Down
50 changes: 10 additions & 40 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,11 @@ name: Unit Tests

on:
push:
branches:
- main
pull_request:
types: [opened, synchronize]

defaults:
run:
shell: bash -l {0}

jobs:
unit-tests:
runs-on: ubuntu-latest
Expand All @@ -19,45 +17,17 @@ jobs:

steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5

- uses: ./.github/actions/install-pkg
with:
python-version: ${{ matrix.python-version }}
- name: Get full python version
id: full-python-version
run: echo "version=$(python -c "import sys; print('-'.join(str(v) for v in sys.version_info))")" >> $GITHUB_OUTPUT
- name: Install poetry
uses: abatilo/actions-poetry@v3
- name: Configure poetry
run: |
poetry config virtualenvs.create true --local
poetry config virtualenvs.in-project true --local
poetry self add setuptools
- name: Set up mypy cache
uses: actions/cache@v4
id: mypy-cache
with:
path: .mypy_cache
key: mypy-${{ runner.os }}-${{ steps.full-python-version.outputs.version }}-${{ hashFiles('poetry.lock') }}
- name: Set up cache
uses: actions/cache@v4
id: cache
with:
path: .venv
key: venv-${{ runner.os }}-${{ steps.full-python-version.outputs.version }}-${{ hashFiles('poetry.lock') }}
- name: Ensure cache is healthy
if: steps.cache.outputs.cache-hit == 'true'
run: poetry run pip --version >/dev/null 2>&1 || rm -rf .venv
- name: Install Dependencies
if: ${{ !env.ACT }}
run: poetry install --quiet
- name: Install Dependencies
if: ${{ env.ACT }}
# When using `act` to run the workflow locally, the `poetry install` command
# may fail due to network issues when running multiple Docker containers.
run: poetry install --quiet || poetry install --quiet || poetry install --quiet

- name: Typecheck
run: mypy

- name: Test
run: poetry run bash scripts/test.sh
run: pytest tests/unit --cov=earthaccess --cov=tests --cov-report=term-missing --capture=no --tb=native --log-cli-level=INFO

- name: Upload coverage
# Don't upload coverage when using the `act` tool to run the workflow locally
if: ${{ !env.ACT }}
Expand Down
6 changes: 6 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,12 @@ repos:
- id: ruff
args: ["--fix", "--exit-non-zero-on-fix"]
- id: ruff-format

- repo: https://github.com/astral-sh/uv-pre-commit
rev: "0.4.7"
hooks:
- id: uv-lock
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lockfile stays up-to-date without user needing to learn to use any new tooling


- repo: https://github.com/google/yamlfmt
rev: v0.13.0
hooks:
Expand Down
48 changes: 0 additions & 48 deletions Makefile
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file wasn't documented or used anywhere.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better also remove the make test step from docs/contributing/index.md.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NICE FIND! How did I miss that!?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file was deleted.

2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ Visit [our quick start guide](https://earthaccess.readthedocs.io/en/latest/quick

## Compatibility

Only **Python 3.8+** is supported.
The _minimum_ supported Python version is **3.9**.


## How to Contribute to `earthaccess`
Expand Down
4 changes: 4 additions & 0 deletions binder/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,7 @@ dependencies:
- hvplot
- holoviews
- panel

- pip
- pip:
- "."
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this install earthaccess from source in the Binder environment?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, but not editable. We can change it to "--editable ." to achieve that if we want. I haven't tested this -- is there a way to test with Binder before merge to main?

6 changes: 0 additions & 6 deletions binder/postBuild

This file was deleted.

26 changes: 0 additions & 26 deletions ci/environment-dev.yml
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me, it doesn't make sense for this to be in the ci directory.

This file was deleted.

29 changes: 0 additions & 29 deletions ci/environment-mindeps.yaml

This file was deleted.

Loading
Loading