Skip to content

Commit

Permalink
refactor: support python 3.12, various updates (#124)
Browse files Browse the repository at this point in the history
* add python 3.12 classifier to `pyproject.toml`
* explicitly specify packages in `pyproject.toml`
* add authors to `pyproject.toml`
* remove version fn from `Executables`
* move tests from main module to `autotest/`
* remove `BIN_PATH` testing environment variable
* remove testing dependency on mf6 & related exes
* remove `pytest-virtualenv` testing dependency
* add `.vscode/` to `.gitignore`
  • Loading branch information
wpbonelli authored Nov 11, 2023
1 parent e464c40 commit cd644fa
Show file tree
Hide file tree
Showing 23 changed files with 206 additions and 242 deletions.
26 changes: 9 additions & 17 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
steps:

- name: Checkout repo
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Setup Python
uses: actions/setup-python@v4
Expand Down Expand Up @@ -80,24 +80,24 @@ jobs:
fail-fast: false
matrix:
os: [ ubuntu-22.04, macos-12, windows-2022 ]
python: [ 3.8, 3.9, "3.10", "3.11" ]
python: [ 3.8, 3.9, "3.10", "3.11", "3.12" ]
env:
GCC_V: 11
steps:

- name: Checkout repo
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
path: modflow-devtools

- name: Checkout modflow6
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
repository: MODFLOW-USGS/modflow6
path: modflow6

- name: Checkout modflow6 examples
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
repository: MODFLOW-USGS/modflow6-examples
path: modflow6-examples
Expand All @@ -113,7 +113,6 @@ jobs:
with:
repository: MODFLOW-USGS/modflow6-largetestmodels
path: modflow6-largetestmodels
token: ${{ github.token }}

- name: Install executables
uses: modflowpy/install-modflow-action@v1
Expand All @@ -128,10 +127,6 @@ jobs:
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python }}
cache: 'pip'
cache-dependency-path: |
modflow-devtools/pyproject.toml
modflow6-examples/etc/requirements*.txt

- name: Install Python packages
working-directory: modflow-devtools
Expand Down Expand Up @@ -159,22 +154,19 @@ jobs:
run: python ci_build_files.py

- name: Run local tests
working-directory: modflow-devtools
working-directory: modflow-devtools/autotest
env:
BIN_PATH: ~/.local/bin/modflow
REPOS_PATH: ${{ github.workspace }}
GITHUB_TOKEN: ${{ github.token }}
# use --dist loadfile to so tests requiring pytest-virtualenv run on the same worker
run: pytest -v -n auto --dist loadfile --durations 0 --ignore modflow_devtools/test/test_download.py
run: pytest -v -n auto --dist loadfile --durations 0 --ignore test_download.py

- name: Run network-dependent tests
# only invoke the GH API on one OS and Python version
# to avoid rate limits (1000 rqs / hour / repository)
# https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits
if: runner.os == 'Linux' && matrix.python == '3.8'
working-directory: modflow-devtools
working-directory: modflow-devtools/autotest
env:
BIN_PATH: ~/.local/bin/modflow
REPOS_PATH: ${{ github.workspace }}
GITHUB_TOKEN: ${{ github.token }}
run: pytest -v -n auto --durations 0 modflow_devtools/test/test_download.py
run: pytest -v -n auto --durations 0 test_download.py
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -128,8 +128,9 @@ dmypy.json
# Pyre type checker
.pyre/

# pycharme
# IDEs
.idea/
.vscode/

# downloaded exe
modflow_devtools/bin/
Expand Down
10 changes: 3 additions & 7 deletions DEVELOPER.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,37 +38,33 @@ This repository's tests use [`pytest`](https://docs.pytest.org/en/latest/) and s

This repository's tests expect a few environment variables:

- `BIN_PATH`: path to MODFLOW 6 and related executables
- `REPOS_PATH`: the path to MODFLOW 6 example model repositories
- `GITHUB_TOKEN`: a GitHub authentication token

These may be set manually, but the recommended approach is to configure environment variables in a `.env` file in the project root, for instance:

```
BIN_PATH=/path/to/modflow/executables
REPOS_PATH=/path/to/repos
GITHUB_TOKEN=yourtoken...
```

The tests use [`pytest-dotenv`](https://github.com/quiqua/pytest-dotenv) to detect and load variables from this file.

**Note:** at minimum, the tests require that the `mf6` (or `mf6.exe` on Windows) executable is present in `BIN_PATH`.

### Running the tests

Tests should be run from the project root. To run the tests in parallel with verbose output:
Tests should be run from the `autotest` directory. To run the tests in parallel with verbose output:

```shell
pytest -v -n auto
```

### Writing new tests

Tests should follow a few conventions for ease of use and maintenance.
Tests follow a few conventions for ease of use and maintenance.

#### Temporary directories

Tests which must write to disk should use `pytest`'s built-in `temp_dir` fixture or one of this package's own scoped temporary directory fixtures.
Tests which must write to disk use `pytest`'s built-in `temp_dir` fixture or one of this package's own scoped temporary directory fixtures.

## Releasing

Expand Down
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion pytest.ini → autotest/pytest.ini
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@ python_files =
test_*.py
*_test*.py
markers =
slow: tests that don't complete in a few seconds
slow: tests not completing in a few seconds
meta: run by other tests (e.g. testing fixtures)
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
from pathlib import Path

import pytest

from modflow_devtools.build import meson_build
from modflow_devtools.markers import requires_pkg

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import pytest
from flaky import flaky

from modflow_devtools.download import (
download_and_unzip,
download_artifact,
Expand Down
26 changes: 26 additions & 0 deletions autotest/test_executables.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import subprocess
import sys
from pathlib import Path
from shutil import which

import pytest

from modflow_devtools.executables import Executables
from modflow_devtools.misc import add_sys_path, get_suffixes

ext, _ = get_suffixes(sys.platform)
exe_stem = "pytest"
exe_path = Path(which(exe_stem))
bin_path = exe_path.parent
exe = f"{exe_stem}{ext}"


@pytest.fixture
def exes():
with add_sys_path(bin_path):
yield Executables(**{exe_stem: bin_path / exe})


def test_access(exes):
# support both attribute and dictionary style access
assert exes.pytest == exes["pytest"] == exe_path
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ def test_keep_class_scoped_tmpdir(tmp_path, arg):
TestKeepClassScopedTmpdirInner.test_keep_class_scoped_tmpdir_inner.__name__,
"-M",
"test_keep",
"-K",
arg,
tmp_path,
]
assert pytest.main(args) == ExitCode.OK
Expand All @@ -160,19 +160,14 @@ def test_keep_module_scoped_tmpdir(tmp_path, arg):
test_keep_module_scoped_tmpdir_inner.__name__,
"-M",
"test_keep",
"-K",
arg,
tmp_path,
]
assert pytest.main(args) == ExitCode.OK
this_path = Path(__file__)
keep_path = (
tmp_path
/ f"{str(this_path.parent.parent.name)}.{str(this_path.parent.name)}.{str(this_path.stem)}0"
tmp_path / f"{str(this_path.parent.name)}.{str(this_path.stem)}0"
)
from pprint import pprint

print(keep_path)
pprint(list(keep_path.glob("*")))
assert test_keep_fname in [f.name for f in keep_path.glob("*")]


Expand All @@ -186,7 +181,7 @@ def test_keep_session_scoped_tmpdir(tmp_path, arg, request):
test_keep_session_scoped_tmpdir_inner.__name__,
"-M",
"test_keep",
"-K",
arg,
tmp_path,
]
assert pytest.main(args) == ExitCode.OK
Expand Down
59 changes: 59 additions & 0 deletions autotest/test_markers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
from os import environ
from platform import python_version, system
from shutil import which

from packaging.version import Version

from modflow_devtools.markers import *

exe = "pytest"


@requires_exe(exe)
def test_require_exe():
assert which(exe)
require_exe(exe)
require_program(exe)


exes = [exe, "python"]


@require_exe(*exes)
def test_require_exe_multiple():
assert all(which(exe) for exe in exes)


@requires_pkg("pytest")
def test_requires_pkg():
import numpy

assert numpy is not None


@requires_pkg("pytest", "pluggy")
def test_requires_pkg_multiple():
import pluggy
import pytest

assert pluggy is not None and pytest is not None


@requires_platform("Windows")
def test_requires_platform():
assert system() == "Windows"


@excludes_platform("Darwin", ci_only=True)
def test_requires_platform_ci_only():
if "CI" in environ:
assert system() != "Darwin"


py_ver = python_version()


@pytest.mark.parametrize("version", ["3.12", "3.11"])
def test_requires_python(version):
if Version(py_ver) >= Version(version):
assert requires_python(version)
35 changes: 5 additions & 30 deletions modflow_devtools/test/test_misc.py → autotest/test_misc.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,13 @@
from typing import List

import pytest
from conftest import project_root_path

from modflow_devtools.misc import (
get_model_paths,
get_namefile_paths,
get_packages,
has_package,
has_pkg,
set_dir,
set_env,
timed,
Expand Down Expand Up @@ -255,35 +256,9 @@ def test_get_namefile_paths_select_packages():
assert len(paths) >= 43


@pytest.mark.slow
def test_has_pkg(virtualenv):
python = virtualenv.python
venv = Path(python).parent
pkg = "pytest"
dep = "pluggy"
print(
f"Using temp venv at {venv} with python {python} to test has_pkg('{pkg}') with and without '{dep}'"
)

# install a package and remove one of its dependencies
virtualenv.run(f"pip install {project_root_path}")
virtualenv.run(f"pip install {pkg}")
virtualenv.run(f"pip uninstall -y {dep}")

# check with/without strict mode
for strict in [False, True]:
cmd = (
f"from modflow_devtools.misc import has_pkg; print(has_pkg('{pkg}'"
+ (", strict=True))" if strict else "))")
)
exp = "False" if strict else "True"
assert (
virtualenv.run(
f'{python} -c "{cmd}"',
capture=True,
).strip()
== exp
)
def test_has_pkg():
assert has_pkg("pytest")
assert not has_pkg("notapkg")


def test_timed1(capfd):
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
from platform import system

import pytest

from modflow_devtools.ostags import (
OSTag,
get_binary_suffixes,
Expand Down
Loading

0 comments on commit cd644fa

Please sign in to comment.