💚 Parallel tests #671

blaginin · 2023-07-31T20:43:23Z

This pull request adds the ability to run tests using several workers using pytest-xdist, significantly improving processing time. For example, on my M1 Max (no CUDA), processing time dropped from 14 minutes to 4 minutes 💨💨💨.

However, this optimization comes at a cost. Previously, tests depended on serial execution. For example, segmentation and prediction methods used to rely on "output" as a folder to store intermediate results. If many functions modified this folder at the same time, the result would be unpredictable. To address this, I made some tweaks alongside #641 and #673 so that functions will not depend on each other.

If we merge this pull request, we will need to start checking that new tests are ready for parallel execution.

Depends on #641 and #673

# Conflicts: # tests/models/test_arch_micronet.py # tests/models/test_semantic_segmentation.py

blaginin · 2023-08-01T00:23:19Z

Cannot check this pull request on CI. It seems that our runner does not have enough memory and crashes even when the number of processes is set to 2.

codecov · 2023-08-01T12:58:37Z

Codecov Report

Merging #671 (82e13d7) into develop (c4ca84e) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop     #671   +/-   ##
========================================
  Coverage    99.85%   99.85%           
========================================
  Files           65       65           
  Lines         7444     7444           
  Branches      1447     1447           
========================================
  Hits          7433     7433           
  Misses           4        4           
  Partials         7        7

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

shaneahmed · 2023-08-02T11:50:06Z

tests/models/test_semantic_segmentation.py

@@ -5,6 +5,7 @@
 # ! The garbage collector
 import gc
 import multiprocessing
+import pathlib


Suggested change

import pathlib

from pathlib import Path

We are using from pathlib import Path now. It would be good if we use Path from now on.

…tests

tests/models/test_semantic_segmentation.py

tests/test_init.py

tests/test_utils.py

Co-authored-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>

# Conflicts: # tests/models/test_patch_predictor.py # tests/models/test_semantic_segmentation.py

shaneahmed · 2023-08-23T11:22:03Z

tiatoolbox/utils/misc.py

@@ -755,6 +755,30 @@ def unzip_data(
        Path.unlink(zip_path)


+try:


if this is only required for tests, then why not move this to conftest? I am a bit concerned about adding this to misc.py. Other than that I am happy to merge this PR.

…tests

shaneahmed

Looks good now. Thanks @blaginin

@TomastpPereira

## 1.5.0 (2023-12-15) ### Major Updates and Feature Improvements - Adds the bokeh visualization tool. #684 - The tool allows a user to launch a server on their machine to visualise whole slide images, overlay the results of deep learning algorithms or to select a patch from whole slide image and run TIAToolbox deep learning engines. - This tool powers the TIA demos server. For details please see https://tiademos.dcs.warwick.ac.uk/. - Extends Annotation to Support Init from WKB #639 - Adds `IOConfig` for NuClick in `pretrained_model.yaml` #709 - Adds functions to save the TIAToolbox Engine outputs to Zarr and AnnotationStore files. #724 - Adds Support for QuPath Annotation Imports #721 ### Changes to API - Adds `model.to(device)` and `model.load_model_from_file()` functionality to make it compatible with PyTorch API. #733 - Replaces `pretrained` with `weights` to make the engines compatible with the new PyTorch API. #621 - Adds support for high-level imports for various utility functions and classes such as `WSIReader`, `PatchPredictor` and `imread` #606, #607, - Adds `tiatoolbox.typing` for type hints. #619 - Fixes incorrect file size saved by `save_tiles`, issue with certain WSIs raised by @TomastpPereira - TissueMasker transform now returns mask instead of a list. #748 - Fixes #732 ### Bug Fixes and Other Changes - Fixes `pixman` incompability error on Colab #601 - Removes `shapely.speedups`. The module no longer has any affect in Shapely >=2.0. #622 - Fixes errors in the slidegraph example notebook #608 - Fixes bugs in WSI Registration #645, #670, #693 - Fixes the situation where PatchExtractor.get_coords() can return patch coords which lie fully outside the bounds of a slide. #712 - Fixes #710 - Fixes #738 raised by @xiachenrui ### Development related changes - Replaces `flake8` and `isort` with `ruff` #625, #666 - Adds `mypy` checks to `root` and `utils` package. This will be rolled out in phases to other modules. #723 - Adds a module to detect file types using magic number/signatures #616 - Uses `poetry` for version updates instead of `bump2version`. #638 - Removes `setup.cfg` and uses `pyproject.toml` for project configurations. - Reduces runtime for some unit tests e.g., #627, #630, #631, #629 - Reuses models and datasets in tests on GitHub actions by utilising cache #641, #644 - Set up parallel tests locally #671 **Full Changelog:** v1.4.0...v1.5.0 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: mostafajahanifar <74412979+mostafajahanifar@users.noreply.github.com> Co-authored-by: John Pocock <John-P@users.noreply.github.com> Co-authored-by: DavidBAEpstein <David.Epstein@warwick.ac.uk> Co-authored-by: David Epstein <22086916+DavidBAEpstein@users.noreply.github.com> Co-authored-by: Ruqayya Awan <18444369+ruqayya@users.noreply.github.com> Co-authored-by: Mark Eastwood <20169086+measty@users.noreply.github.com> Co-authored-by: adamshephard <39619155+adamshephard@users.noreply.github.com> Co-authored-by: adamshephard <adam.shephard@warwick.ac.uk> Co-authored-by: Abdol <a@fkrtech.com> Co-authored-by: Jiaqi-Lv <60471431+Jiaqi-Lv@users.noreply.github.com> Co-authored-by: Abishek <abishekraj6797@gmail.com> Co-authored-by: Dmitrii Blaginin <blaginin@mbp.lan>

@TomastpPereira

## 1.5.0 (2023-12-15) ### Major Updates and Feature Improvements - Adds the bokeh visualization tool. #684 - The tool allows a user to launch a server on their machine to visualise whole slide images, overlay the results of deep learning algorithms or to select a patch from whole slide image and run TIAToolbox deep learning engines. - This tool powers the TIA demos server. For details please see https://tiademos.dcs.warwick.ac.uk/. - Extends Annotation to Support Init from WKB #639 - Adds `IOConfig` for NuClick in `pretrained_model.yaml` #709 - Adds functions to save the TIAToolbox Engine outputs to Zarr and AnnotationStore files. #724 - Adds Support for QuPath Annotation Imports #721 ### Changes to API - Adds `model.to(device)` and `model.load_model_from_file()` functionality to make it compatible with PyTorch API. #733 - Replaces `pretrained` with `weights` to make the engines compatible with the new PyTorch API. #621 - Adds support for high-level imports for various utility functions and classes such as `WSIReader`, `PatchPredictor` and `imread` #606, #607, - Adds `tiatoolbox.typing` for type hints. #619 - Fixes incorrect file size saved by `save_tiles`, issue with certain WSIs raised by @TomastpPereira - TissueMasker transform now returns mask instead of a list. #748 - Fixes #732 ### Bug Fixes and Other Changes - Fixes `pixman` incompability error on Colab #601 - Removes `shapely.speedups`. The module no longer has any affect in Shapely >=2.0. #622 - Fixes errors in the slidegraph example notebook #608 - Fixes bugs in WSI Registration #645, #670, #693 - Fixes the situation where PatchExtractor.get_coords() can return patch coords which lie fully outside the bounds of a slide. #712 - Fixes #710 - Fixes #738 raised by @xiachenrui ### Development related changes - Replaces `flake8` and `isort` with `ruff` #625, #666 - Adds `mypy` checks to `root` and `utils` package. This will be rolled out in phases to other modules. #723 - Adds a module to detect file types using magic number/signatures #616 - Uses `poetry` for version updates instead of `bump2version`. #638 - Removes `setup.cfg` and uses `pyproject.toml` for project configurations. - Reduces runtime for some unit tests e.g., #627, #630, #631, #629 - Reuses models and datasets in tests on GitHub actions by utilising cache #641, #644 - Set up parallel tests locally #671 **Full Changelog:** v1.4.0...v1.5.0

blaginin added 9 commits July 19, 2023 09:32

💚 Add parallel tests

2ab38f6

💚 add pr trigger

05797ef

💚 fix pytest arg

b414e5d

Merge branch 'dev' into parallel-tests

b4ed5bc

✅ fix fixtures to work in parallel

b2b0e73

⏪ CI changes

cd1e5e7

🔨 set parallel tests as a default option

905c4da

Merge branch 'dev' into parallel-tests

8d7ae0e

# Conflicts: # tests/models/test_arch_micronet.py # tests/models/test_semantic_segmentation.py

💚 Add own folder for .predict( methods

fe72af4

➕ add pytest-xdist dependency

0b0da94

blaginin added the dev tools Changes/Updates in Development tools label Aug 1, 2023

shaneahmed added this to the Release v1.5.0 milestone Aug 1, 2023

shaneahmed requested review from John-P and shaneahmed August 1, 2023 12:21

Merge branch 'develop' into parallel-tests

ea39e10

blaginin mentioned this pull request Aug 1, 2023

🚀 Switch to More Powerful Worker #679

Closed

shaneahmed reviewed Aug 2, 2023

View reviewed changes

shaneahmed assigned blaginin Aug 4, 2023

blaginin added 7 commits August 9, 2023 23:16

🔀 merge dev

cec8459

Merge remote-tracking branch 'upstream/parallel-tests' into parallel-…

87d43d0

…tests

♻️ switched to Path

62fc857

Merge branch 'develop' into parallel-tests

dc0ddd4

Merge branch 'develop' into parallel-tests

32a1d3f

♻️ use tmp_path instead of the default directory

e4fefe2

Merge remote-tracking branch 'upstream/parallel-tests' into parallel-…

54f9f6f

…tests

blaginin marked this pull request as ready for review August 12, 2023 20:12

blaginin added 2 commits August 12, 2023 21:15

♻️ add -n auto to Makefile

d7b50f2

♻️ remove -n auto to Makefile

370ee2c