TST: Add Test Execution For Notebook Execution

[John-P]

This PR adds a set of simple unit tests that checks notebooks execute without error. It currently only checks the first two four notebooks (CPU only notebooks). nbconvert may need to be added to requirements (to set the version) if this fails on CI.

There are other ways to test notebooks including special packages. However, this does not introduce any dependencies, runs as if a normal notebook and also gives us an easy way to control which notebooks run (e.g. no GPU).

To Do / Done

• Clean up file dumping. Some notebooks use a tmp directory to dump files, but others (patch extraction) just dump into the root. This ends up putting many files in with all the notebooks (.ipynb) when testing. Resolved by first copying the notebook to a tmp directory for the test.
• Filter out apt-get and pip commands. Resolved by loading the notebook JSON and removing lines which match the regex ^!\s*(apt(-get)?|pip).*install. A better solution that could make these cells only execute on Colab directly in the notebook by simply only executing if the COLAB_GPU environment variable exists. Then there will no need to filter lines, and running locally will not execute apt. You can test this in colab vs locally with [[ -z "${COLAB_GPU}" ]] && echo "Not In COLAB" || echo "In COLAB".
• Inject the tiatoolbox repo to the python path (sys.path) to import correctly on CI and locally during development.
• Fix bug in coverage aggregation. There is a bug in aggregating multiprocessing test coverage after tests have run (coverage.exceptions.DataError: Can't combine arc data with line data). Supposedly fixed here:
  • CoverageException: Can't combine line data with arc data when updating to repository tip nedbat/coveragepy#399
  • Internal error "Can't combine line data with arc data" for code using subprocesses pytest-dev/pytest-cov#243
    but still having issues with this.

Example Error Output With Partial stderr For Failing GPU Notebook

./tests/test_notebooks.py::TestNotebook::test_notebook[Patch Prediction] Failed: [undefined]Failed: Notebook failed to execute: examples/05-patch-prediction.ipynb Tail of notebook output:
	    209 if not hasattr(torch._C, '_cuda_getDeviceCount'):
	--> 210     raise AssertionError("Torch not compiled with CUDA enabled")
	    211 if _cudart is None:
	    212     raise AssertionError(
	    213         "libcudart functions unavailable. It looks like you have a broken build?")

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[John-P]

See #417 for a simpler test of notebook syntax correctness.

[John-P]

@shaneahmed Shall we delete this PR and branch if we are moving to static analysis of notebooks?