Run demo notebooks as part of test suite

[munichpavel]

Resources

• https://www.thedataincubator.com/blog/2016/06/09/testing-jupyter-notebooks/

[munichpavel]

I get a permissions error when running test for notebooks:

___________________________________________________________ test_ipynb[notebook_path0] ____________________________________________________________

notebook_path = PosixPath('/Users/pauldev/delo/projects/fake-data-for-learning/notebooks/bayesian-network.ipynb')

    @pytest.mark.parametrize(
        'notebook_path',
        [
            notebook_dir / 'bayesian-network.ipynb',
            notebook_dir / 'conditional-probability-tables-with-constraints.ipynb'
        ]
    )
    def test_ipynb(notebook_path):
>       nb, errors = _notebook_run(notebook_path)

tests/test_demo_notebooks.py:26:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_demo_notebooks.py:40: in _notebook_run
    subprocess.check_call(args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (['python', '-m', 'nbconvert', '--to', 'notebook', '--execute', ...],), kwargs = {}, retcode = 1
cmd = ['python', '-m', 'nbconvert', '--to', 'notebook', '--execute', ...]

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.

        The arguments are the same as for the call function.  Example:

        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['python', '-m', 'nbconvert', '--to', 'notebook', '--execute', '--ExecutePreprocessor.timeout=60', '--output', '/var/folders/xf/jv1nj4vd0lb05xfnp6gly8n40000gq/T/tmpyrm5g5q9.ipynb', PosixPath('/Users/pauldev/delo/projects/fake-data-for-learning/notebooks/bayesian-network.ipynb')]' returned non-zero exit status 1.

../../../miniconda3/lib/python3.7/subprocess.py:363: CalledProcessError
-------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/pauldev/miniconda3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/pauldev/miniconda3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/pauldev/.virtualenvs/fake-data/lib/python3.7/site-packages/nbconvert/__main__.py", line 1, in <module>
    from .nbconvertapp import main
  File "/Users/pauldev/.virtualenvs/fake-data/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 156, in <module>
    class NbConvertApp(JupyterApp):
  File "/Users/pauldev/.virtualenvs/fake-data/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 241, in NbConvertApp
    """.format(formats=get_export_names()))
  File "/Users/pauldev/.virtualenvs/fake-data/lib/python3.7/site-packages/nbconvert/exporters/base.py", line 146, in get_export_names
    e = get_exporter(exporter_name)(config=config)
  File "/Users/pauldev/.virtualenvs/fake-data/lib/python3.7/site-packages/nbconvert/exporters/base.py", line 103, in get_exporter
    if getattr(exporter(config=config), 'enabled', True):
  File "/Users/pauldev/.virtualenvs/fake-data/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 332, in __init__
    super().__init__(config=config, **kw)
  File "/Users/pauldev/.virtualenvs/fake-data/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 114, in __init__
    self._init_preprocessors()
  File "/Users/pauldev/.virtualenvs/fake-data/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 499, in _init_preprocessors
    conf = self._get_conf()
  File "/Users/pauldev/.virtualenvs/fake-data/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 519, in _get_conf
    if conf_path.exists():
  File "/Users/pauldev/miniconda3/lib/python3.7/pathlib.py", line 1356, in exists
    self.stat()
  File "/Users/pauldev/miniconda3/lib/python3.7/pathlib.py", line 1178, in stat
    return self._accessor.stat(self)
PermissionError: [Errno 13] Permission denied: '/usr/local/share/jupyter/nbconvert/templates/conf.json'

Perhaps related to https://discourse.jupyter.org/t/500-internal-server-error-while-trying-to-open-python-notebook-on-jupyterhub/9738/6.

Will try downgrading nbconvert as per above chat.

NOPE, no help. Not surprising. It is a permissions error, so I would expect the solution involves fixing some broken permissions, and not changing package versions.

This seems clearly a local issue, so will trouble-shoot elsewhere.

[munichpavel]

Resuming debugging of above error

Not just a permission error, as after changing ownership in a responsible way, I still get

Seems related to jupyter/jupyter_client#637

Maybe it's also related to still using python 3.7 ???

Could be a coincidence, but I upgraded to python 3.9, and now the test notebooks test passes (again). I wanted to upgrade my local python anyway.

This is something I could test with https://pypi.org/project/tox/