Errors loading jupytext with jupyter lab

[mdtisdall]

I'm trying to enable jupytext as the content manager for my notebooks. However, it seems the python module isn't available to the loader during start-up with jupyter lab (error being ModuleNotFoundError: No module named 'jupytext')

I made a gist of my shell.nix file and the output of jupyter lab here:

(https://gist.github.com/mdtisdall/3446714f86373697609961b8b5d593bb)

[guaraqe]

The error message says that the jupytext server extension is missing. Server extensions, like JupyterLab extensions, have to be installed by Jupyter. JupyterWith is not yet capable of managing server extensions. You can try two things:

• Generate the JupyterLab folder manually, as described in the README. You can adapt the code written here to try to adapt it to server extensions.
• Or, take a look at this comment, that talks about a test of server extensions support.

[mdtisdall]

Thanks. I've updated my shell.nix to

let
  jupyter = import (builtins.fetchGit {
    url = https://github.com/tweag/jupyterWith;
    rev = "";
  }) {};

  iPython = jupyter.kernels.iPythonWith {
    name = "notebook";
    packages = p: with p; [
      numpy
      matplotlib
      jupytext
    ];
  };

  jupyterEnvironment =
    jupyter.jupyterlabWith {
      kernels = [ iPython ];
      directory = ./jupyterlab;
      extraPackages = p: [p.python3Packages.jupytext];
      extraInputsFrom = p: [p.python3Packages.jupytext];
    };
in
  jupyterEnvironment.env

and used generate-directory jupyterlab_vim juptyerlab-jupytext to build the appropriate directory. However, I'm still getting the ModuleNotFoundError: No module named 'jupytext' error on jupyter lab. This seems odd to me, because when I run python3 from within nix-shell environment, and then run import jupytext, it seems to happily find the jupytext module. Is jupyter lab using some different method to resolve the modules?

[guaraqe]

The installation line for jupytext seems to be:

jupyter nbextension install --py jupytext

So, there seems to be two differences:

• We do not pass --py, take a look at the code of generate-directory.
• You wrote juptyerlab-jupytext (this seems to be a typo)

I recommend creating your folder with generate-directory, but without jupytext, and launching the extension installation with the command above for jupytext, by specifying --app-dir="$DIRECTORY" so that it uses the correct directory.

Tell me if that works.

[mdtisdall]

Thanks for following up on this. Following your suggestion, I was able to build a jupyterlab directory without jupytext, and then get it to install with

jupyter nbextension install --py jupytext --nbextensions=./jupyterlab/nbextensions

However, this still fails to validate; when I run

jupyter nbextension enable jupytext --py --debug

I get

      - Validating: problems found:
        - require?  X jupytext/index

I've tried digging into this a little more. I think the installation command you quoted there is only needed for the notebook extension which adds the notebook menu features. However, I don't need those right away, so I'm okay if they don't work. As such, I'm wondering whether the core issue is regarding which python environment I've installed jupytext in. I realize it's in the iPythonWith environment, but the directions for jupytext say

If you want to use Jupytext within Jupyter Notebook or JupyterLab, make sure you install Jupytext in the Python environment where the Jupyter server runs.

and suggests running

/path_to_your_jupyter_environment/python -m pip install jupytext --upgrade --user

I thought I had this covered in my shell.nix file with the extraPackages = p: [p.python3Packages.jupytext]; addition, but perhaps I've misunderstood how to get the jupytext package installed in the same python environment as JupyterLab?

Another error message suggesting jupytext module isn't visible to jupyter is if I add this line to my jupyter_notebook_config.py:

c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"

jupyter lab fails with the error

[C 18:26:32.526 LabApp] The 'contents_manager_class' trait of <jupyterlab.labapp.LabApp object at 0x7effd0c24050> instance must be a type, but 'jupytext.TextFileContentsManager' could not be imported

which again sounds like a failure to find the jupytext module.

[guaraqe]

I think that the command you executed (and the one I mentioned before) is the one that adds the menu features. I am able to install jupytext at the moment of the creation of the folder (following here) :

jupyter labextension install jupyterlab-jupytext --app-dir=./jupyterlab

And the extension appears when I start jupyter lab. Can you test if it works? Make sure to change to directory = ./jupyterlab; in the shell.nix. There is a chance I am not using it correctly, since I don't really know jupytext, so if you have a command or sequence of steps I can take to have the same error, please.

The point about the nbextension enable is indeed a problem. I will try to investigate a little more. Keep me updated.

[mdtisdall]

Thanks for your help, I think I'm now close to having jupytext working to manage my notebooks! After doing some more reading, it became clear to me that the part I wanted was just the server extension, not the notebook extension, or the lab extension.

As such, I've got the following shell.nix file now (which I realize is bad Nix form):

let
  jupyter = import (builtins.fetchGit {
    url = https://github.com/tweag/jupyterWith;
    rev = "";
  }) {};

  iPython = jupyter.kernels.iPythonWith {
    name = "notebook";
    packages = p: with p; [
      numpy
      matplotlib
    ];
  };

  jupyterEnvironment =
    jupyter.jupyterlabWith {
      kernels = [ iPython ];
      directory = ./jupyterlab;
      extraPackages = p: [p.python3Packages.jupytext];
      extraJupyterPath = "/nix/store/plg5kram23icmpi42sqwabn9d9svvyh6-python3.7-jupytext-1.2.4/lib/python3.7/site-packages";
    };
in
  jupyterEnvironment.env

I can then run

jupyter serverextension enable jupytext

and see that it successfully validates the jupytext server extension. Similarly, when I run jupyter lab it now successfully imports the jupytext python module, and I've tested this is working as expected when I configure it as my notebook manager.

So my outstanding question relates to me being a Nix newbie: how can I extract the correct path to add to extraJupyterPath, rather than hardcoding it like that?

[guaraqe]

I see. We can do that if we make the extraJupyterPath parameter a function that takes Nixpkgs as an argument, so you just need to use interpolation. I will push a PR and mark you there.

[guaraqe]

Is there something else we can document to avoid the problems you had?

[mdtisdall]

It may be an esoteric point, but it was initially unclear to me that there were three separate versions of python involved. Perhaps just clarifying that, and the correct method for injecting packages into each of them, would help. I think for someone like me, who was pretty confident I understood the python error but didn't understand I was adding the package to the wrong environment, this might help.

Regardless, thanks very much for your help addressing this!

[guaraqe]

Thank you too! We aim to make all Jupyter capabilities accessible with JupyterWith, and you just helped us with that.

I added some comments to the README to avoid these problems in the future.