Installing additional python packages in running container is no longer recognised by vscode jupyter

[knoriy]

Applies To

• Notebooks (.ipynb files)
• Interactive Window and/or Cell Scripts (.py files with #%% markers)

What happened?

Steps

• Start Docker container
• Attach the container to VSCode using Remote - Containers
• Open .jpynb
• Either in terminal or in notebook cell run pip install package (package used was torchaudio and tgt)
• restart notebook
• import package
• Error: ModuleNotFoundError: No module named 'tgt'

expectation and result

The package should be available in the notebook session after notebook restart but gets an error that the module is not found.
(I have used these exact steps in the past without any issue)

I have tested this in a normal notebook session (running in the same container instance) via jupyter lab --ip 0.0.0.0 --no-browser, the package is installed and is seen by python as expected.

dockerfile

FROM pytorch/pytorch:1.11.0-cuda11.3-cudnn8-runtime
RUN apt-get update

## START - Creating user
RUN apt-get -y install sudo
RUN adduser --disabled-password --gecos '' user
RUN adduser user sudo
RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers

RUN apt-get install -y tree \
                        libsndfile1 

## START - Jupyter notebook setup
RUN pip install jupyterlab

## START - Python packages
ADD requirements.txt .
RUN pip install -r requirements.txt
## END - Python packages

WORKDIR /workspace
RUN chown user:user /workspace
USER user

ENV type lab

CMD ["bash","-c", "jupyter ${type} --ip 0.0.0.0 --no-browser"]

VS Code Version

Version: 1.67.0 Commit: 57fd6d0195bb9b9d1b49f6da5db789060795de47 Date: 2022-05-04T12:08:42.978Z Electron: 17.4.1 Chromium: 98.0.4758.141 Node.js: 16.13.0 V8: 9.8.177.13-electron.0 OS: Linux x64 5.16.19-76051619-generic

Jupyter Extension Version

v2022.4.1011282140

Jupyter logs

info 13:43:49.771: Got new session e36d8642-720a-4ede-a5e2-91bcb3bf14f1
info 13:43:49.771: Started new restart session
info 13:43:49.772: Dispose Kernel process
info 13:43:49.774: Executing silently Code (idle) = import sys\nprint(sys.executable)
info 13:43:50.316: Executing silently Code (completed) = import sys\nprint(sys.executable)
info 13:43:50.450: UpdateWorkingDirectoryAndPath in Kernel
info 13:43:50.450: Executing silently Code (idle) = import os\nimport sys\n%cd "/workspace"\nif os.getcwd() not in sys.path:\nsys.path.insert(0, os.getc
info 13:43:50.463: Executing silently Code (completed) = import os\nimport sys\n%cd "/workspace"\nif os.getcwd() not in sys.path:\nsys.path.insert(0, os.getc
info 13:43:50.464: Waiting for idle on (kernel): e36d8642-720a-4ede-a5e2-91bcb3bf14f1 -> idle
info 13:43:50.464: Finished waiting for idle on (kernel): e36d8642-720a-4ede-a5e2-91bcb3bf14f1 -> idle
info 13:44:0.384: Execute Cell 0 /workspace/test.ipynb
info 13:44:0.477: Cell 0 executed with state Success
info 13:44:15.862: Execute Cell 6 /workspace/test.ipynb
info 13:44:17.476: Cancel all remaining cells true || Error || undefined
info 13:44:17.476: Cancel pending cells
info 13:44:17.476: Cell 6 executed with state Error

Coding Language and Runtime Version

Python 3.8

Language Extension Version (if applicable)

Pylance v2022.4.3

Anaconda Version (if applicable)

conda 4.11.0

Running Jupyter locally or remotely?

Remote

[DonJayamanne]

Thanks for filing this issue, however I'm unable to test this at my end without the requirements.txt file.
Please could you provide that file.

[knoriy]

Hi Don, You can find the requirements.txt below.

Please let me know if there is anything else I can do :)

pandas
matplotlib
numpy
hparm
unidecode
inflect
librosa
scipy

Edit: @DonJayamanne I removed torchaudio and tgt from requirements.txt because these were added in the Dockerfile later to workaround the issues mentioned above.

[derekschinke]

Does this mean you're unable to import packages after installing them with %pip install? Because this is an issue I've been facing with new installs on 3 different computers in the past week. Magic installs work fine when starting jupyter from the terminal and opening the .ipynb files in-browser.

[knoriy]

Does this mean you're unable to import packages after installing them with %pip install? Because this is an issue I've been facing with new installs on 3 different computers in the past week. Magic installs work fine when starting jupyter from the terminal and opening the .ipynb files in-browser.

I believe it may be the same or a very similar issue you faced. Installing packages via terminal gives me a ModuleNotFoundError. While installing packages via %pip install results in:

ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/opt/conda/lib/python3.8/site-packages/itsdangerous'
Consider using the `--user` option or check the permissions.

[derekschinke]

Actually, that error is different and expected. Running it again with --user makes it seem like it installs as expected, but it actually doesn't (running import <package> fails and returns ModuleNotFoundError). So I think it's a similar issue.

Edit: submitted #9995

[DonJayamanne]

@derekschinke @knoriy
Please note, as this is a conda environment hence you'd need to use %conda install tgt.
Unfortunately I'm on a Mac M1, and looks like some of these pacakges don't yet install on Mac M1.

Please could you try re-installing using %conda install tgt and restarting the kernel to confirm this works.
If %conda install tgt fails, generally there will be some guidelines on getting this working, e.g. providing the name of the channel or the like.
If all fails, please provide a screenshot of the error.

Here's a link to our WIKI with bit more information https://github.com/microsoft/vscode-jupyter/wiki/Installing-Python-packages-in-Jupyter-Notebooks & https://jakevdp.github.io/blog/2017/12/05/installing-python-packages-from-jupyter/

[DonJayamanne]

Closing this issue as its been over 4 weeks, since the information was requested.
We'll be happy to reopen the issue when the requested information has been provided.