Managing conda/pip hybrid python environment

[unkcpz]

Move the post-PR discussion of conda/pip hybrid environment #338 (comment) as an issue here for further discussion.

Summary:

• Not a very good idea to mix use pip in a conda environment
• Have to do it at the moment in AiiDAlab since we want to make it is possible that app can be install without publishing the PyPI or conda-forge. This reduces the burden for app developers.
• The principle is conda only for packages that need to be compiled (how we know this in advance, for instance, the pymatgen) and pip for or regular python packages.

[danielhollas]

Some other things to consider

1. Currently, one can install conda dependencies via the post_install script that runs automatically when the app is installed. However, although in the new stack it is now possible to install into the base environment in /opt/conda, this doesn't really work since these dependencies would not persist when the container exits.
2. One can create a new conda environment, but than the dependencies are not available to the jupyter server, which runs under the base environment. In QeApp, a new conda environment is used for installing Quantum Espresso, and this environment is activated in the prepend_text when the AiiDA daemon runs the calculations. But how do we install conda dependencies that should be available in the AiiDAlab app itself? This is a problem I am also facing in my App.
3. As we saw with AWB dependencies, pip will happily downgrade packages when installing into ~/.local directory, which may break the environment. We probably need to use the pip constraint file (requirements.txt) more, at least we should constrain the Jupyter dependencies.

[danielhollas]

Given the above, a counter proposal would be to use conda for most of the stuff.

I think this could be done if we cloned the base environment into ~/.conda at first startup (we could call it local_base), and activate this new environment before the Jupyter server is started. Then the apps could install into this new base environment and these dependencies would be available to Jupyter server.

Instead of Apps having dependencies in setup.cfg, we would have them in a separate file, e.g. conda-environment.txt and they would be automatically installed by conda, which is much better at ensuring environment consistency than pip. In this world, we would only use pip to build and install the AiiDAlab application itself.

cc @yakutovicha @unkcpz

[danielhollas]

Possibly relevant, how to run ipython kernels from different conda environments.

https://towardsdatascience.com/get-your-conda-environment-to-show-in-jupyter-notebooks-the-easy-way-17010b76e874

@yakutovicha

[yakutovicha]

@danielhollas, @unkcpz and I had a brainstorming session on the topic. The discussion was broader than initially intended since this issue interferes with other subjects. Here is a summary.

1. We should fix the aiida-core version in the requirements.txt file. Reminder, this file is used to constraint packages for pip.
2. We need to fix more jupyter-related dependencies in the requirements.txt file. Otherwise, pip might downgrade some of them when installing in packages into the ~/.local directory
3. Currently, one can install conda dependencies via the post_install script that runs automatically when the app is installed. However, although in the new stack - it is now possible to install into the base environment in /opt/conda, this doesn't work since these dependencies would not persist when the container exits.
4. One possible solution for 3. would be to create a new conda environment, but the dependencies will be unavailable to the jupyter server, which runs under the base environment.
5. Similar to 4, one can install all AiiDA-related dependencies into a separate conda environment and set it as a jupyter kernel.
   • This approach is possible, but it will either require an explicit kernel change in every jupyter notebook or kernel will have to be provided as an extra URL parameter. Both approaches are somewhat tricky.
   • The default kernel set in the jupyter config has the lowest priority. The jupyter notebook itself has a kernel specification, which will override the system default.
   • If the kernel is not available, it won’t be used. Another option is to remove the “old” kernel.
6. [contradicts with 5]The base conda environment should never be installed into ~/. Otherwise, a user won't be able to set up a "clean" environment while keeping old data.
7. We heavily use pip install --user, but this contradicts with best practices .
8. Do we want to support multiple AiiDAlab applications in one AiiDAlab instance? [definitely yes]
   • Pip should refuse any attempt to install conflicting dependencies. [it does]
   • Some apps might have conflicting dependencies. We should suggest ways around it in AiiDAlab documentation.
9. @chrisjsewell and @giovannipizzi suggest moving to one management system.
   • If "one system" is pip - we won’t have the possibility to install packages like openbabel, xtb-python.
   • If "one system" is conda - we won't have the possibility to install PyPI packages not yet released on conda (which seems to be the case for many AiiDA plugins).
10. We should check out the administrator-controlled conda installation.