reticulate does not care about the environment

[jan-glx]

reticulate does not care about the environment

... sorry for the hopefully click bait title ...

Several open issues, namely, #292, #311, #338, #396, satijalab/seurat#677 are caused by python loaded by reticulate from a conda environment not finding shared libraries which would be found by python started directly from that environment. (leading to e.g. ImportError: /usr/lib64/libstdc++.so.6: version 'CXXABI_1.3.9' not found)

With this issue I hope to initiate a discussion on:

How should and how can reticulate handle the process environment & shared library search paths?

Here is what I think to understand about it:

• reticulate loads libpython => R and python live in the same process
• when loading python, reticulate modifies the environment a bit but not as much as conda activate would
• these modifications could cause issues for the rest of the R session (might add an incompatible gcc version etc)
• there is this partial workaround for windows
• changing LD_LIBRARY_PATH of a running linux process has no effect
• the problem only surfaces if the shared library versions are not available in the R environment (= always when using a HPC cluster, almost never when using your own up-to date machine )
• there could be silent errors
• probably the same points apply to virtualenv

My conclusion is:

• reticulate cannot use a different (virtual) environment than the R process it is running in 😢
• use_condaenv cannot work (without starting a separate process which it never will) and should be removed/deprecated in favor of starting the R session in the adequate conda environment (having conda manage R as well would help to handle conflicts between R and python requirements)

What do you think?

[imoutsatsos]

Also issue #367

[kevinushey]

I think the answer here is that ultimately use_condaenv() requires extra orchestration by the user to ensure that R is launched with the required environment set, and AFAIK it's not possible to dynamically modify the current R environment to match what e.g. conda activate might do.

[jan-glx]

Aaron Lun's package basilisk might be a suitable work around if your are affected by this. (It runs python in a separate process if necessary.)

[carmensandoval]

I have an issue that is related to this, and I wonder why it's not possible to simply tell reticulate to look for those shared libraries in the conda env in question.

#1285