NEURON wheel without X11 => Ability to turn off GUI loading in wheel

[arnaudon]

Would it be possible to have the pip install NEURON without any X11, so that it can be used without the $DISPLAY variable and not have warning such as

Warning: no DISPLAY environment variable.
--No graphics will be displayed.

Thanks!

[nrnhines]

can be used without the $DISPLAY variable

Actually, except for the warning message, that is the way it is now. I.e. the process continues on, but just without InterViews graphics. The warning message is turned off if nrniv is launched with the -nogui arg. There is presently no way to turn off this message when launching python and importing NEURON. There are two ways for me to change this behavior. First, I can make it so that a user can add the nogui property to a .nrn.defaults file in their HOME folder. I suppose that would mean I should also allow the option -gui so it can be turned on for a specific session. Second, I can allow the user to import neuron_options which would provide an option dictionary that would be interrogated on the subsequent import neuron.

Did you experience this on mac or linux? On the mac, the pip wheel dynamically loads X11 only if XQuartz is installed. @pramodk , I don't recall at the moment if the wheel for linux
does the same. In either case, it seems reasonable to check the DISPLAY environment variable and if that is not set, avoid dlopening the X11 library and the warning altogether.

[arnaudon]

Thanks for your quick reply! I see, it is not so easy. I noticed that because after starting an sbatch job running neuron, and logging out of my session (on supercomputer), it was killing my job with a xserver error message. Then I realised that it was not my code, but neuron that was using X11, which I had recently installed via pip install NEURON. The easy fix is to not allow neuron to connect to X11, and only shows this warning, which is ok to leave with!

[nrnhines]

starting an sbatch job running neuron

This actually raises a puzzle. If NEURON runs in parallel with launch of nrniv -mpi ... then the -mpi arg turns off InterViews graphics. Since you installed with pip, I presume this means you are launching python. In this case, if you do not from mpi4py import MPI prior to import neuron, then NEURON will not know it is supposed to turn off its interviews graphics and call MPI_Init prior to where it checks for DISPLAY. I.e. it is too late to call h.nrnmpi_init() to prevent the DISPLAY check. This is an unanticipated flaw in my implementation of h.nrnmpi_init(). Here again, an import neuron_options could come to the rescue. Until then, the only work around, beside from mpi4py import MPI prior to from neuron import h is to set the environment variable export NEURON_INIT_MPI=1 and import neuron prior to any import mpi4py.

[arnaudon]

I see, but I'm not using MPI for this, I do many single cell simulations, using simple multiprocessing and each neuron run in a different isolated process. But never mind, I can leave with the warning message in the logs. My usecase is quite specific, and not sure it'll be worth you time to fix that just for me.
Thanks for you help anyway!

[wvangeit]

@nrnhines I wonder if it would not be possible to build a separate wheel for nrn without gui. I think it would be beneficial for people working on cluster or on machine that don't need X11 at all.
(it could be a separate pypi package, like NEURON-nogui or so)

[nrnhines]

Although we are getting to the point with travis and azure where we do not have to expend a lot of manual effort and time to prepare distributions. I think the user confusion resulting from too many installation choices is to be avoided. I think choice makes more sense at the time when the user can more easily understand the implications of the choice. In this particular case I'm leaning toward eliminating the DISPLAY message for distributions where X11 is dynamically loaded ( linux wheels and all mac?). I think the puzzlement that will result when no graphics appears will be rare as there is normally a DISPLAY environment variable. And then to avoid graphics in arnaudon's situation just consists in ```unset DISPLAY`` before the sbatch launch.

[arnaudon]

In fact, having neuron connecting to X11 could be 'bad'. I was trying to run many small neuron computations on multiple nodes with dask (or even starting independent multiprocessing jobs on different nodes), and it was breaking if too many processes where launched at the same time, because of a lack of available socket for neuron to use X11. I couldn't understand what was causing that, but now I know! So unset DISPLAY is crucial in some cases! I don't know how make this easier to debug for newcomers such as me, but again, I have a fairly specific usecase ( I guess!).

[pramodk]

Thanks @arnaudon for the detailed information!

Similar issue was raised during the last neuron-dev meeting specifically in the context of OSX and X11/XQuartz initialisation. We will discuss this again during next week meeting because there are other related issues and will come back here.

[wvangeit]

@nrnhines actually the nrn_options package also sounds tempting. Not sure if it would 100% solve this particular issue, since people would still need to know that they have to load the package etc.
But it would definitely be useful in general, so that we can set variables like NFRAME etc.

[nrnhines]

Pull request #602 responds to this issue. For -nogui and -NFRAME args one can

import __main__
__main__.neuron_options = {
  "-nogui":None,
  "-NFRAME":"3000",
}
from neuron import h

You can leave out the __main__ unless the code comes in via an import.

[arnaudon]

Just in case, I noticed that the warning message I get many times is here:

nrn/src/ivoc/ivocmain.cpp

Line 661 in 6d23a39

fprintf(stderr, "Warning: no DISPLAY environment variable.\

[nrnhines]

After discussion at the June developer meeting, it was concluded that polluting __main__ with neuron_options was not the best way to proceed. (Although, it is hard to imagine a user not adding names to __main__). e.g import neuron often adds neuron to __main__). Anyway, @alexsavulescu added a comment to #597 to the effect that it would be useful to allow options to be specified with an environment variable. #602 now allows this with, e.g.

export NEURON_MODULE_OPTIONS="-nogui -NFRAME 3210 -background #00ff00"

[wvangeit]

Was the NEURON_MODULE_OPTIONS ability ever merged? @alex4200 and me were wondering how we can disable X11 in the context of a MOOC jupyter notebook. There is an external script that opens nrngui.hoc, but we would like to prevent it opening the nrn X11 window. I've tried setting NEURON_MODULE_OPTIONS="-nogui" in the notebook but it doesn't seem to have an effect.

I think it would still be nice to have a wheel (e.g. to be installed as an extra with 'pip install neuron[nogui]') that doesn't have an X11 dependency.

[ramcdougal]

Could this be worked-around by having a script unset DISPLAY before importing neuron?

I'd like a programmatic way around connecting to xquartz on a mac (which takes a long time if xquartz isn't running), because I want the option of using the gui, but I almost never want it, and it's especially silly whenever I'm trying to use wx for the GUI instead.

[wvangeit]

I tried unsetting DISPLAY, but that didn't work (not 100% sure it's because of jupiter or nrn)

[wvangeit]

The option is good, but it would also be nice to have a package that doesn't depend on X11 at all. In some contexts one is 100% sure no GUI will be used (HPC, notebooks, etc.), so having the dependency is just an overhead in these cases.

[nrnhines]

For temporary purposes, does

export NEURON_INIT_MPI=1

work for you. That may not help @ramcdougal as it likely turns off his wx stuff as well as not loading x11 (if configured so that InterViews is dynamically loaded). Maybe #602 should be broken into two PRs, one of which does only the
export NEURON_MODULE_OPTIONS="..." perhaps enhanced with -noiv as an alternative to -nogui.

[nrnhines]

@ramcdougal For temporary purposes, one can have the effect of -nogui (and no loading of x11) by

hines@hines-T7500:~/neuron/noivload/build/install/lib$ mv libivx11dynam.so libivx11dynam.so.x

Let me know if that turns off your wx stuff as well?

[pramodk]

I am reopening this issue but changing title to Ability to turn off GUI loading in wheel.

[alex4200]

@nrnhines I just wanted to try your solution to turn the GUI off by renaming that so file - but I could not find it anywhere!

Note: I did install NEURON using pip install neuron.

[pramodk]

@alex4200 : Michael was mentioning above command from build directory when install neuron from source. For python wheel installation, I tried:

→ pip install neuron
Collecting neuron
  Using cached NEURON-7.8.2-cp38-cp38-macosx_10_9_x86_64.whl (11.6 MB)
Requirement already satisfied: numpy>=1.9.3 in ./vnew/lib/python3.8/site-packages (from neuron) (1.20.2)
Installing collected packages: neuron
Successfully installed neuron-7.8.2

→ python
Python 3.8.9 (v3.8.9:a743f8192b, Apr  2 2021, 08:12:53)
>>> import neuron
Warning: no DISPLAY environment variable.
--No graphics will be displayed.


# NOW, remove / rename `libivx11dynam.so` library

# check location of site packages dir in virtual env
→ python -c 'import site; print(site.getsitepackages()[0])'
/Users/pramod/Downloads/vnew/lib/python3.8/site-packages

# now temporarily rename that shared library
→ mv /Users/pramod/Downloads/vnew/lib/python3.8/site-packages/neuron/.data/lib/libivx11dynam.dylib /Users/pramod/Downloads/vnew/lib/python3.8/site-packages/neuron/.data/lib/libivx11dynam.dylib.new

# if I try to import neuron then I don't see DISPLAY warning

→ python
Python 3.8.9 (v3.8.9:a743f8192b, Apr  2 2021, 08:12:53)
>>> import neuron

Obviously this is for testing purposes and we would like to have a proper option for this.

[alex4200]

Thanks, that workaround seems to work.

Here is my complete code that I can run in a jupyter notebook to disable the GUI:

import os, site
libpath = site.getsitepackages()[0] + "/neuron/.data/lib/libivx11dynam.dylib"
os.rename(libpath, libpath + ".new")

[anilbey]

Does it work now? How can we use it?

[arnaudon]

export NEURON_MODULE_OPTIONS='-nogui'