make sure no modules are loaded when EasyBuild is started

[boegel]

(old internal ticket 346)

EasyBuild should check whether (software) modules (built with EasyBuild) were loaded when it stars, and refuse to continue if it detects it.

Having modules loaded outside of EasyBuild could make or break build, and the result may be different with an environment in which no modules were loaded.

Currently, it only checks whether the software it is trying to built is loaded, but it should do the same for dependencies and non-dependencies.

[boegel]

comment by @nudded:

Also non-dependencies? You want nothing loaded when building begins? or only things that are present in the repository should not be loaded?

[boegel]

I think you should check this with the SOFTROOT/EBROOT stuff. If any variable like that is set, a module built with EasyBuild is loaded, and may yield surprises.
Loading module Bar before EasyBuild starts may yield a different build of software Foo if Bar is an optional dependency of Foo. We don't want that.
Demanding that no modules created by EasyBuild is loaded manually is ok imho.

[boegel]

comment by @nudded:

Yeah it seems like a sane sanitycheck. if it only checks those variables should be fairly simple to implement.

[boegel]

This will have to wait because of https://bugzilla.redhat.com/show_bug.cgi?id=834580 .

modulecmd sometimes crashes when large (e.g. PETSc) modules are loaded.

The only way around this is to load a couple of modules before running EasyBuild. Implementing the feature this ticket asks for would break that workaround.

We need to figure out the issue with environment-modules first, and get it fixed upstream.

[boegel]

was fixed in pull request #73, but reverted in pull request #76 because of https://bugzilla.redhat.com/show_bug.cgi?id=834580

[boegel]

A patch is available now for the modulecmd segfault issue, see https://bugzilla.redhat.com/show_bug.cgi?id=834580 .

We should probably wait to implement the loaded modules check in EasyBuild until a new modules release is available that includes this patch. Then, we can document/mention the need for (at least) this particular version of environment modules in the EasyBuild FAQ, so that people can easily resolve this issue when they run into it.

[boegel]

Along with this, we should also support an override parameter for this check, so that a user can allow having modules loaded when using EasyBuild, for whatever reason (but, the check should be enabled by default, obviously).

[damianam]

Update on this ancient issue:

I think it makes sense to have an option to unload everything except a set of modules listed in a whitelist. At least in our setup that will be a nice way to sanitize the environment without breaking other things.

[wpoely86]

@damianam use sticky modules?

[damianam]

We have a module for setting up EB related variables. And another one for GC3Pie. And another one for EB itself. I don't want to make them sticky and force me to use --purge in my shell just to allow EB to have an easier way to sanitize its environment prior to running. That is for our case, but other people might have other reasons to keep other modules loaded, and not make them sticky. It has implications in the environment outside of EB, doesn't seem like a clean solution to me.

[ocaisa]

You could just make sure there are no modules loaded that define EBROOT*, that's pretty clean I think

[damianam]

It is clean in our case, but if somebody mixes EB modules with other modules that shouldn't be loaded when installing software it wouldn't be enough. Besides that, it would unload the EasyBuild module (Edit to add: if installed by EasyBuild), so you would have to make at least one exception already

[ocaisa]

I think you'd be safe if you only unloaded the modules after the EB process has finished grabbing its configuration

[migueldiascosta]

I think I just hit this problem with a PythonPackage (e.g. GPAW) that depends on another one (e.g. ASE), in a counterintuitive way: if ASE is already loaded, the sanity check fails saying ASE cannot be found, but if ASE is not loaded, the sanity check passes

Because it is counterintuive, it would be helpful that at least some warning was issued at the beginning saying a dependency shouldn't be loaded

(in any case, why doesn't the sanity check find/reload the dependency when it was already loaded?)

[vanzod]

I also hit the same problem when trying to build Python with foss toolchain when the ncurses library built with the dummy toolchain has been built with separate terminfo symbols (see PR easybuilders/easybuild-easyconfigs#3545) as described in easybuilders/easybuild-easyconfigs#4049.

I would personally prefer to have EB purge the environment if modules are loaded (with the exception of the EB module) and then restore them after the installation is completed. This behavior can be overridden with a flag for those who want to keep the modules loaded.

[boegel]

I guess different sites/people would have differences preferences here...

We could implement a detection mechanism for this (that ignores a loaded EasyBuild module), with a couple of different options as to how EasyBuild should respond:

• --detect-loaded-modules=warn: print a big fat warning if any (EasyBuild-generated) modules are loaded, mentioning that this may affect the installations done by EasyBuild
• --detect-loaded-modules=error: same, but make it a fatal error
• --detect-loaded-modules=warn,purge: warn about having modules loaded, but run a module purge and continue; it should be OK to also unload the EasyBuild module, since EasyBuild is already running (not 100% sure about this one though...)

The detection mechanism should probably look at $EBROOT* environment variables, since just checking module names in the output of module list is not going to work imho.

I guess we'll also need some kind of ignore list (--ignore-loaded-modules=...), since some stuff may be provided to EasyBuild via a module installed with EasyBuild, e.g. GC3Pie (for --job), PyYAML (for .yeb easyconfigs), etc...

[boegel]

fixed in #2228, let's follow up there in context of the proposed enhancements...