Allow user-local modules with hierarchical naming schemes #1472

geimer · 2015-11-12T23:08:49Z

If one uses EB's default MNS, user-local modules are no problem, as a site only needs to 'module use' something like '$HOME/.local/modules' in their modules setup script. However, when using an HMNS, this per-user location needs to be taken into account by the EB-generated modules which extend
$MODULEPATH.

This PR introduces a new option 'subdir-user-modules' (empty by default) which allows to define a location relative to the user's $HOME in which a user-local modules hierarchy (according to the module naming scheme, configurable via det_user_modpath_extensions) is searched. The corresponding modules will then include an additional module use statement to pick up modules from the this location. To prevent errors if the directory does not exist (C modules), this module use is guarded by a check.

…nsions

hpcugentbot · 2015-11-12T23:08:50Z

Automatic reply from Jenkins: Can I test this?

boegel · 2015-11-13T00:05:31Z

Jenkins: ok to test

hpcugentbot · 2015-11-13T02:03:57Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2329/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

boegel · 2015-12-14T23:07:44Z

easybuild/framework/easyblock.py

+                user_modpath_exts = ActiveMNS().det_user_modpath_extensions(self.cfg)
+                self.log.debug("Including user module path extensions returned by naming scheme: %s"
+                               % user_modpath_exts)
+                quoted_user_modpath_exts = [quote_str(os.path.join(user_modpath, ext)) for ext in user_modpath_exts]


maybe we should be quoting in module_generator.use instead, if it's so important to do this?

I think quoting only gets important if the path contains some crazy characters that are evaluated by Tcl/Lua. That is, the quotes are more a "safety net" rather than required. AFAIR, I did the quoting outside of module_generator.use to allow both prefix and path to be expressions that are evaluated when loading the modulefile. But I agree that this may be over-engineered.

boegel · 2015-12-14T23:23:39Z

@geimer: sorry it took me so long to give this a proper review, you should have pinged me earlier

There's a couple of loose ends here, so too late to make it in for EB v2.5.0 I'm afraid, so I'm going to put 2.6.0 as milestone on this...

Next to the remarks inline:

a test should be added for the --subdir-user-modules option somewhere, to verify that it's actually picked up
a test should be added for module_generator.det_home, or actually for get_env (see other remarks)

Other than that, this looks really reasonable to me, you have to stop pretending to not know Python. ;-)

Let me know if you need any help in tackling the remarks (or just throw your hands up and let me take over, that's fine by me).

geimer · 2016-01-18T13:59:51Z

@boegel: OK, I believe the culprit is in ModulesTool.modpath_extensions_for... Hard-coded regex to search for module use... I don't think this ever worked for Lua syntax... Shouldn't the regrex be better queried from module_generator? Also, the regex has to become more complicated to account for prefix, which can also be an expression. Urgh...

Seems like this is getting more and more unrelated to the original topic of this PR, just because it triggers already existing bugs in other areas of the code... What do you think?

boegel · 2016-01-18T14:20:50Z

@geimer: it sounds like you're hitting the problem that @t-brown was working on during the hackathon in Austin, cfr. PR #1474

geimer · 2016-01-18T14:50:51Z

@boegel: Well, not really. I just noticed as well that the Lua syntax was missing here.

The more difficult issue is that with user-local modules, the argument given to module use may be an expression in Lua/Tcl syntax that is evaluated at module load time. My current understanding of the code is that it assumes absolute paths in plain text, that it can then pass to path_matches, which implicitly calls os.path.exists and os.path.samefile. This won't work for the Lua/Tcl expression...

boegel · 2016-01-18T14:54:50Z

@geimer: it sounds like we'll need to rework that to parse the output of module use rather than parsing the module file themselves?

geimer · 2016-01-18T15:40:23Z

@boegel: Something like this, yes. But since module use doesn't output anything, you probably need to load the module, see whether it extends $MODULEPATH, and unload it again if not. But I'm still trying to figure out what exactly modpath_extensions_for and path_to_top_of_module_tree are doing...

boegel · 2016-01-18T17:33:21Z

@geimer: I meant to say module show rather than module use; the output of module show should always include the modifications to $MODULEPATH (unless the directory is not there in $HOME, in which case we don't care, I think)

geimer · 2016-01-18T18:53:44Z

@boegel: Looks like module show doesn't seem to work for Tcl modules... :-(

boegel · 2016-01-18T19:01:27Z

@geimer In what sense, exactly? And do you mean with module files written in Tcl, or using modulecmd.tcl?

geimer · 2016-01-18T19:19:11Z

@boegel: I tried module show on a modulefile including a module use statement using modulecmd.tcl, and the output didn't include anything w.r.t. $MODULEPATH. May be a non-issue, though, as module hierarchies only work reasonably well with Lmod. But still...

boegel · 2016-01-18T21:20:09Z

@geimer: it doesn't even include the module use as is?

boegel · 2016-01-18T21:27:25Z

Seems like you're right, urgh.

test module:

$ cat ~/.local/easybuild/modules/all/toy/0.0 
#%Module
module use /tmp

'show' outputs shows no clue of use or $MODULEPATH

$ tclsh tcl-modulecmd/tcl/modulecmd.tcl bash show toy/0.0                           
-------------------------------------------------------------------
/Users/kehoste/.local/easybuild/modules/all/toy/0.0:

-------------------------------------------------------------------

'load' does what it should though:

$ tclsh tcl-modulecmd/tcl/modulecmd.tcl bash load toy/0.0
_LMFILES__modshare=/Users/kehoste/.local/easybuild/modules/all/toy/0.0:1; export _LMFILES__modshare;
LOADEDMODULES_modshare=toy/0.0:1; export LOADEDMODULES_modshare;
MODULEPATH_modshare=/tmp:1:/Users/kehoste/.local/easybuild/modules/all:1; export MODULEPATH_modshare;
_LMFILES_=/Users/kehoste/.local/easybuild/modules/all/toy/0.0; export _LMFILES_;
LOADEDMODULES=:toy/0.0; export LOADEDMODULES;
MODULEPATH=/tmp:/Users/kehoste/.local/easybuild/modules/all; export MODULEPATH;

'show' output with Lmod:

$ lmod bash show toy/0.0
LMOD_DEFAULT_MODULEPATH="/Users/kehoste/.local/easybuild/modules/all";
export LMOD_DEFAULT_MODULEPATH;
MODULEPATH="/Users/kehoste/.local/easybuild/modules/all";
export MODULEPATH;
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   /Users/kehoste/.local/easybuild/modules/all/toy/0.0:
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
prepend_path("MODULEPATH","/tmp")

LMOD_DEFAULT_MODULEPATH="/Users/kehoste/.local/easybuild/modules/all";
export LMOD_DEFAULT_MODULEPATH;
MODULEPATH="/Users/kehoste/.local/easybuild/modules/all";
export MODULEPATH;
_ModuleTable001_="X01vZHVsZVRhYmxlXz17WyJhY3RpdmVTaXplIl09MCxiYXNlTXBhdGhBPXsiL1VzZXJzL2tlaG9zdGUvLmxvY2FsL2Vhc3lidWlsZC9tb2R1bGVzL2FsbCIsfSxbImNfcmVidWlsZFRpbWUiXT1mYWxzZSxbImNfc2hvcnRUaW1lIl09ZmFsc2UsZmFtaWx5PXt9LGluYWN0aXZlPXt9LG1UPXt9LG1wYXRoQT17Ii9Vc2Vycy9rZWhvc3RlLy5sb2NhbC9lYXN5YnVpbGQvbW9kdWxlcy9hbGwiLH0sWyJzeXN0ZW1CYXNlTVBBVEgiXT0iL1VzZXJzL2tlaG9zdGUvLmxvY2FsL2Vhc3lidWlsZC9tb2R1bGVzL2FsbCIsWyJ2ZXJzaW9uIl09Mix9";
export _ModuleTable001_;
_ModuleTable_Sz_="1";
export _ModuleTable_Sz_;

'show' output with Tcl/C env mods:

$ modulecmd bash show toy/0.0
-------------------------------------------------------------------
/Users/kehoste/.local/easybuild/modules/all/toy/0.0:

module       use /tmp 
-------------------------------------------------------------------

geimer · 2016-01-20T07:20:54Z

@boegel: I've adjusted the regex in ModulesTool.modpath_extensions_for to account for quoted module use statements. But this is obviously not the full solution. I suggest to fix the modpath_extensions_for routine and its unit test (which doesn't check Lua module syntax) in a separate PR.

W.r.t. ''.join(...) vs. '\n'.join(...): The latter feels more natural, agreed. But it introduces a number of extra newlines as, for example, conditional_statement already includes a trailing newline. As fixing this is again something that can have wide-spread impacts, addressing this in a separate PR would IMHO be preferred.

Apart from this, I believe all your comments have been addressed. Please take another look.

hpcugentbot · 2016-01-20T08:03:28Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2523/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

boegel · 2016-01-20T10:03:41Z

easybuild/tools/modules.py

@@ -588,7 +588,7 @@ def modpath_extensions_for(self, mod_names):
        modpath_exts = {}
        for mod_name in mod_names:
            modtxt = self.read_module_file(mod_name)
-            useregex = re.compile(r"^\s*module\s+use\s+(\S+)", re.M)
+            useregex = re.compile(r'^\s*module\s+use\s+"?([^" \t\n\r\f\v]+)"?', re.M)


@geimer: please use \s instead of \t\n\r\f\v, cfr. https://docs.python.org/2/library/re.html

\s When the UNICODE flag is not specified, it matches any whitespace character, this is equivalent to the set [ \t\n\r\f\v].

@boegel: I wasn't aware that one can use character classes within character classes. But makes sense.

boegel · 2016-01-20T10:06:40Z

@geimer: one single small style remark, looks excellent otherwise!

W.r.t. modpath_extensions_for: can you clarify what's left to fix there? Just that it's blissfully unaware of Lua syntax (which is basically what #1474 is about), or is there more to it?

hpcugentbot · 2016-01-20T11:38:42Z

EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2528/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

boegel · 2016-01-20T12:28:41Z

tests failed due to a temporary issue with GitHub, it seems, let's try again

Jenkins: test this please

hpcugentbot · 2016-01-20T13:21:31Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2529/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

geimer · 2016-01-20T16:32:15Z

@boegel: W.r.t. modpath_extensions_for: With the current usage model, I believe things should be fine. However, if at some point in the future someone uses the prefix argument of the use method also for module paths that need to be considered in the modpath_extensions_for routine, things will break (for a potential use case, see below). Thus it's more about being future-proof.

And now the potential use case: Assume that we have a system-wide software installation using EB in <prefix>, using a hierarchical module naming scheme. In the long run, I would like to be able to extend it with my own local software/module collection in $HOME (this is what this PR is good for), but also use EB for installing this stuff (e.g., using --installpath=$HOME/.local). One example might be to install a special debugging version or a version with a specific patch of an MPI library, and then some software on top of it. Then, the module paths in my $HOME and the modpath extensions of modules installed there need to be taken into account in the modpath_extensions_for, right?

I guess a bunch of other things may need to be resolved before the use case above will work with EB. But it is something I would like to see at some point (and I know at least one other guy in our group who already asked whether this would be possible).

boegel · 2016-01-20T23:02:55Z

@geimer: OK, I see what you're saying.

The current implementation of modpath_extensions_for only looks for module use "...", so it will miss the module use [ file join ... ] entries we have there for user-local module path extensions.

For your use case to work, we will need a way of dealing with those too.
I'm not sure how we can do that since module show doesn't seem to be an option that always works, but we can figure something out (e.g. push what we get through Tcl first to evaluate it, or whatever, or just look at how $MODULEPATH changes if we fake-load the module).

Can you please open a separate issue on this with largely a copy-paste of your last comment, and a pointer to this PR?

This is good to go btw, thanks a lot for your work and dedicated follow-up on this, much appreciated.

Allow user-local modules with hierarchical naming schemes

ocaisa · 2017-06-28T15:09:28Z

easybuild/tools/module_naming_scheme/mns.py

@@ -126,6 +126,18 @@ def det_modpath_extensions(self, ec):
        # by default: an empty list of subdirectories to extend $MODULEPATH with
        return []

+    def det_user_modpath_extensions(self, ec):


@geimer @boegel This does not seem to be respecting EASYBUILD_SUFFIX_MODULES_PATH (which defaults to all):

conflict("GCCcore") prepend_path("MODULEPATH", "/usr/local/software/jureca/Stages/Devel-2017a/modules/all/Compiler/GCCcore/5.4.0") if isDir(pathJoin(os.getenv("HOME"), ".local/EasyBuild/modules/Compiler/GCCcore/5.4.0")) then prepend_path("MODULEPATH", pathJoin(os.getenv("HOME"), ".local/EasyBuild/modules/Compiler/GCCcore/5.4.0")) end

Notice the missing all in the local path.

@ocaisa what was the full EasyBuild configuration here (eb --show-config)?

-bash-4.2$ eb --show-config # # Current EasyBuild configuration # (C: command line argument, D: default value, E: environment variable, F: configuration file) # buildpath (E) = /dev/shm/ocaisa experimental (E) = True fixed-installdir-naming-scheme (E) = True group-writable-installdir (E) = True hide-deps (E) = ANTLR, APR, APR-util, AT-SPI2-ATK, AT-SPI2-core, ATK, Autoconf, Automake, adwaita-icon-theme, ant, assimp, Bison, babl, binutils, byacc, bzip2, CUSP, Coreutils, cairo, configurable-http-proxy, DB, DBus, DocBook-XML, Dyninst, dbus-glib, damageproto, ETSF_IO, Exiv2, eudev, expat, FFmpeg, FLTK, FTGL, fixesproto, fontsproto, fontconfig, freeglut, freetype, GCCcore, GDAL, GEGL, GL2PS, GLEW, GLib, GLPK, GPC, GObject-Introspection, GTI, GTK+, GTS, Gdk-Pixbuf, Ghostscript, GraphicsMagick, GtkSourceView, g2clib, g2lib, gc, gexiv2, glproto, gperf, guile, grib_api, gsettings-desktop-schemas, gettext, HarfBuzz, icc, ifort, inputproto, intltool, itstool, JUnit, JSON-C, JSON-GLib, JasPer, jhbuild, kbproto, LZO, LibTIFF, LibUUID, Libint, LittleCMS, libGLU, libICE, libSM, libX11, libXau, libXaw, libXcursor, libXdamage, libXdmcp, libXext, libXfixes, libXfont, libXft, libXi, libXinerama, libXmu, libXpm, libXrandr, libXrender, libXt, libXtst, libcerf, libcroco, libctl, libdap, libdrm, libdwarf, libelf, libepoxy, libevent, libffi, libfontenc, libgd, libgeotiff, libglade, libidn, libjpeg-turbo, libmatheval, libmypaint, libpng, libpciaccess, libpthread-stubs, libreadline, librsvg, libtool, libunistring, libunwind, libyaml, libxcb, libxkbcommon, libxml2, libxslt, libyuv, M4, Mesa, makedepend, motif, msgpack-c, NASM, ncurses, nettle, nodejs, nvenc_sdk, nvidia, OPARI2, OTF2, PCRE, PDT, PROJ, Pango, Pmw, PnMPI, PyCairo, PyGObject, Python-Xpra, pixman, pkg-config, pkgconfig, popt, pscom, Qhull, Qt, Qt5, qrupdate, randrproto, recordproto, renderproto, S-Lang, SCons, SIP, SQLite, SWIG, Serf, Szip, scrollkeeper, Tcl, Tk, texinfo, UDUNITS, util-linux, vpx, wxPropertyGrid, wxWidgets, XML-Parser, XZ, XKeyboardConfig, x264, x265, xbitmaps, xcb-proto, xcb-util, xcb-util-image, xcb-util-keysyms, xcb-util-renderutil, xcb-util-wm, xextproto, xineramaproto, xorg-macros, xprop, xproto, xtrans, YAXT, Yasm, zlib hide-toolchains (E) = GCCcore include-easyblocks (E) = /work/zam/swmanage/EasyBuild/Custom_EasyBlocks/Devel-2017a/*.py, /work/zam/swmanage/EasyBuild/Custom_EasyBlocks/Devel-2017a/generic/*.py include-module-naming-schemes (E) = /work/zam/swmanage/EasyBuild/Custom_MNS/Devel-2017a/*.py include-toolchains (E) = /work/zam/swmanage/EasyBuild/Custom_Toolchains/Devel-2017a/*.py installpath (E) = /usr/local/software/jureca/Stages/Devel-2017a job-backend-config (E) = /usr/local/software/FZJ/gc3pie.cfg job-cores (E) = 48 job-max-walltime (E) = 1 minimal-toolchains (E) = True module-naming-scheme (E) = CustomHierarchicalMNS packagepath (E) = /usr/local/software/jureca/Stages/Devel-2017a/packages prefix (E) = /usr/local/software/jureca/Stages/Devel-2017a repositorypath (E) = /usr/local/software/jureca/Stages/Devel-2017a/eb_repo robot (E) = /work/zam/swmanage/EasyBuild/Golden_Repo/Devel-2017a, /usr/local/software/jureca/Stages/Devel-2017a/eb_repo, /work/zam/swmanage/EasyBuild/Golden_Repo/Devel-2017a robot-paths (E) = /work/zam/swmanage/EasyBuild/Golden_Repo/Devel-2017a, /usr/local/software/jureca/Stages/Devel-2017a/eb_repo, /work/zam/swmanage/EasyBuild/Golden_Repo/Devel-2017a set-gid-bit (E) = True sourcepath (E) = /work/zam/swmanage/EasyBuild/sources subdir-user-modules (E) = .local/EasyBuild/modules umask (E) = 002 use-existing-modules (E) = True

I get the same effect with using HierarchicalMNS

This occurs with a vanilla instance and the command line:

eb --module-naming-scheme=HierarchicalMNS --subdir-user-modules=.local/EasyBuild/modules GCCcore-5.4.0.eb -x

fix implemented in #2250

geimer added 7 commits November 12, 2015 04:49

Add 'subdir-user-modules' option

b0bdb17

Extend module naming scheme to provide user-specific module path exte…

67c39ec

…nsions

Generate 'module use' statements for user-specific module paths

2335cce

Make unit tests work again

60ea159

Include det_user_modpath_extensions in naming scheme tests

7562542

Add unit test for guarded 'module use' statements with prefix

9bd6a1e

Fix typo

df718d0

boegel modified the milestone: v2.5.0 Nov 13, 2015

boegel reviewed Dec 14, 2015
View reviewed changes

boegel modified the milestones: v2.5.0, ASAP Dec 14, 2015

Adjusted regex to account for quoted "module use" statements

e43787e

boegel reviewed Jan 20, 2016
View reviewed changes

boegel mentioned this pull request Jan 20, 2016

Bug fix for modpath_extensions_for function. #1474

Merged

Simplified regex

3506ee1

boegel added a commit that referenced this pull request Jan 20, 2016

Merge pull request #1472 from geimer/user_modules

69cfe84

Allow user-local modules with hierarchical naming schemes

boegel merged commit 69cfe84 into easybuilders:develop Jan 20, 2016

geimer mentioned this pull request Jan 25, 2016

Handle expressions in module use statements in ModulesTool.modpath_extensions_for #1573

Open

geimer deleted the user_modules branch January 25, 2016 17:47

ocaisa mentioned this pull request Mar 6, 2017

Problems when trying to use EB as a normal user with HMNS #2143

Open

ocaisa reviewed Jun 28, 2017

View reviewed changes

ocaisa mentioned this pull request Jun 28, 2017

respect --suffix-modules-path value for user-specific module path extensions #2250

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow user-local modules with hierarchical naming schemes #1472

Allow user-local modules with hierarchical naming schemes #1472

geimer commented Nov 12, 2015

hpcugentbot commented Nov 12, 2015

boegel commented Nov 13, 2015

hpcugentbot commented Nov 13, 2015

boegel Dec 14, 2015

geimer Dec 18, 2015

boegel commented Dec 14, 2015

geimer commented Jan 18, 2016

boegel commented Jan 18, 2016

geimer commented Jan 18, 2016

boegel commented Jan 18, 2016

geimer commented Jan 18, 2016

boegel commented Jan 18, 2016

geimer commented Jan 18, 2016

boegel commented Jan 18, 2016

geimer commented Jan 18, 2016

boegel commented Jan 18, 2016

boegel commented Jan 18, 2016

geimer commented Jan 20, 2016

hpcugentbot commented Jan 20, 2016

boegel Jan 20, 2016

geimer Jan 20, 2016

boegel commented Jan 20, 2016

hpcugentbot commented Jan 20, 2016

boegel commented Jan 20, 2016

hpcugentbot commented Jan 20, 2016

geimer commented Jan 20, 2016

boegel commented Jan 20, 2016

ocaisa Jun 28, 2017

boegel Jun 28, 2017

ocaisa Jun 28, 2017

ocaisa Jun 28, 2017

ocaisa Jun 28, 2017

boegel Jun 28, 2017

Allow user-local modules with hierarchical naming schemes #1472

Allow user-local modules with hierarchical naming schemes #1472

Conversation

geimer commented Nov 12, 2015

hpcugentbot commented Nov 12, 2015

boegel commented Nov 13, 2015

hpcugentbot commented Nov 13, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boegel commented Dec 14, 2015

geimer commented Jan 18, 2016

boegel commented Jan 18, 2016

geimer commented Jan 18, 2016

boegel commented Jan 18, 2016

geimer commented Jan 18, 2016

boegel commented Jan 18, 2016

geimer commented Jan 18, 2016

boegel commented Jan 18, 2016

geimer commented Jan 18, 2016

boegel commented Jan 18, 2016

boegel commented Jan 18, 2016

geimer commented Jan 20, 2016

hpcugentbot commented Jan 20, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boegel commented Jan 20, 2016

hpcugentbot commented Jan 20, 2016

boegel commented Jan 20, 2016

hpcugentbot commented Jan 20, 2016

geimer commented Jan 20, 2016

boegel commented Jan 20, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment