Finding modules with multiple modulepaths and HMNS

[Micket]

Currently, EB won't find anything from your normal MODULEPATHs, only stuff that's in --installpath-modules.
Not to bad for sysadmins to work around (just expand the MODULEPATH to also include the "/all" module directory directly, but it gets painful when lettings users build their stuff (especially if they use multiple paths as well).
t would be nicer to just deduce these from the normal MODULEPATH's instead.

From the zoom discussion, some options presented were

1. can EasyBuild automatically add to each entry in $MODULEPATH when /Core is there already?
2. can we avoid that EasyBuild uses the full module names internally when checking for dependencies?
3. also module files that extend $MODULEPATH should iterate over existing $MODULEPATH for other "opportunities" to extend $MODULEPATH?

I think option 1 seems most promising, with a configuration option to disable it. Heck, it can even be off by default since it would be simple to enable for those who use this so it would be perfectly safe feature to add.

@Flamefire ping (since you are also interested in this).

[Flamefire]

To clarify:
There are 2 dimensions on this (or 2 issues):
First is the treatment of the --installpath <prefix2> when having multiple module trees:

• Assume MODULEPATH=<prefix1>/all/Core
• EB will extend MODULEPATH by <prefix2>/all
• EB will then try to load/find modules with full names, e.g. Core/GCC/10.2.0 which fails to find those modules from <prefix1>

--> For this options 1&2 are suitable: Either (also) check for GCC/10.2.0 or in step 2 above also add <prefix1>/all to MODULEPATH since it ends with /Core

The second issue is a problem with HMNS in general when having multiple trees: In the example above installing a compiler module into <prefix1> will only expand MODULEPATH in <prefix1>, but not in <prefix2> Although (assuming) MODULEPATH=<prefix1>/all/Core:<prefix2>/all/Core is set.
For that Solution 3 would be great: Currently it is only expanding the single path or relying on EASYBUILD_ENVVARS_USER_MODULES and such which requires users(!) to export env variables with module trees and admins to (re)build modules to support that. To avoid this the module file could extend the MODULEPATH by all existing folders with the new suffix. E.g. for each <path> in $MODULEPATH that ends in /Core check for <path>/../<suffix> and add that to MODULEPATH

[ocaisa]

You can now extend MODULEPATH in any direction you want from the system installation via the use of environment variables. We are using that configuration at JSC for user installations, which is provided as a module. The settings we use at JSC are

--subdir-user-modules=easybuild/$SYSTEMNAME/modules --envvars-user-modules=PROJECT,HOME

where $SYSTEMNAME is a default envvar set per system. The one thing that this will not give you is any builds the user makes using SYSTEM toolchain since the userpath to all/Core is not in MODULEPATH (although you could do this in the user-facing module).

If you'd like you can take a look at our module at https://gist.github.com/ocaisa/b13f243936f6de0b434edd9208ca1c76 . The basic stuff is

-- Set up exactly where we put our installations
local subdir_user_modules = pathJoin("easybuild", systemname)
local home_eb = pathJoin(os.getenv("HOME"), subdir_user_modules)
local project_eb = nil
local install_eb = home_eb
local home_install_eb = true
if os.getenv("PROJECT") then
    project_eb = pathJoin(os.getenv("PROJECT"), subdir_user_modules)
    if not os.getenv("PREFER_USER") then
        install_eb = project_eb
        home_install_eb = false
        if mode()=="load" then
            LmodMessage(yellow.."\nFound $PROJECT in the environment, using that for installation. To override this and do \n"..
                        "a personal installation set the environment variable PREFER_USER=1 and reload this module.\n"..normal)
        end
    end
end

-- ensure this is a symlink since there is very little room in HOME directories (and give advice if it isn't)
if mode()=="load" and home_install_eb then
    if lfs.symlinkattributes(pathJoin(os.getenv("HOME"),"easybuild"), "mode") ~= "link" then
        LmodError(yellow.."\nPlease configure ~/easybuild to be a symlink to a location with significant data quotas,\n"..
                  "for example with: \n  mkdir -p $PROJECT/$USER/easybuild && ln -s $PROJECT/$USER/easybuild $HOME/easybuild"..normal)

    else
        LmodMessage(yellow.."\nPerforming a personal installation. To do a project wide installation, \n"..
                   "set the $PROJECT environment variable (for example via jutil).\n"..normal)
    end
end

To get around the issue of MODULEPATH and Core we use an alias

-- Configure an alias for the eb command so that modules in JSC tree are found by eb
-- (it searches for them based on the prefix)
local stage_path = pathJoin(stages, stage)
local stage_base_moduledir = pathJoin(stage_path, "modules/all")
local stage_base_moduledir_core = pathJoin(stage_base_moduledir, "Core")
set_alias("eb","MODULEPATH=" .. stage_base_moduledir .. ':' .. stage_base_moduledir_core .. ':$MODULEPATH STAGE=' .. stage .. " eb")

[ocaisa]

I should say, we use hooks to disallow that user builds things that extend the module path (compiler, mpi), instead only using the toolchains that we provide (though this can be overridden).

[Flamefire]

I should say, we use hooks to disallow that user builds things that extend the module path (compiler, mpi), instead only using the toolchains that we provide (though this can be overridden).

Another use case: I'm regularly testing ECs and EBs using the system installations as a base but also install compilers (or more often MPIs) into a "user" module-tree. So even for devs this whole thing is a hassle.

And yes something like your module works (thanks for that) but I think with the proposed changes we could provide first-class support for HMNS and make it "just work".

[ocaisa]

For sure option 1 would avoid the need for the alias, which I think would be a good move and should not be very difficult to support (I imagine).

[ocaisa]

Another use case: I'm regularly testing ECs and EBs using the system installations as a base but also install compilers (or more often MPIs) into a "user" module-tree. So even for devs this whole thing is a hassle.

This is only relevant where you are overriding a compiler/mpi available on the system (or you intend to in the future), otherwise the branch would be completely unpopulated. In that case you can still leverage the environment variable approach, but from the user installations module. I can imagine something like

--subdir-user-modules=easybuild/modules --envvars-user-modules=SYS_INSTALL

were SYS_INSTALL is evaluated and exported in the user module.

[Flamefire]

Yes, I don't say it is impossible, just that it currently is quite some trouble and things to understand until it works. And that we should improve upon that :)

[akesandgren]

The main problem with basing things on the subdir-user-modules is that it locks down the prefix to be $HOME.

With project storage, something we and other sites use, one would want to use some freely choosen prefix, like /proj/nobackup/some-project/easybuild for instance.

Doing that excludes the usage of subdir-user-modules.

But a combination of options 1 and 3 should be able to handle this.

[Flamefire]

The main problem with basing things on the subdir-user-modules is that it locks down the prefix to be $HOME.

--envvars-user-modules can be set to a list of env vars containing the prefix, so $HOME is only the default, you can set it as you like hence your use case can already be realized.

[ocaisa]

The main problem with basing things on the subdir-user-modules is that it locks down the prefix to be $HOME.

--envvars-user-modules can be set to a list of env vars containing the prefix, so $HOME is only the default, you can set it as you like hence your use case can already be realized.

@Flamefire is right, see #3703 (comment) above for the exact use case you (@akesandgren) describe

[akesandgren]

Ah ok, didn't remember how those changes affected the use of that...