NotImplementedError on --fetch with external_module used in easyconfig

[Flamefire]

After a lengthy discussion in Slack and a longer search for the issue I think this is a bug:

eb EP-Stability-WF-1.0.4-intel-2020b-DD-3.37.0.eb --fetch -f
== Temporary log file in case of crash /tmp/eb-064p5o3j/easybuild-m72q5q32.log
== found valid index for /home/.local/easybuild/easyconfigs, so using it...
Traceback (most recent call last):
  File "/opt/compilers/python/3.9.4/none/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/compilers/python/3.9.4/none/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/.local/lib/python3.9/site-packages/easybuild/main.py", line 597, in <module>
    main()
  File "/home/.local/lib/python3.9/site-packages/easybuild/main.py", line 424, in main
    easyconfigs, generated_ecs = parse_easyconfigs(paths, validate=not options.inject_checksums)
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/tools.py", line 405, in parse_easyconfigs
    easyconfigs.extend(process_easyconfig(ec_file, **kwargs))
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 2069, in process_easyconfig
    ec = EasyConfig(spec, build_specs=build_specs, validate=validate, hidden=hidden)
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 514, in __init__
    self.parse()
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 750, in parse
    self['dependencies'] = remove_false_versions(self._parse_dependency(dep) for dep in self['dependencies'])
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 748, in remove_false_versions
    return [dep for dep in deps if not (isinstance(dep, dict) and dep['version'] is False)]
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 748, in <listcomp>
    return [dep for dep in deps if not (isinstance(dep, dict) and dep['version'] is False)]
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 750, in <genexpr>
    self['dependencies'] = remove_false_versions(self._parse_dependency(dep) for dep in self['dependencies'])
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1558, in _parse_dependency
    dependency.update(self.handle_external_module_metadata(dep[0]))
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1408, in handle_external_module_metadata
    probed_metadata = self.probe_external_module_metadata(mod_name, existing_metadata=metadata)
  File "/home/.local/lib/python3.9/site-packages/easybuild/framework/easyconfig/easyconfig.py", line 1350, in probe_external_module_metadata
    prefix = self.modules_tool.get_setenv_value_from_modulefile(mod_name, prefix_var_name)
  File "/home/.local/lib/python3.9/site-packages/easybuild/tools/modules.py", line 1180, in get_setenv_value_from_modulefile
    raise NotImplementedError
NotImplementedError

What I found:

• Error is triggered because the (dummy) NoModulesTool class is used which doesn't implement get_setenv_value_from_modulefile
• An external module is used in the dependencies which triggers

  easybuild-framework/easybuild/framework/easyconfig/easyconfig.py

  Line 1562 in fca8b93

  dependency.update(self.handle_external_module_metadata(dep[0]))

• That calls the probe_* function at

  easybuild-framework/easybuild/framework/easyconfig/easyconfig.py

  Line 1418 in fca8b93

  probed_metadata = self.probe_external_module_metadata(partial_mod_name, existing_metadata=metadata)

  using the modules-tool functionality at

  easybuild-framework/easybuild/framework/easyconfig/easyconfig.py

  Lines 1354 to 1355 in fca8b93

  	prefix = self.modules_tool.get_setenv_value_from_modulefile(mod_name, prefix_var_name)
  	version = self.modules_tool.get_setenv_value_from_modulefile(mod_name, version_var_name)

• --fetch is even documented as ignoring the modules tool at

  easybuild-framework/easybuild/tools/options.py

  Line 410 in fca8b93

  "implies --stop=fetch, --ignore-osdeps and ignore modules tool", None, 'store_true', False),

• It does so at

  easybuild-framework/easybuild/tools/options.py

  Line 1190 in fca8b93

  self.options.modules_tool = None

• with modules_tool==None the class NoModulesTool is used at

  easybuild-framework/easybuild/tools/modules.py

  Line 1680 in fca8b93

  modules_tool_class = avail_modules_tools().get(modules_tool, NoModulesTool)

  leading to the failure

So in summary: --fetch must not ignore the modules tool as resolving EC dependencies may not be possible without it.

[boegel]

--fetch was deliberately implemented in #2457 such that no modules tool is required, so eb --fetch can also be used on your laptop or another system where no modules tool may be available.
With that in mind, I don't think we should start requiring a modules tool to use --fetch.

Question is then how this problem can be fixed in another way.

Maybe get_setenv_value_from_modulefile should be implemented generically in ModulesTool, without requiring an actual modules tool (so without the call to get_value_from_modulefile which runs module show)?

[Flamefire]

Question is then how this problem can be fixed in another way.

Maybe get_setenv_value_from_modulefile should be implemented generically in ModulesTool, without requiring an actual modules tool (so without the call to get_value_from_modulefile which runs module show)?

I have no idea how this would be possible: How could you get_*_from_modulefile without a "modulefile"?

Maybe we can rather not call probe_external_module_metadata if there is no modules tool? Not sure what external_module_metadata is used for and if it is really required.

But it all boils down to: --fetch needs to resolve dependencies which isn't possible without a modules-tool, is it?
Maybe we can add a flag to ignore the modules tool and only consider dependencies defined in ECs and download everything for that use case?

[boegel]

Question is then how this problem can be fixed in another way.
Maybe get_setenv_value_from_modulefile should be implemented generically in ModulesTool, without requiring an actual modules tool (so without the call to get_value_from_modulefile which runs module show)?

I have no idea how this would be possible: How could you get_*_from_modulefile without a "modulefile"?

We can get the contents of a module file without running module show, just by reading the module file directly instead?

Maybe we can rather not call probe_external_module_metadata if there is no modules tool? Not sure what external_module_metadata is used for and if it is really required.

See https://docs.easybuild.io/using-external-modules/#using_external_modules_metadata w.r.t. metadata for external modules.

It definitely not required for just fetching sources.
In fact, external modules are totally irrelevant when doing --fetch.

But it all boils down to: --fetch needs to resolve dependencies which isn't possible without a modules-tool, is it? Maybe we can add a flag to ignore the modules tool and only consider dependencies defined in ECs and download everything for that use case?

Resolving dependencies doesn't require a modules tool, that's purely done based on *dependencies in easyconfigs.
Checking which easyconfigs are installed already is what requires a modules tool (since that implies using module avail).

[Flamefire]

We can get the contents of a module file without running module show, just by reading the module file directly instead?

But how do you find the module file? Especially finding the correct module file would required duplicating logic module show does (inside e.g. lmod).

Resolving dependencies doesn't require a modules tool, that's purely done based on *dependencies in easyconfigs. Checking which easyconfigs are installed already is what requires a modules tool (since that implies using module avail).

Ah ok, so installed modules are ignored and sources for all ECs in the dependency chain are collected even if they are installed, aren't they?

If so I'd say that probe_external_module_metadata can simply return an empty dict when "NoModulesTool" is used as-if it wasn't found.