Using "--include-*" makes EB distributed classes in the same namespace unavailable for import #1677

chrisacbr · 2016-03-18T01:40:01Z

Given the following custom toolchain class:

from easybuild.toolchains.compiler.inteliccifort import IntelIccIfort
class NCIIccIfort(IntelIccIfort):
    NAME = 'nci_iccifort'

In EasyBuild 2.6.0 the following succeeds:

$ eb --list-toolchains --include-toolchains=/local/z00/apps/.easybuild/toolchains/*.py | grep nci
    nci_iccifort: icc, ifort

But in the development version (deployed using the latest bootstrap script with source tarballs from github) it fails with:

$ eb --list-toolchains --include-toolchains=/local/z00/apps/.easybuild/toolchains/*.py
ERROR: Failed to parse configuration options: 'Failed to import easyblock easybuild.toolchains.nci_iccifort from /local/z00/cma900/tmp/eb-_5bN_6/included-toolchains/easybuild/toolchains: No module named inteliccifort'

The toolchains package namespace search path appears to be missing the original module path and instead has the temporary path listed twice:

(Pdb) where
  /usr/lib64/python2.6/runpy.py(122)_run_module_as_main()
-> "__main__", fname, loader, pkg_name)
  /usr/lib64/python2.6/runpy.py(34)_run_code()
-> exec code in run_globals
  /local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_framework-2.7.0.dev0-py2.6.egg/easybuild/main.py(408)<module>()
-> main()
  /local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_framework-2.7.0.dev0-py2.6.egg/easybuild/main.py(197)main()
-> eb_go = eboptions.parse_options(args=args)
  /local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_framework-2.7.0.dev0-py2.6.egg/easybuild/tools/options.py(981)parse_options()
-> go_args=args, error_env_options=True, error_env_option_method=raise_easybuilderror)
  /local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_framework-2.7.0.dev0-py2.6.egg/easybuild/tools/options.py(178)__init__()
-> super(EasyBuildOptions, self).__init__(*args, **kwargs)
  /local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/vsc_base-2.4.18-py2.6.egg/vsc/utils/generaloption.py(985)__init__()
-> self.postprocess()
  /local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_framework-2.7.0.dev0-py2.6.egg/easybuild/tools/options.py(588)postprocess()
-> self._postprocess_include()
  /local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_framework-2.7.0.dev0-py2.6.egg/easybuild/tools/options.py(645)_postprocess_include()
-> include_toolchains(self.tmpdir, self.options.include_toolchains)
  /local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_framework-2.7.0.dev0-py2.6.egg/easybuild/tools/include.py(280)include_toolchains()
-> verify_imports([os.path.splitext(mns)[0] for mns in included_toolchains], 'easybuild.toolchains', tcs_dir)
  /local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_framework-2.7.0.dev0-py2.6.egg/easybuild/tools/include.py(135)verify_imports()
-> pymod = __import__(pymod_spec, fromlist=[pypkg])
> /local/z00/cma900/tmp/eb-IW66la/included-toolchains/easybuild/toolchains/nci_iccifort.py(2)<module>()
-> from easybuild.toolchains.compiler.inteliccifort import IntelIccIfort
(Pdb) p sys.modules['easybuild'].__path__
['/local/z00/cma900/tmp/eb-oJq1Lz/included-toolchains/easybuild', '/local/z00/cma900/tmp/eb-oJq1Lz/included-module-naming-schemes/easybuild', '/local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_framework-2.7.0.dev0-py2.6.egg/easybuild', '/local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_easyblocks-2.7.0.dev0-py2.6.egg/easybuild']
(Pdb) p sys.modules['easybuild.toolchains'].__path__
['/local/z00/cma900/tmp/eb-oJq1Lz/included-toolchains/easybuild/toolchains', '/local/z00/cma900/tmp/eb-oJq1Lz/included-toolchains/easybuild/toolchains']
(Pdb) p sys.modules['easybuild.toolchains.compiler'].__path__
['/local/z00/cma900/tmp/eb-oJq1Lz/included-toolchains/easybuild/toolchains/compiler', '/local/z00/cma900/tmp/eb-oJq1Lz/included-toolchains/easybuild/toolchains/compiler']
(Pdb) n
ImportError: 'No module named inteliccifort'
(Pdb) import setuptools
(Pdb) p setuptools.__path__
['/local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/setuptools-20.3-py2.6.egg/setuptools']

Similarily, MNS import fails as well:

$ eb --avail-module-naming-schemes  --include-module-naming-schemes=...
ERROR: Failed to parse configuration options: 'import_available_modules: Failed to import easybuild.tools.module_naming_scheme.categorized_hmns: No module named categorized_hmns'

This is on a CentOS 6 system:

$ rpm -q python
python-2.6.6-64.el6.x86_64

Cc: @ocaisa

The text was updated successfully, but these errors were encountered:

chrisacbr · 2016-03-18T02:18:39Z

ps. Looks like the module search path gets mangled here:

> /local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_framework-2.7.0.dev0-py2.6.egg/easybuild/tools/include.py(273)include_toolchains()
-> reload(easybuild.toolchains)
(Pdb) p sys.path
['/local/z00/cma900/tmp/eb-36DhGQ/included-toolchains', '/local/z00/cma900/tmp/eb-36DhGQ/included-module-naming-schemes', '', '/local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/distribute-0.7.3-py2.6.egg', '/local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/setuptools-20.3-py2.6.egg', '/local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/vsc_install-0.9.19-py2.6.egg', '/local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_framework-2.7.0.dev0-py2.6.egg', '/local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/vsc_base-2.4.18-py2.6.egg', '/local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_easyblocks-2.7.0.dev0-py2.6.egg', '/local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_easyconfigs-2.7.0.dev0-py2.6.egg', '/local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages', '/usr/lib64/python26.zip', '/usr/lib64/python2.6', '/usr/lib64/python2.6/plat-linux2', '/usr/lib64/python2.6/lib-tk', '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', '/usr/lib64/python2.6/site-packages', '/usr/lib64/python2.6/site-packages/gtk-2.0', '/usr/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info']
(Pdb) p sys.modules['easybuild.toolchains'].__path__
['/local/z00/cma900/tmp/eb-36DhGQ/included-toolchains/easybuild/toolchains', '/local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_framework-2.7.0.dev0-py2.6.egg/easybuild/toolchains']
(Pdb) n
> /local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_framework-2.7.0.dev0-py2.6.egg/easybuild/tools/include.py(274)include_toolchains()
-> easybuild.toolchains.__path__.insert(0, os.path.join(toolchains_path, 'easybuild', 'toolchains'))
(Pdb) p sys.modules['easybuild.toolchains'].__path__
['/local/z00/cma900/tmp/eb-36DhGQ/included-toolchains/easybuild/toolchains']
(Pdb) n
> /local/z00/easybuild/software/EasyBuild/2.7.0.dev0/lib/python2.6/site-packages/easybuild_framework-2.7.0.dev0-py2.6.egg/easybuild/tools/include.py(275)include_toolchains()
-> for subpkg in toolchain_subpkgs:
(Pdb) p sys.modules['easybuild.toolchains'].__path__
['/local/z00/cma900/tmp/eb-36DhGQ/included-toolchains/easybuild/toolchains', '/local/z00/cma900/tmp/eb-36DhGQ/included-toolchains/easybuild/toolchains']

ocaisa · 2016-03-18T07:51:35Z

Thanks for reporting this, I ran into it the other day but didn't have the time to source the problem (and I think your python fu is better than mine). For reference here was my gist https://gist.github.com/ocaisa/2294ee06303e5b5c3016
The PR that introduced it (or maybe exposed it) is #1593 with f420d39
The system (Centos7) python has setuptools 0.9.8 (at least that's the output from easy_install --version) but at runtime for the commands I default to having GC3Pie loaded which brings setuptools 18.1 with it

ocaisa · 2016-03-18T08:47:19Z

Funny thing is I get different errors with the two different version of setuptools:

-bash-4.2$ eb /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/GC3Pie-2.4.2.eb -D
ERROR: Failed to parse configuration options: 'Failed to import easyblock easybuild.tools.module_naming_scheme.custom_hierarchical_mns from /tmp/eb-5m8gqP/included-module-naming-schemes/easybuild/tools/module_naming_scheme: No module named hierarchical_mns'
-bash-4.2$ ml

Currently Loaded Modules:
  1) Stages/Devel (S)   2) Developers/InstallSoftware-2016a   3) EasyBuild/.dev   4) GC3Pie/2.4.2

  Where:
   (S):  Module is Sticky, requires --force to unload or purge

-bash-4.2$ module unload GC3Pie/2.4.2 
-bash-4.2$ eb /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/GC3Pie-2.4.2.eb -D
ERROR: Failed to parse configuration options: 'Failed to import easyblock easybuild.toolchains.imvapich2c from /tmp/eb-CCCJVO/included-toolchains/easybuild/toolchains: No module named inteliccifort'

boegel · 2016-03-18T09:16:28Z

@ocaisa what are the setuptools versions?

python -c "import setuptools; print setuptools.__version__

ocaisa · 2016-03-18T09:18:02Z

system is 0.9.8, I updated GC3Pie to use 20.3

ocaisa · 2016-03-18T09:21:20Z

Does the fact that the error in both cases says Failed to import easyblock... in both cases mean anything, since in one case we are talking about toolchains and the other the MNS (i.e., not easyblocks).

boegel · 2016-03-18T11:16:20Z

@ocaisa The error message is wrong, it always says "Failed to import easyblock" (see the verify_imports function in easybuild/tools/include.py), that should be corrected to "Failed to import module"

boegel · 2016-03-18T11:34:39Z

@chrisacbr Thanks for the detailed description, I'm able to reproduce this too now...

ocaisa · 2016-03-18T11:49:56Z

The error is being thrown in fixup_namespace_packages(easyblocks_path), there's a blank entry in sys.path...is that a problem?

boegel · 2016-03-18T11:54:52Z

I think the problem is that fixup_namespace_packages isn't being called in the subpackages (e.g. easybuild.easyblocks.generic or easybuild.toolchains.compiler).

The blank entry in sys.path is because the current directory ($PWD) is included in the Python search path, that's 'normal' in Python, that's not the issue.

ocaisa · 2016-03-18T11:56:36Z

Should we also be inserting at position 1 rather than 0 so the current dir always has preference?

boegel · 2016-03-18T16:42:32Z

Not a final solution, but this patch seems to resolve the problem.

I'm still figuring out i) why the reload statements break things, ii) why I included them in the first place, iii) why the tests aren't triggering this problem...

diff --git a/easybuild/tools/include.py b/easybuild/tools/include.py
index 7f596c9..f4867ea 100644
--- a/easybuild/tools/include.py
+++ b/easybuild/tools/include.py
@@ -177,9 +177,9 @@ def include_easyblocks(tmpdir, paths):
     # make sure easybuild.easyblocks(.generic) are imported;
     # reload them to pick up all possible paths they may be in
     import easybuild.easyblocks
-    reload(easybuild.easyblocks)
+    #reload(easybuild.easyblocks)
     import easybuild.easyblocks.generic
-    reload(easybuild.easyblocks.generic)
+    #reload(easybuild.easyblocks.generic)

     # hard inject location to included (generic) easyblocks into Python search path
     # only prepending to sys.path is not enough due to 'declare_namespace' in easybuild/easyblocks/__init__.py
@@ -217,7 +217,7 @@ def include_module_naming_schemes(tmpdir, paths):
     # inject path into Python search path, and reload modules to get it 'registered' in sys.modules
     sys.path.insert(0, mns_path)
     fixup_namespace_packages(mns_path)
-    reload(easybuild.tools.module_naming_scheme)
+    #reload(easybuild.tools.module_naming_scheme)

     # hard inject location to included module naming schemes into Python search path
     # only prepending to sys.path is not enough due to 'declare_namespace' in module_naming_scheme/__init__.py
@@ -267,11 +267,11 @@ def include_toolchains(tmpdir, paths):

     # reload toolchain modules and hard inject location to included toolchains into Python search path
     # only prepending to sys.path is not enough due to 'declare_namespace' in toolchains/*/__init__.py
-    reload(easybuild.toolchains)
+    #reload(easybuild.toolchains)
     easybuild.toolchains.__path__.insert(0, os.path.join(toolchains_path, 'easybuild', 'toolchains'))
     for subpkg in toolchain_subpkgs:
         tcpkg = 'easybuild.toolchains.%s' % subpkg
-        reload(sys.modules[tcpkg])
+        #reload(sys.modules[tcpkg])
         sys.modules[tcpkg].__path__.insert(0, os.path.join(toolchains_path, 'easybuild', 'toolchains', subpkg))

     # sanity check: verify that included toolchain modules can be imported (from expected location)

boegel · 2016-03-18T22:11:14Z

A pull request with a fix for this problem is provided in #1679.

@chrisacbr If would be great if you can confirm this fix.

chrisacbr · 2016-03-19T03:38:43Z

@boegel Yep, works for me now. Thanks very much!

boegel · 2016-03-19T08:24:15Z

@chrisacbr Thanks for the confirmation, PR #1679 is merged, so closing this issue.

ocaisa mentioned this issue Mar 18, 2016

easybuild is a namespace package (2nd attempt) (REVIEW) #1593

Merged

boegel added this to the v2.7.0 milestone Mar 18, 2016

ocaisa mentioned this issue Mar 18, 2016

--include-easyblocks broken with recent setuptools (19.4 or newer) #1678

Closed

boegel mentioned this issue Mar 18, 2016

remove reload statements in include.py, since they are not required and break --include-toolchains #1679

Merged

boegel closed this as completed Mar 19, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using "--include-*" makes EB distributed classes in the same namespace unavailable for import #1677

Using "--include-*" makes EB distributed classes in the same namespace unavailable for import #1677

chrisacbr commented Mar 18, 2016

chrisacbr commented Mar 18, 2016

ocaisa commented Mar 18, 2016

ocaisa commented Mar 18, 2016

boegel commented Mar 18, 2016

ocaisa commented Mar 18, 2016

ocaisa commented Mar 18, 2016

boegel commented Mar 18, 2016

boegel commented Mar 18, 2016

ocaisa commented Mar 18, 2016

boegel commented Mar 18, 2016

ocaisa commented Mar 18, 2016

boegel commented Mar 18, 2016

boegel commented Mar 18, 2016

chrisacbr commented Mar 19, 2016

boegel commented Mar 19, 2016

Using "--include-*" makes EB distributed classes in the same namespace unavailable for import #1677

Using "--include-*" makes EB distributed classes in the same namespace unavailable for import #1677

Comments

chrisacbr commented Mar 18, 2016

chrisacbr commented Mar 18, 2016

ocaisa commented Mar 18, 2016

ocaisa commented Mar 18, 2016

boegel commented Mar 18, 2016

ocaisa commented Mar 18, 2016

ocaisa commented Mar 18, 2016

boegel commented Mar 18, 2016

boegel commented Mar 18, 2016

ocaisa commented Mar 18, 2016

boegel commented Mar 18, 2016

ocaisa commented Mar 18, 2016

boegel commented Mar 18, 2016

boegel commented Mar 18, 2016

chrisacbr commented Mar 19, 2016

boegel commented Mar 19, 2016