bootstrap script fails on Ubuntu 16.04

[riccardomurri]

The bootstrap script apparently fails on Ubuntu 16.04 ("xenial"), because of a
conflict between the "distribute" package included in bootstrap_eb.py and the
system-provided setuptools:

+++ STAGE 0: installing distribute via included (patched) distribute_setup.py...


[[DEBUG]] preparing environment...
[[DEBUG]] Preparing for path /tmp/tmppD0F_b
[[DEBUG]] os.environ['PYTHONPATH'] after reset:
[[DEBUG]] $EASYBUILD_MODULES_TOOL set to Lmod
[[DEBUG]] distribute_setup.__file__: /tmp/tmppD0F_b/distribute_setup.py
Downloading http://hpcugent.github.io/easybuild/files/distribute-0.6.49-patched1.tar.gz
Extracting in /tmp/tmp_9cDov
Now working in /tmp/tmp_9cDov/distribute-0.6.49
Installing Distribute
Traceback (most recent call last):
  File "setup.py", line 45, in <module>
    exec(init_file.read(), d)
  File "<string>", line 8, in <module>
  File "/tmp/tmp_9cDov/distribute-0.6.49/setuptools/__init__.py", line 2, in <module>
    from setuptools.extension import Extension, Library
  File "/tmp/tmp_9cDov/distribute-0.6.49/setuptools/extension.py", line 5, in <module>
    from setuptools.dist import _get_unpatched
  File "/tmp/tmp_9cDov/distribute-0.6.49/setuptools/dist.py", line 6, in <module>
    from setuptools.command.install import install
  File "/tmp/tmp_9cDov/distribute-0.6.49/setuptools/command/__init__.py", line 8, in <module>
    from setuptools.command import install_scripts
  File "/tmp/tmp_9cDov/distribute-0.6.49/setuptools/command/install_scripts.py", line 3, in <module>
    from pkg_resources import Distribution, PathMetadata, ensure_directory
  File "/tmp/tmp_9cDov/distribute-0.6.49/pkg_resources.py", line 2897, in <module>
    add_activation_listener(lambda dist: dist.activate())
  File "/tmp/tmp_9cDov/distribute-0.6.49/pkg_resources.py", line 712, in subscribe
    callback(dist)
  File "/tmp/tmp_9cDov/distribute-0.6.49/pkg_resources.py", line 2897, in <lambda>
    add_activation_listener(lambda dist: dist.activate())
  File "/tmp/tmp_9cDov/distribute-0.6.49/pkg_resources.py", line 2329, in activate
    self.insert_on(path)
  File "/tmp/tmp_9cDov/distribute-0.6.49/pkg_resources.py", line 2430, in insert_on
    "with distribute. Found one at %s" % str(self.location))
ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /usr/lib/python2.7/dist-packages
Something went wrong during the installation.
See the error message above.
[[ERROR]] Installing distribute which should deliver easy_install failed?

[riccardomurri]

Here's the MD5 of the bootstrap script, for future reference::

$ md5sum bootstrap_eb.py
616ff1c135c0db78ffcc7c84a3c4c828  bootstrap_eb.py

[riccardomurri]

And here's the installed setuptools::

$ dpkg -l *setuptools*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                      Version                   Architecture              Description
+++-=========================================-=========================-=========================-=======================================================================================
ii  python-setuptools                         20.7.0-1                  all                       Python Distutils Enhancements
un  python-setuptools-doc                     <none>                    <none>                    (no description available)
un  python-setuptools-whl                     <none>                    <none>                    (no description available)
un  python3-setuptools                        <none>                    <none>                    (no description available)

[riccardomurri]

For the record, we have run into similar issues with GC3Pie installation in the past; our solution has been to first create a virtualenv (isolated, with --no-site-packages) and then start installing Python stuff into it.

[boegel]

@riccardomurri can you check whether disabling stage 0 fixes the issue? See http://easybuild.readthedocs.io/en/latest/Installation.html#skipping-the-installation-of-easy-install-stage-0 .

Maybe we can enhance the bootstrap script to skip stage 0 if a sufficiently recent setuptools is readily available.

Thanks for the suggestion on using virtualenv, but doesn't that thicken the problem by introducing another dependency (i.e. a sufficiently recent version of virtualenv)?

[boegel]

BTW, this looks awfully similar to an issue we've hit on Travis recently: #1869

[riccardomurri]

Skipping stage0 solves the immediate issue, but the bootstrap script dies later
on::

[[INFO]] running post install command 'easy_install --upgrade --prefix=/tmp/tmpcEJxtd/eb_stage1 vsc-base'
Searching for vsc-base
Reading https://pypi.python.org/simple/vsc-base/
Best match: vsc-base 2.5.1
Processing vsc_base-2.5.1-py2.7.egg
vsc-base 2.5.1 is already the active version in easy-install.pth
Installing optcomplete.bash script to /tmp/tmpcEJxtd/eb_stage1/bin
Installing startlogdaemon script to /tmp/tmpcEJxtd/eb_stage1/bin
Installing logdaemon script to /tmp/tmpcEJxtd/eb_stage1/bin

Using /tmp/tmpcEJxtd/eb_stage1/lib/python2.7/site-packages/vsc_base-2.5.1-py2.7.egg
Processing dependencies for vsc-base
Finished processing dependencies for vsc-base
Traceback (most recent call last):
  File "bootstrap_eb.py", line 790, in <module>
    main()
  File "bootstrap_eb.py", line 590, in main
    templates = stage1(tmpdir, sourcepath, distribute_egg_dir)
  File "bootstrap_eb.py", line 359, in stage1
    os.environ['PYTHONPATH'] = distribute_egg_dir
  File "/usr/lib/python2.7/os.py", line 473, in __setitem__
    putenv(key, item)
TypeError: putenv() argument 2 must be string, not None

It seems there is an assumption that distribute is available anyway...

Why prefer distribute (old and unmaintained) over a recent release of
setuptools?

[riccardomurri]

The following patch solves the issue with the distribute egg path::

$ diff -wu bootstrap_eb.py{.orig,}
--- bootstrap_eb.py.orig    2016-08-29 13:12:28.215370160 +0000
+++ bootstrap_eb.py 2016-08-29 13:09:47.739382373 +0000
@@ -356,7 +356,10 @@

     # clear the Python search path, we only want the individual eggs dirs to be in the PYTHONPATH (see below)
     # this is needed to avoid easy-install.pth controlling what Python packages are actually used
+    if distribute_egg_dir is not None:
     os.environ['PYTHONPATH'] = distribute_egg_dir
+    else:
+   del os.environ['PYTHONPATH']

     # template string to inject in template easyconfig
     templates = {}

Unfortunately, EB's bootstrap procedure fails now at stage 2::

+++ STAGE 2: installing EasyBuild in /opt/easybuild with EasyBuild from stage 1...
[...]
== postprocessing...
== sanity checking...
== FAILED: Installation ended unsuccessfully (build directory: /tmp/tmpmksiGU/EasyBuild/2.8.2/dummy-dummy): build failed (first 300 chars): Sanity check failed: EasyBuild failed to install, cmd '/usr/bin/python -c "import easybuild"' (stdin: None) output: Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named easybuild
, sanity check command eb --version exited with code 1 (output: /usr/bi
== Results of the build can be found in the log file(s) /tmp/eb-9kYMCe/easybuild-EasyBuild-2.8.2-20160829.131055.tMzpa.log
Traceback (most recent call last):
  File "bootstrap_eb.py", line 793, in <module>
    main()
  File "bootstrap_eb.py", line 596, in main
    stage2(tmpdir, templates, install_path, distribute_egg_dir, sourcepath)
  File "bootstrap_eb.py", line 514, in stage2
    easybuild_main()
  File "/tmp/tmpmksiGU/eb_stage1/lib/python2.7/site-packages/easybuild_framework-2.8.2-py2.7.egg/easybuild/main.py", line 389, in main
    ecs_with_res = build_and_install_software(ordered_ecs, init_session_state, exit_on_failure=exit_on_failure)
  File "/tmp/tmpmksiGU/eb_stage1/lib/python2.7/site-packages/easybuild_framework-2.8.2-py2.7.egg/easybuild/main.py", line 149, in build_and_install_software
    raise EasyBuildError(test_msg)
easybuild.tools.build_log.EasyBuildError: 'Build of /tmp/tmpmksiGU/EasyBuild-2.8.2.eb failed (err: \'build failed (first 300 chars): Sanity check failed: EasyBuild failed to install, cmd \\\'/usr/bin/python -c "import easybuild"\\\' (stdin: None) output: Traceback (most recent call last):\\n  File "<string>", line 1, in <module>\\nImportError: No module named easybuild\\n, sanity check command eb --version exited with code 1 (output: /usr/bi\')'

In this case, the trouble might come from /usr/bin/python being Python 3 on Ubuntu 16.04.

[riccardomurri]

No, wait, /usr/bin/python was fine -- the issue was a missing tclsh interpreter.
So maybe this is a different issue: bootstrap_eb.py depends on TCL-syntax modules even if Lmod is being used with pure-Lua modules. (In other words, bootstrap_eb.py generates a TCL module file regardless of EASYBUILD_MODULE_SYNTAX)

To summarize, in order to bootstrap EB on a Ubuntu 16.04 host, one needs:

1. sudo apt-get install -y python-minimal python-setuptools
2. sudo apt-get install -y lmod tcl
3. source /usr/share/lmod/lmod/init/bash
4. download bootstrap_eb.py and apply the patch above
5. export EASYBUILD_BOOTSTRAP_SKIP_STAGE0=1
6. python2.7 bootstrap_eb.py $EASYBUILD_PREFIX

[boegel]

Why prefer distribute (old and unmaintained) over a recent release of setuptools?

Initially this was done so we have a setuptools installation that we know how to deal with (I've been bitten in the ass by behavioural changes in setuptools too many times in the past), but it looks like we won't be able to keep that up... Instead, we'll need to find a sufficiently recent yet properly behaving version of setuptools we can use in stage0.

It's probably not a bad idea to make the bootstrap script smarter such that it tries to figure out whether a usable setuptools is already available...

[boegel]

@riccardomurri I've enhanced the bootstrap script to automagically skip stage0 in case a suitable setuptools is already found, cfr. #1946

This won't solve all the problems you listed, but it's a start.

I'll look into the module syntax issue later, and maybe also to modernise the setuptools that is being installed in stage0.

[boegel]

@riccardomurri I started looking into the issue of $EASYBUILD_MODULE_SYNTAX not being picked up by the bootstrap script...

It should be picked up, but I think it may only be picked up when it's exported, i.e.:

# bootstrap picks up $EASYBUILD_MODULE_SYNTAX correctly like this?
export EASYBUILD_MODULE_SYNTAX=Lua
python bootstrap_eb.py <prefix>

vs

# bootstrap doesn't pick up $EASYBUILD_MODULE_SYNTAX when defined like this?
EASYBUILD_MODULE_SYNTAX=Lua python bootstrap_eb.py <prefix>

See also the changes in the Travis config file at #2003

[boegel]

@riccardomurri ping?