Test failure of EB 2.8.2

[jhein32]

Hi Kenneth,

Me again - I was hoping to do the first easybuild upgrade without failures, but nope.

I did the bootstrap thing and test as follows:

-bash-4.2$ module load EasyBuild
-bash-4.2$ module list

Currently Loaded Modules:
  1) lmod/6.0.24   2) EasyBuild/2.8.2

-bash-4.2$ export TEST_EASYBUILD_MODULES_TOOL=Lmod
-bash-4.2$ python -m test.framework.suite

I get the following failures/errors:

======================================================================
ERROR: test_toy_dumped_easyconfig (test.framework.toy_build.ToyBuildTest)
Test dumping of file in eb_filerepo in both .eb and .yeb format
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sw/easybuild/software/Core/EasyBuild/2.8.2/lib/python2.7/site-packages/easybuild_framework-2.8.2-py2.7.egg/test/framework/toy_build.py", line 1149, in test_toy_dumped_easyconfig
    ec = EasyConfig(args[0])
  File "/sw/easybuild/software/Core/EasyBuild/2.8.2/lib/python2.7/site-packages/easybuild_framework-2.8.2-py2.7.egg/easybuild/framework/easyconfig/easyconfig.py", line 256, in __init__
    raise EasyBuildError("EasyConfig __init__ expected a valid path")
EasyBuildError: 'EasyConfig __init__ expected a valid path'

======================================================================
FAIL: test_environment_command (test.framework.modulestool.ModulesToolTest)
Test setting cmd via enviroment
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sw/easybuild/software/Core/EasyBuild/2.8.2/lib/python2.7/site-packages/easybuild_framework-2.8.2-py2.7.egg/test/framework/modulestool.py", line 108, in test_environment_command
    self.assertEqual(bmmt.cmd, cmd_abspath)
  File "/sw/easybuild/software/Core/EasyBuild/2.8.2/lib/python2.7/site-packages/vsc_install-0.10.11-py2.7.egg/vsc/install/testing.py", line 79, in assertEqual
    raise AssertionError("%s:\nDIFF%s:\n%s" % (msg, limit, ''.join(diff[:self.ASSERT_MAX_DIFF])))
AssertionError: '/usr/bin/echo' != '/bin/echo':
DIFF:
- /usr/bin/echo? ----


======================================================================
FAIL: test_mock (test.framework.modulestool.ModulesToolTest)
Test the mock module
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sw/easybuild/software/Core/EasyBuild/2.8.2/lib/python2.7/site-packages/easybuild_framework-2.8.2-py2.7.egg/test/framework/modulestool.py", line 88, in test_mock
    self.assertEqual(mmt.cmd, cmd_abspath)
  File "/sw/easybuild/software/Core/EasyBuild/2.8.2/lib/python2.7/site-packages/vsc_install-0.10.11-py2.7.egg/vsc/install/testing.py", line 79, in assertEqual
    raise AssertionError("%s:\nDIFF%s:\n%s" % (msg, limit, ''.join(diff[:self.ASSERT_MAX_DIFF])))
AssertionError: '/usr/bin/echo' != '/bin/echo':
DIFF:
- /usr/bin/echo? ----


----------------------------------------------------------------------
Ran 420 tests in 1335.354s

FAILED (failures=2, errors=1)
ERROR: Not all tests were successful.
Log available at /tmp/eb-4_vmaz/eb-PSJuwb/easybuild-tests-rGQ_2q.log 

Not sure why it is always here that we have issue. We are on Centos 7 if that matters.

[geimer]

I can confirm the first error (in test.framework.toy_build.ToyBuildTest: "Test dumping of file in eb_filerepo in both .eb and .yeb format") on a system running Debian 8.5 (jessie), though I don't get the other two. However, I got the following additional failure when running the test suite from a user account that doesn't have write access to the installation location:

======================================================================
FAIL: test_obtain_file (test.framework.easyblock.EasyBlockTest)
Test obtain_file method.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/eb/software/Core/EasyBuild/2.8.2/lib/python2.7/site-packages/easybuild_framework-2.8.2-py2.7.egg/test/framework/easyblock.py", line 714, in test_obtain_file
    self.assertErrorRegex(EasyBuildError, error_regex, eb.obtain_file, fn, urls=['file://%s' % tmpdir_subdir])
  File "/opt/eb/software/Core/EasyBuild/2.8.2/lib/python2.7/site-packages/vsc_install-0.10.11-py2.7.egg/vsc/install/testing.py", line 122, in assertErrorRegex
    self.assertTrue(regex.search(msg), "Pattern '%s' is found in '%s'" % (regex.pattern, msg))
AssertionError: None is not true : Pattern 'Couldn't find file thisisclearlyanonexistingfile anywhere, and downloading it didn't work either' is found in 'Failed to create directory /opt/eb/software/Core/EasyBuild/2.8.2/lib/python2.7/site-packages/easybuild_framework-2.8.2-py2.7.egg/test/framework/sandbox/sources/t/toy: [Errno 13] Permission denied: '/opt/eb/software/Core/EasyBuild/2.8.2/lib/python2.7/site-packages/easybuild_framework-2.8.2-py2.7.egg/test/framework/sandbox/sources/t''

[geimer]

Interestingly, I get additional errors (see below) when running the unit tests from my git clone used for development... vsc-base and vsc-install are up-to-date with master (i.e., v2.5.1 and v0.10.11, respectively), while all three easybuild repositories are up-to-date with develop (i.e., at this point are identical to the v2.8.2 release except for the version number). This is really weird...

======================================================================
ERROR: test_include_easyblocks (test.framework.options.CommandLineOptionsTest)
Test --include-easyblocks.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/geimer/Projects/EasyBuild/easybuild-framework/test/framework/options.py", line 1807, in test_include_easyblocks
    self.eb_main(args, logfile=dummylogfn, raise_error=True)
  File "/home/geimer/Projects/EasyBuild/easybuild-framework/test/framework/utilities.py", line 292, in eb_main
    raise myerr
EasyBuildError: 'Failed to parse configuration options: No module named python'

======================================================================
ERROR: test_include_generic_easyblocks (test.framework.options.CommandLineOptionsTest)
Test --include-easyblocks with a generic easyblock.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/geimer/Projects/EasyBuild/easybuild-framework/test/framework/options.py", line 1885, in test_include_generic_easyblocks
    self.eb_main(args, logfile=dummylogfn, raise_error=True)
  File "/home/geimer/Projects/EasyBuild/easybuild-framework/test/framework/utilities.py", line 292, in eb_main
    raise myerr
EasyBuildError: 'Failed to parse configuration options: No module named python'

======================================================================
ERROR: test_include_easyblocks_priority (test.framework.include.IncludeTest)
Test whether easyblocks included via include_easyblocks() get prioroity over others.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/geimer/Projects/EasyBuild/easybuild-framework/test/framework/include.py", line 116, in test_include_easyblocks_priority
    self.assertTrue(os.path.samefile(foo_path, test_easyblocks))
  File "/usr/lib/python2.7/posixpath.py", line 162, in samefile
    s1 = os.stat(f1)
OSError: [Errno 2] No such file or directory: '/tmp/eb-dyDbEA/eb-cU0hHn/eb-f4_aTz/eb-Ivf9vY/eb-W8n08H/included-easyblocks/easybuild'

======================================================================
FAIL: test_dump_extra (test.framework.easyconfig.EasyConfigTest)
Test EasyConfig's dump() method for files containing extra values
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/geimer/Projects/EasyBuild/easybuild-framework/test/framework/easyconfig.py", line 1362, in test_dump_extra
    self.assertEqual(rawtxt, ectxt)
  File "/home/geimer/Projects/EasyBuild/vsc-install/lib/vsc/install/testing.py", line 79, in assertEqual
    raise AssertionError("%s:\nDIFF%s:\n%s" % (msg, limit, ''.join(diff[:self.ASSERT_MAX_DIFF])))
AssertionError: 'easyblock = \'EB_foo\'\n\nname = \'foo\'\nversion = \'0.0.1\'\nversionsuffix = \'_bar\'\n\nhomepage = \'http://foo.com/\'\ndescription = "foo description"\n\ntoolchain = {\'version\': \'dummy\', \'name\': \'dummy\'}\n\ndependencies = [\n    (\'GCC\', \'4.6.4\', \'-test\'),\n    (\'MPICH\', \'1.8\', \'\', (\'GCC\', \'4.6.4\')),\n    (\'bar\', \'1.0\'),\n    (\'foobar/1.2.3\', EXTERNAL_MODULE),\n]\n\nfoo_extra1 = \'foobar\'' != 'easyblock = \'EB_foo\'\n\nname = \'foo\'\nversion = \'0.0.1\'\nversionsuffix = \'_bar\'\n\nhomepage = \'http://foo.com/\'\ndescription = "foo description"\n\ntoolchain = {\'version\': \'dummy\', \'name\': \'dummy\'}\n\ndependencies = [\n    (\'GCC\', \'4.6.4\', \'-test\'),\n    (\'MPICH\', \'1.8\', \'\', (\'GCC\', \'4.6.4\')),\n    (\'bar\', \'1.0\'),\n    (\'foobar/1.2.3\', EXTERNAL_MODULE),\n]':
DIFF:
  dependencies = [
      ('GCC', '4.6.4', '-test'),
      ('MPICH', '1.8', '', ('GCC', '4.6.4')),
      ('bar', '1.0'),
      ('foobar/1.2.3', EXTERNAL_MODULE),
+ ]- ]
- 

[geimer]

@boegel: As suggested during the conf call yesterday, I removed all .pyc and .pyo files in the framework and easyblocks repositories of my development tree and ran the testsuite again. However, I still get the failures mentioned in my previous comment...

[jhein32]

Hi,

To update, with EB 2.9.0 the

FAIL: test_environment_command (test.framework.modulestool.ModulesToolTest)
FAIL: test_mock (test.framework.modulestool.ModulesToolTest)

are no longer there.

The

======================================================================
ERROR: test_toy_dumped_easyconfig (test.framework.toy_build.ToyBuildTest)
Test dumping of file in eb_filerepo in both .eb and .yeb format
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/test/framework/toy_build.py", line 1195, in test_toy_dumped_easyconfig
    ec = EasyConfig(args[0])
  File "/sw/easybuild/software/Core/EasyBuild/2.9.0/lib/python2.7/site-packages/easybuild_framework-2.9.0-py2.7.egg/easybuild/framework/easyconfig/easyconfig.py", line 256, in __init__
    raise EasyBuildError("EasyConfig __init__ expected a valid path")
EasyBuildError: 'EasyConfig __init__ expected a valid path'

----------------------------------------------------------------------

Still exists. Since this got moved to EB 3.0, I assume this is not a surprise.

[boegel]

I haven't found time to figure out what is causing the failing tests, but as far as I can tell they're mostly isolation issues of the tests themselves.

Let's evaluate after EasyBuild v3.0 is released, and then try and find some time to debug this.

[geimer]

@boegel One more data point: With the current develop branch, the following tests were failing for me on a Debian GNU/Linux 8.6 (jessie). However, they succeeded after I installed the python-yaml package. So I guess for those two tests, it's just a matter of only executing them if the yaml module is available.

• test_add_easyconfig (test.framework.repository.RepositoryTest)
• test_toy_dumped_easyconfig (test.framework.toy_build.ToyBuildTest)

[boegel]

@geimer last issue is tackled in #2081