Local (--user) installation through distutils does not place easyconfigs in the standard path

[itkovian]

Typically, installed files end up in

$HOME/.local/lib/python2.x/site-packages

However, when installing easybuild-easyconfigs using python 2.7, with distutils, they end up being copied to

$HOME/.local

And thus live in

$HOME/.local/easybuild/easyconfigs

There, they are not picked up by the framework, see easybuild-framework for the issue.

How to reproduce:

On the UGent HPC cluster login nodes:

module load Python
cd <easybuild-easyconfigs repo clone>
python ./setup.py install --user

[boegel]

This issue should be reported in the easybuild-framework repo.

The fact that easybuild-easyconfigs get installed differently by distutils is not something we can change easily, so we should just support the path that it gets installed in in the framework.

Can you open the issue there, and close this one by mentioning it was moved there (with mention of the issue number in easybuild-framework)?

[itkovian]

Well, I see no reason why this should be different from the install done for the easyblocks. So it it ends up being different, something is wrong imho with the setup for easybuild-easyconfigs. The other issue, that you should be able to deal with other locations has been openen on the easybuild-framework repo, see easybuilders/easybuild-framework#356

[boegel]

Are you saying that easybuild-easyblock is being installed as an egg, i.e. to $HOME/.local/lib/python2.x/site-packages instead of directly in $HOME/.local ?

[itkovian]

No, easybuild-easyblocks ends up in the local site-pckages dir, easybuild-easyconfig does not, which strikes me as odd and suspicious. There should be no reason why this happensm if easybuild-easyconfig gets installed as a regular python package. Locally, that is.

[boegel]

For easybuild-easyconfigs, the parameter zip_safe is set to False in setup.py, which is not the case for the other EasyBuild packages, which is why the behavior is different.

See http://packages.python.org/distribute/setuptools.html#setting-the-zip-safe-flag for documentation on zip_safe for setuptools (it enforces easy_install to install unzipped eggs).

setuptools is very conservative, and because we use stuff like __path__ and __file__ in the framework/easyblocks code, it also install these as unzipped eggs, just to be sure.

It appears that distutils is interpreting the zip_safe flag differently though, and thus installs the easyconfigs differently.

We will need to keep the zip_safe flag for easybuild-easyconfigs, otherwise the easyconfigs will be installed in a zipped eggs, which makes them difficult to access once installed.

So, we should probably also use zip_safe = False for framework/easyblocks (to get the same behavior as with easyconfigs and avoid confusion) and fix easybuilders/easybuild-framework#356 so that eb picks up the easyconfigs also in that case.

[itkovian]

But it should never install in some other directory, zipped or not.

[itkovian]

And under distutils, the easyblocks repo is installed to the right path, i.e., /site-packages. So it does not have anything to do with zipping or unzipping, imho.

[boegel]

My guess is that with zip_flag = False, distutils is being stupid and installs the easyconfigs directly under the prefix.

Can you check that? I.e., clone easybuild-easyconfigs, adjust the setup.py to remove the zip_false = True, and then do the installation?

[JensTimmerman]

I don't think we can fix what distutils is doing wrong,
and if you prefer using distutils, you have to go with what it does...

[itkovian]

Sure, but if the rest cannot pick up the easyconfigs, then you should not state that it works with distutils.

[JensTimmerman]

Well, we can fix this by giving a warning when we can't find it in the expected place, and then go on and checking the prefix directory itself.

But this is different to trying to fix the installation process.
We need to have a good way to do automatic discovery of easyconfigs anyway when they are not explicitly configured.

[boegel]

We can fix this easily in the framework, by also trying the option of looking for the path easybuild/easyconfigs directly in the prefix.

I discussed this with Andy on the train, we indeed shouldn't try to fix distutils, because it does what it thinks it should be doing, i.e. installing data files directly in the prefix (setuptools does that in an (unzipped) egg, and it's hard to make it do something else).

So, we should just go with that, and just support both install paths, as will be done in easybuilders/easybuild-framework#356 .

So, unless someone objects, I'm going to close this (non-)issue, it doesn't require any changes in easybuild-easyconfigs.