generated container recipes are broken because latest pip is used which is not compatible with Python 2 anymore

[nortex]

Hi,

I am currently testing the work of easybuild containers (using Singularity). No matter which image i am trying to build, it is failing during the pip installation. Here is an example:

$ [easybd] eb -C --container-config bootstrap=yum,osversion=7 TensorFlow-1.13.1-foss-2019a-Python-3.7.2.eb --container-build-image
/usr/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py:47: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now d
eprecated in cryptography, and will be removed in the next release.
  from cryptography import x509
== Temporary log file in case of crash /apps/easybd/easybuild/temp/easybuild-jB0vVf.log

WARNING: Found one or more non-allowed loaded (EasyBuild-generated) modules in current environment:
* EasyBuild/4.4.2

This is not recommended since it may affect the installation procedure(s) performed by EasyBuild.

To make EasyBuild allow particular loaded modules, use the --allow-loaded-modules configuration option.
To specify action to take when loaded modules are detected, use --detect-loaded-modules={error,ignore,purge,unload,warn}.

See http://easybuild.readthedocs.io/en/latest/Detecting_loaded_modules.html for more information.

== found valid index for /apps/easybd/easybuild/software/EasyBuild/4.4.2/easybuild/easyconfigs, so using it...
== found valid index for /apps/easybd/easybuild/software/EasyBuild/4.4.2/easybuild/easyconfigs, so using it...
== singularity tool found at /apps/easybd/singularity/3.8.0/bin/singularity
== singularity version '3.8.0' is 2.4 or higher ... OK
== sudo tool found at /usr/bin/sudo
== Singularity definition file created at /apps/easybd/easybuild/containers/Singularity.TensorFlow-1.13.1-foss-2019a-Python-3.7.2
== Running 'sudo  /apps/easybd/singularity/3.8.0/bin/singularity build  /apps/easybd/easybuild/containers/TensorFlow-1.13.1-foss-2019a-Python-3.7.2.sif /apps/easybd/easybuild/con
tainers/Singularity.TensorFlow-1.13.1-foss-2019a-Python-3.7.2', you may need to enter your 'sudo' password...
== (streaming) output for command 'sudo  /apps/easybd/singularity/3.8.0/bin/singularity build  /apps/easybd/easybuild/containers/TensorFlow-1.13.1-foss-2019a-Python-3.7.2.sif /ap
ps/easybd/easybuild/containers/Singularity.TensorFlow-1.13.1-foss-2019a-Python-3.7.2':
INFO:    Starting build...
INFO:    Skipping GPG Key Import
INFO:    Adding owner write permission to build path: /tmp/build-temp-575194767/rootfs
INFO:    Running post scriptlet
+ yum install --quiet --assumeyes epel-release
warning: /var/cache/yum/x86_64/7/extras/packages/epel-release-7-11.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for epel-release-7-11.noarch.rpm is not installed
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-9.2009.0.el7.centos.x86_64 (@base)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
+ yum install --quiet --assumeyes python setuptools Lmod
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
+ pip install -U pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/52/e1/06c018197d8151383f66ebf6979d951995cf495629fc54149491f5d157d0/pip-21.2.4.tar.gz (1.6MB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-zSvaOT/pip/setup.py", line 7
        def read(rel_path: str) -> str:
                         ^
    SyntaxError: invalid syntax

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-zSvaOT/pip/
You are using pip version 8.1.2, however version 21.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
FATAL:   While performing build: while running engine: exit status 1
ERROR: cmd "sudo  /apps/easybd/singularity/3.8.0/bin/singularity build  /apps/easybd/easybuild/containers/TensorFlow-1.13.1-foss-2019a-Python-3.7.2.sif /apps/easybd/easybuild/con
tainers/Singularity.TensorFlow-1.13.1-foss-2019a-Python-3.7.2" exited with exit code 255 and output:

[boegel]

@nortex The problem here most likely is that the most recent versions of pip don't support Python 2 anymore, so the attempt to update pip fails because the system Python (2.7) is being used...

[nortex]

@nortex The problem here most likely is that the most recent versions of pip don't support Python 2 anymore, so the attempt to update pip fails because the system Python (2.7) is being used...

Hi,

By system pip you mean the container system pip? How this can be fixed so i could build singularity containers from easybuild?

This happens with every single container i am trying to build using easybuild...

[nortex]

A reminder - is there any suggestions on how to fix this issue?

[scimerman]

Hey, yesterday I have stumbled upon the same issue. I partially solved it by taking the created Singularity container recipe and fixed the content by switching the python 2, to python 3, and then by updating the pip3 to newer version.

My workaround at the moment is: replace in the yum install part
python > python3
python-pip > python3-pip
and for the python
pip install -U pip > pip3 install -U pip
pip install easybuild > pip3 install easybuild

In my case I still had later the issue with executing the "eb" command in the container, but this is something I am resolving at the moment.

[nortex]

Hey, yesterday I have stumbled upon the same issue. I partially solved it by taking the created Singularity container recipe and fixed the content by switching the python 2, to python 3, and then by updating the pip3 to newer version.

My workaround at the moment is: replace in the yum install part python > python3 python-pip > python3-pip and for the python pip install -U pip > pip3 install -U pip pip install easybuild > pip3 install easybuild

In my case I still had later the issue with executing the "eb" command in the container, but this is something I am resolving at the moment.

Thanks for your replay, can you explain how can i change the singularity recipe that is created by easybuild? it is creating the recipe during the build and directly using it, no?

How other users are using the EasyBuild container feature? Will this problem is going to be solved anytime?

[scimerman]

Hey, yesterday I have stumbled upon the same issue. I partially solved it by taking the created Singularity container recipe and fixed the content by switching the python 2, to python 3, and then by updating the pip3 to newer version.
My workaround at the moment is: replace in the yum install part python > python3 python-pip > python3-pip and for the python pip install -U pip > pip3 install -U pip pip install easybuild > pip3 install easybuild
In my case I still had later the issue with executing the "eb" command in the container, but this is something I am resolving at the moment.

Thanks for your replay, can you explain how can i change the singularity recipe that is created by easybuild? it is creating the recipe during the build and directly using it, no?

How other users are using the EasyBuild container feature? Will this problem is going to be solved anytime?

@nortex I am not a developer, just a user, so I guess we are both interested in the answer of the second part of your question :)

About editing:
You run the command with --container-build-image, which also starts automatically building the container. If you omit this last option, the EB will just create the definition file. You can edit it first, and then build the container yourself.
To find the file, check at the output of EB, look at the part that says Singularity definition file created at ...
Edit that file and then build the singularity container by yourself. F.e. sudo singularity build someimagename.sif some/path/to/Singularity.GCC-7.3.0

[nortex]

@nortex I am not a developer, just a user, so I guess we are both interested in the answer of the second part of your question :)

About editing: You run the command with --container-build-image, which also starts automatically building the container. If you omit this last option, the EB will just create the definition file. You can edit it first, and then build the container yourself. To find the file, check at the output of EB, look at the part that says Singularity definition file created at ... Edit that file and then build the singularity container by yourself. F.e. sudo singularity build someimagename.sif some/path/to/Singularity.GCC-7.3.0

I see, so you basically build the image directly with Singularity, that is what i thought of too.

There is an option in EasyBuild to use recipe template, so you can use the created definition file from easybuild, change the python lines and use the flag: --container-template-recipe Singularity.foss-2019a.tmpl

and EasyBuild should create the container taking in account the fixed template. The problem is that is not working too and i am getting EasyBuild errors... I think that new easybuild version is not working well with the containerize featute...

Thank you, the second question was indeed directed to developers :)

[scimerman]

I also tested the template option, and yes, it fails. This is due to the fact that template just leaves a variable of %post installing package list. The list itself is "currupted". Therefore the template without all the correct python and pip versions, like what I suggested, will also fail.
I found the issue inside easybuild-framework repository, and suggested a fix. I checked and that fix should work for most of the platforms (all the rhel based, ubuntu and sles). Lets see if will be accepted

#3945