TensorFlow builds fail on JURECA and JUWELS

[SebastianAchilles]

Follow up on the failing test reports in #13877

Building TensorFlow 2.4.1, 2.5.0 and 2.6.0 fails on JURECA and JUWELS, but not on my minimalist rocky 8.4 container.

EB Config and system info

• JURECA: AMD EPYC 7742 + RTX 8000 or A100
• JUWELS: Intel(R) Xeon(R) Gold 6148 + P100 or V100
• rocky 8.4 Container: Intel(R) Core(TM) i7-6900K

Test Reports:

• JURECA TensorFlow 2.4.1, 2.5.0, 2.6.0 foss FAILED
• JURECA TensorFlow 2.4.1, 2.5.0, 2.6.0 foss + CUDA FAILED
• JUWELS TensorFlow 2.4.1, 2.5.0, 2.6.0 foss FAILED
• JUWELS TensorFlow 2.4.1, 2.5.0, 2.6.0 foss + CUDA FAILED
• rocky 8.4 Container TensorFlow 2.4.1, 2.5.0, 2.6.0 foss SUCCESS

I am trying to understand why the builds fail on JURECA and JUWELS. And whether this is caused by to the specific configuration of these systems. Maybe @Flamefire or @boegel can give me a hint to better understand the problem?

[boegel]

@SebastianAchilles Anything useful in the logs for the failing csv_dataset_test?

[SebastianAchilles]

2021-09-09 20:56:46.792936: F tensorflow/core/platform/default/env.cc:73] Check failed: ret == 0 (11 vs. 0)Thread tf_data_private_threadpool creation via pthread_create() failed.
Fatal Python error: Aborted

This looks like there is a problem with the creation of the threads. I assume this is caused by our (too?) strict limitations on the login nodes.
On the backend nodes I can not build TensorFlow at the moment because UnZip is not in our lightweight OS image, but is required for building. I have already opened PRs for this:

• add UnZip build dependency to TensorFlow 2.4.1 easyconfigs #13958
• add UnZip build dependency to TensorFlow 2.5.0 easyconfigs #13959
• add UnZip build dependency to TensorFlow 2.6.0 easyconfigs #13960

[SebastianAchilles]

It seems like

$ ulimit -u
1024

caused the problem on the login nodes. A larger values would be needed for running the tests. @boegel you use 10k, right?

With UnZip added, I get a bit further on the backend node: Now I see

external/com_github_grpc_grpc/src/core/tsi/alts/crypt/aes_gcm.cc:25:10: fatal error: openssl/bio.h: No such file or directory
   25 | #include <openssl/bio.h>
      |          ^~~~~~~~~~~~~~~
compilation terminated.

c.f. #13960 (comment)

@boegel Do you have openssl-devel installed on your system? On the lightweight OS of the backend nodes we only have:

$ yum list installed | grep openssl
compat-openssl10.x86_64                                  1:1.0.2o-3.el8                             @AppStream   
openssl.x86_64                                           1:1.1.1g-15.el8_3                          @BaseOS      
openssl-libs.x86_64                                      1:1.1.1g-15.el8_3                          @BaseOS      
openssl-pkcs11.x86_64                                    0.4.10-2.el8                               @BaseOS

OpenSSL-1.1.eb used the fallback option and installed OpenSSL-1.1.1k.eb with EasyBuild. But it seems like openssl-devel is still required as OS dependency.

[Flamefire]

With UnZip added, I get a bit further on the backend node

What was the error when that was not added?

But it seems like openssl-devel is still required as OS dependency.

Bazel sends its greetings... Will send an easyblock PR to fix this.

[branfosj]

With UnZip added, I get a bit further on the backend node

Did it fail on Dill? We use 'source_tmpl': '%(name)s-%(version)s.zip', for that - so the UnZip would be needed to unpack that if you do not have that installed in the OS. This works because the build dependencies are loaded before extensions are unpacked.

Also, does SciPy-bundle fail in the same way? There numpy is a SOURCE_ZIP.

[SebastianAchilles]

With UnZip added, I get a bit further on the backend node

Did it fail on Dill?

Yes, it failed on Dill. On the lightweight OS of the backend nodes we do not have UnZip installed.

Also, does SciPy-bundle fail in the same way? There numpy is a SOURCE_ZIP.

Yes, I got the same error unzip not found with SciPy-bundle on our backend nodes.

But it seems like openssl-devel is still required as OS dependency.

Bazel sends its greetings... Will send an easyblock PR to fix this.

Thank you!

[Flamefire]

@SebastianAchilles Can you test with easybuilders/easybuild-easyblocks#2575?

[SebastianAchilles]

@SebastianAchilles Can you test with easybuilders/easybuild-easyblocks#2575?

I will start a test report after 6PM. Today we have maintenance day on all systems.