Add support for running TensorFlow CPU and GPU tests separately and enhance test failure reporting #2312

Flamefire · 2021-01-19T15:35:49Z

(created using eb --new-pr)

Based on

~~create less temporary directories for TensorFlow by (only) using --output_user_root #2293 (needs rebase to clean up changes)~~

boegel · 2021-02-02T16:35:42Z

@Flamefire #2293 merged, please rebase so we can get this in too before the next EasyBuild release, since it makes a couple of changes compared to #2263 (and #2292)

easybuild/easyblocks/t/tensorflow.py

boegel · 2021-02-02T17:02:49Z

easybuild/easyblocks/t/tensorflow.py

+
+            # TF can only run tests explicitely marked as 'gpu' on GPUs and those don't work on cpu
+            # See https://github.com/tensorflow/tensorflow/issues/45664
+            if device == 'cpu':


Shouldn't this be gpu?

If not, please clarify the comment, since this doesn't make sense at first glance...

Oh, it's -1... 🤦

OK, please mention what -1 does in the comment, it's easy to overlook that as 1 which strongly suggests "enable"...

I changed the comment to say "Disable GPUs..." (or so), is that enough?

easybuild/easyblocks/t/tensorflow.py

Flamefire · 2021-02-08T19:21:42Z

Test report by @Flamefire

Overview of tested easyconfigs (in order)

SUCCESS TensorFlow-2.3.1-foss-2019b-Python-3.7.4.eb
SUCCESS TensorFlow-2.2.0-fosscuda-2019b-Python-3.7.4.eb
SUCCESS TensorFlow-2.0.0-fosscuda-2019b-Python-3.7.4.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
taurusml28 - Linux RHEL 7.6, POWER, 8335-GTX, Python 2.7.5
See https://gist.github.com/5d5822c30ce87ca2c4482d33eccc2cbb for a full test report.

boegel

@Flamefire Some minor tweaks done in Flamefire#5, please check/merge so we can (finally) get this show on the road :)

boegel · 2021-02-12T13:16:47Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS TensorFlow-1.15.2-foss-2019b-Python-3.7.4.eb
SUCCESS TensorFlow-2.1.0-fosscuda-2019b-Python-3.7.4.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
node2715.swalot.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (haswell), Python 3.6.8
See https://gist.github.com/8ffd1ebd27024437e642d3ab176e5b22 for a full test report.

Flamefire · 2021-02-12T13:41:55Z

As discussed at easybuilders/easybuild-easyconfigs#11637 (comment) I added 2 more changes to a) allow running on non-GPU nodes and b) propagate cuda cache variables, see also easybuilders/easybuild-framework#3569

easybuild/easyblocks/t/tensorflow.py

boegel · 2021-02-12T14:33:10Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS protobuf-3.10.0-GCCcore-8.3.0.eb
SUCCESS Zip-3.0-GCCcore-8.3.0.eb
SUCCESS git-2.23.0-GCCcore-8.3.0-nodocs.eb
SUCCESS Java-1.8.0_281.eb
SUCCESS Java-1.8.eb
SUCCESS Bazel-0.26.1-GCCcore-8.3.0.eb
SUCCESS pkgconfig-1.5.1-GCCcore-8.3.0-Python-3.7.4.eb
SUCCESS CUDA-10.1.243-GCC-8.3.0.eb
SUCCESS gcccuda-2019b.eb
SUCCESS cuDNN-7.6.4.38-gcccuda-2019b.eb
SUCCESS NCCL-2.4.8-gcccuda-2019b.eb
SUCCESS OpenMPI-3.1.4-gcccuda-2019b.eb
SUCCESS gompic-2019b.eb
SUCCESS HDF5-1.10.5-gompic-2019b.eb
SUCCESS FFTW-3.3.8-gompic-2019b.eb
SUCCESS ScaLAPACK-2.0.2-gompic-2019b.eb
SUCCESS fosscuda-2019b.eb
SUCCESS SciPy-bundle-2019.10-fosscuda-2019b-Python-3.7.4.eb
SUCCESS h5py-2.10.0-fosscuda-2019b-Python-3.7.4.eb
SUCCESS TensorFlow-1.15.5-fosscuda-2019b-Python-3.7.4.eb

Build succeeded for 20 out of 20 (1 easyconfigs in total)
node3406.kirlia.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz (cascadelake), Python 3.6.8
See https://gist.github.com/4c6aa56918521bafca9e736cf83b6989 for a full test report.

Flamefire · 2021-02-12T17:23:00Z

Test report by @Flamefire

Overview of tested easyconfigs (in order)

SUCCESS TensorFlow-2.4.1-fosscuda-2020b.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
taurusi8021 - Linux centos linux 7.9.2009, x86_64, AMD EPYC 7352 24-Core Processor (zen2), Python 2.7.5
See https://gist.github.com/6f20da0dd6c66f23b78ca8a47aae2dae for a full test report.

boegel · 2021-02-12T18:20:09Z

@Flamefire This needs non-trivial merge conflict fixing after the merging of #2314...

… failure reporting

…leanup and enhanced comments

…sted on systems without GPUs

Flamefire · 2021-02-13T13:43:31Z

Rebased. Merge conflict was minor (self.with_cuda usage)

Flamefire · 2021-02-13T13:49:18Z

Also found 2 small issues when double-checking the code. I hope that's all, otherwise I'll check back on Monday morning

boegel · 2021-02-13T14:47:05Z

Test report by @boegel

Overview of tested easyconfigs (in order)

FAIL (build issue) TensorFlow-1.15.5-fosscuda-2019b-Python-3.7.4.eb (partial log available at https://gist.github.com/94ad9d4925bb9e1272902b5aa0ebebaa)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
node3525.doduo.os - Linux RHEL 8.2, x86_64, AMD EPYC 7552 48-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/034d4d42639626a17523682113ed3034 for a full test report.

edit: failed due to All mirrors are down: [GET returned 404 Not Found, connect timed out 🙄

boegel · 2021-02-13T16:58:32Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS TensorFlow-1.15.5-fosscuda-2019b-Python-3.7.4.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node3525.doduo.os - Linux RHEL 8.2, x86_64, AMD EPYC 7552 48-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/5619d14f59809e58cf5ea1414712447d for a full test report.

boegel · 2021-02-13T19:32:14Z

This has been extensively tested, both with existing TensorFlow easyconfigs (which shouldn't be impacted at all by these changes), as well as with easybuilders/easybuild-easyconfigs#11637 where the TensorFlow tests are being run, which shows that this is working as intended.

So let's get this merged, thanks a lot for your efforts here @Flamefire!

Flamefire mentioned this pull request Jan 19, 2021

{lib}[fosscuda/2019b] TensorFlow v2.4.1 w/ Python 3.7.4 easybuilders/easybuild-easyconfigs#11637

Merged

6 tasks

boegel changed the title ~~Add support for running CPU and GPU tests separately and enhance test failure reporting~~ Add support for running TensorFlow CPU and GPU tests separately and enhance test failure reporting Jan 20, 2021

boegel added the enhancement label Jan 20, 2021

boegel added this to the next release (4.3.3?) milestone Jan 20, 2021

boegel added the change label Jan 20, 2021

Flamefire force-pushed the 20210119163542_new_pr_UgyaqaPRcl branch from ed75407 to 2b599dd Compare January 25, 2021 11:31

Flamefire mentioned this pull request Jan 25, 2021

Update optional feature support of TensorFlow #2314

Merged

Flamefire force-pushed the 20210119163542_new_pr_UgyaqaPRcl branch from 2b599dd to 07cbbf9 Compare February 2, 2021 16:42

boegel requested changes Feb 2, 2021

View reviewed changes

boegel mentioned this pull request Feb 12, 2021

add constants for CPU/GPU device for TensorFlow tests + minor style cleanup and enhanced comments Flamefire/easybuild-easyblocks#5

Merged

boegel requested changes Feb 12, 2021

View reviewed changes

Flamefire mentioned this pull request Feb 12, 2021

{lib}[fosscuda/2020b] TensorFlow v2.4.1 w/ Python 3.8.6 easybuilders/easybuild-easyconfigs#12149

Merged

2 tasks

Flamefire commented Feb 12, 2021

View reviewed changes

easybuild/easyblocks/t/tensorflow.py Outdated Show resolved Hide resolved

boegel requested changes Feb 12, 2021

View reviewed changes

easybuild/easyblocks/t/tensorflow.py Outdated Show resolved Hide resolved

Flamefire and others added 7 commits February 13, 2021 14:41

Add support for running CPU and GPU tests separately and enhance test…

4ab5da5

… failure reporting

Adress review

547347f

Fix crash when test_jobs is unset and CUDA is not used

ff63156

add constants for CPU/GPU device for TensorFlow tests + minor style c…

d680ef7

…leanup and enhanced comments

Propagate cuda cache variables to tests

f1f7854

Rename test_jobs to test_max_parallel and allow TF to be build and te…

711e45f

…sted on systems without GPUs

fix typo in help message for test_max_parallel

71582c9

Flamefire force-pushed the 20210119163542_new_pr_UgyaqaPRcl branch from 42aeda3 to 71582c9 Compare February 13, 2021 13:43

Flamefire added 2 commits February 13, 2021 14:45

Fix missed use of max_num_test_jobs

1a5d48a

Set GPU test variables only for GPU tests

40d316d

boegel approved these changes Feb 13, 2021

View reviewed changes

boegel merged commit eec61d7 into easybuilders:develop Feb 13, 2021

Flamefire deleted the 20210119163542_new_pr_UgyaqaPRcl branch February 13, 2021 23:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for running TensorFlow CPU and GPU tests separately and enhance test failure reporting #2312

Add support for running TensorFlow CPU and GPU tests separately and enhance test failure reporting #2312

Flamefire commented Jan 19, 2021 •

edited by boegel

Loading

boegel commented Feb 2, 2021

boegel Feb 2, 2021

boegel Feb 3, 2021

Flamefire Feb 3, 2021

Flamefire commented Feb 8, 2021

boegel left a comment

boegel commented Feb 12, 2021

Flamefire commented Feb 12, 2021

boegel commented Feb 12, 2021

Flamefire commented Feb 12, 2021

boegel commented Feb 12, 2021

Flamefire commented Feb 13, 2021

Flamefire commented Feb 13, 2021

boegel commented Feb 13, 2021 •

edited

Loading

boegel commented Feb 13, 2021

boegel commented Feb 13, 2021

Add support for running TensorFlow CPU and GPU tests separately and enhance test failure reporting #2312

Add support for running TensorFlow CPU and GPU tests separately and enhance test failure reporting #2312

Conversation

Flamefire commented Jan 19, 2021 • edited by boegel Loading

boegel commented Feb 2, 2021

boegel Feb 2, 2021

Choose a reason for hiding this comment

boegel Feb 3, 2021

Choose a reason for hiding this comment

Flamefire Feb 3, 2021

Choose a reason for hiding this comment

Flamefire commented Feb 8, 2021

Overview of tested easyconfigs (in order)

boegel left a comment

Choose a reason for hiding this comment

boegel commented Feb 12, 2021

Overview of tested easyconfigs (in order)

Flamefire commented Feb 12, 2021

boegel commented Feb 12, 2021

Overview of tested easyconfigs (in order)

Flamefire commented Feb 12, 2021

Overview of tested easyconfigs (in order)

boegel commented Feb 12, 2021

Flamefire commented Feb 13, 2021

Flamefire commented Feb 13, 2021

boegel commented Feb 13, 2021 • edited Loading

Overview of tested easyconfigs (in order)

boegel commented Feb 13, 2021

Overview of tested easyconfigs (in order)

boegel commented Feb 13, 2021

Flamefire commented Jan 19, 2021 •

edited by boegel

Loading

boegel commented Feb 13, 2021 •

edited

Loading