Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance CMakeMake easyblock to check whether correct Python installation was picked up by CMake #3399

Merged
merged 7 commits into from
Oct 16, 2024

Conversation

Flamefire
Copy link
Contributor

This is #3233 squashed and rebased to 5.0x

There was 1 commit by Danilo Gonzalez which mostly moved and removed code introduced earlier by @MartinsNadia which then was removed/changed again in the "revised version".

To make the rebase possible I squashed that together with all commits from @MartinsNadia into a single one which should be fine as the "revised version" was almost a complete rewrite.

MartinsNadia and others added 2 commits July 31, 2024 09:22
- Reintroduce function such that missing early returns can be used
- Fix regexp (Superflous group, optional stuff)
- Handle the case where the cache path is not found
- Handle $EBROOTPYTHON not set and comparing it in the presence of symlinks
Micket
Micket previously approved these changes Oct 10, 2024
Copy link
Contributor

@Micket Micket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@Micket
Copy link
Contributor

Micket commented Oct 11, 2024

One check is just stuck and I see no other way to restart it.

@Micket Micket closed this Oct 11, 2024
@Micket Micket reopened this Oct 11, 2024
@Micket
Copy link
Contributor

Micket commented Oct 11, 2024

Test report by @Micket

Overview of tested easyconfigs (in order)

Build succeeded for 7 out of 11 (11 easyconfigs in total)
vera-skylake-build - Linux Rocky Linux 8.9, x86_64, Intel Xeon Processor (Skylake, IBRS, no TSX), Python 3.6.8
See https://gist.github.com/Micket/eb25fe4d270ea3181261c70ed5cd1e69 for a full test report.

@boegel
Copy link
Member

boegel commented Oct 12, 2024

for EveryBeam-0.5.2-foss-2023b.eb (required by WSClean and DP3):

-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 

for ESPResSo-4.2.2-foss-2023b.eb:

-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 

@Micket So in both cases we're just missing Doxygen as (build) dependency?

@Flamefire
Copy link
Contributor Author

Flamefire commented Oct 12, 2024

@Micket So in both cases we're just missing Doxygen as (build) dependency?

Yes, although: "The following OPTIONAL packages have not been found: Doxygen", might still be better to have it?

Actual error is related to this PR though:

Python path errors:
Python executable does not exist: 1 (at easybuild/easyblocks/generic/cmakemake.py:392 in check_python_paths)

I'll have to check the CMakeCache.txt why the "1" is there which doesn't seem to make sense.

Micket
Micket previously requested changes Oct 12, 2024
"include_dir": [],
"library": [],
}
python_regex = re.compile(r"_?(?:Python|PYTHON)\d?_(EXECUTABLE|INCLUDE_DIR|LIBRARY)[^=]*=(.*)")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The [^=]* part matches to much. From my test builds, I have things like

PYTHON_EXECUTABLE-ADVANCED:INTERNAL=1

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe sufficient to just add a : ?

        python_regex = re.compile(r"_?(?:Python|PYTHON)\d?_(EXECUTABLE|INCLUDE_DIR|LIBRARY):[^=]*=(.*)")

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is not enough to match variables with a potential suffixes like "_DIR" or "_DEBUG", like PYTHON_LIBRARY_DEBUG. However limiting to word characters should work. See the where I also added some examples in the commit message

Variables we want to match include:
- PYTHON_EXECUTABLE:FILEPATH=/usr/bin/python3
- PYTHON_INCLUDE_DIR:PATH=
- PYTHON_LIBRARY:FILEPATH=/usr/lib/x86_64-linux-gnu/libpython3.6m.so
- PYTHON_LIBRARY_DEBUG:FILEPATH=PYTHON_LIBRARY_DEBUG-NOTFOUND
- PYTHON_LIBRARY_RELEASE:FILEPATH=PYTHON_LIBRARY_RELEASE-NOTFOUND

But not:
- PYTHON_EXECUTABLE-ADVANCED:INTERNAL=1
- PYTHON_INCLUDE_DIR-ADVANCED:INTERNAL=1

The type (after colon) is optional.
Hence only match word characters but not dashes and explicitely exclude the colon-type combination.
@boegel
Copy link
Member

boegel commented Oct 12, 2024

Test report by @boegel

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 2 (2 easyconfigs in total)
node3580.doduo.os - Linux RHEL 8.8, x86_64, AMD EPYC 7552 48-Core Processor, Python 3.6.8
See https://gist.github.com/boegel/0fde9883e29c5d8555392095abaf8840 for a full test report.

@boegel
Copy link
Member

boegel commented Oct 12, 2024

@Flamefire Still not there yet, the two easyconfigs in the test report fail with errors like:

== FAILED: Installation ended unsuccessfully: Python path errors:
Python executable path does not exist: :INTERNAL
Python include_dir path does not exist: :INTERNAL
Python library path does not exist: :INTERNAL

@Flamefire
Copy link
Contributor Author

@Flamefire Still not there yet, the two easyconfigs in the test report fail with errors like:

I added an extra capture group with the last change, so it picked up the type instead of the value...
I removed the non-capturing group which IMO makes the regex easier to read (avoids having (?::\w+)) and fixed the indices.

@boegel
Copy link
Member

boegel commented Oct 13, 2024

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS ALAMODE-1.4.2-foss-2022b.eb
  • SUCCESS ANTs-2.5.0-foss-2022b.eb
  • SUCCESS AOMP-13.0-2-GCCcore-10.2.0.eb
  • SUCCESS Arcade-Learning-Environment-0.8.1-foss-2023a.eb
  • SUCCESS Arrow-16.1.0-gfbf-2023b.eb
  • SUCCESS awscli-2.11.21-GCCcore-11.3.0.eb
  • SUCCESS btllib-1.7.0-GCC-12.3.0.eb
  • SUCCESS CVXPY-1.4.2-foss-2023a.eb
  • SUCCESS CheckM2-1.0.2-foss-2022b.eb
  • FAIL (build issue) cppyy-3.1.2-GCCcore-13.2.0.eb (partial log available at https://gist.github.com/boegel/3f48c98416f098b27fd9951040e7b24e)
  • SUCCESS DIRAC-23.0-intel-2023a.eb
  • SUCCESS Doxygen-1.9.1-GCCcore-10.3.0.eb
  • SUCCESS EGTtools-0.1.11-foss-2022a.eb
  • SUCCESS FLINT-3.1.1-gfbf-2023b.eb
  • SUCCESS FlexiBLAS-3.3.1-GCC-13.2.0.eb
  • SUCCESS flatbuffers-23.1.4-GCCcore-12.2.0.eb
  • SUCCESS fumi_tools-0.18.2-GCC-11.2.0.eb
  • SUCCESS Gymnasium-0.29.1-foss-2023a.eb
  • SUCCESS HepMC3-3.2.6-GCC-12.3.0.eb
  • SUCCESS HiGHS-1.7.0-gfbf-2023b.eb
  • SUCCESS kineto-0.4.0-GCC-12.3.0.eb
  • SUCCESS LLVM-18.1.8-GCCcore-13.3.0.eb
  • SUCCESS MDI-1.4.29-gompi-2023b.eb
  • SUCCESS MEGAHIT-1.2.9-GCCcore-9.3.0.eb
  • SUCCESS MetaBAT-2.15-gompi-2021b.eb
  • SUCCESS ncbi-vdb-3.1.1-gompi-2023b.eb
  • SUCCESS ONNX-1.15.0-gfbf-2023a.eb
  • SUCCESS OpenAI-Gym-0.26.2-foss-2022a.eb
  • FAIL (build issue) OpenCV-4.8.1-foss-2023a-contrib.eb (partial log available at https://gist.github.com/boegel/1c9736e79946cb7b75976f39150b0cdc)
  • SUCCESS OpenMEEG-2.5.7-foss-2023a.eb
  • SUCCESS OpenMM-8.0.0-foss-2022a.eb
  • SUCCESS PICI-LIGGGHTS-3.8.1-foss-2022a.eb
  • SUCCESS PyTorch-bundle-2.1.2-foss-2023a.eb
  • SUCCESS pFUnit-4.7.3-gompi-2022a.eb
  • SUCCESS poppler-24.04.0-GCC-13.2.0.eb
  • SUCCESS pybind11-2.9.2-GCCcore-11.3.0.eb
  • SUCCESS pypmt-1.2.0-gfbf-2023a.eb
  • SUCCESS python-mujoco-3.1.4-foss-2023a.eb
  • SUCCESS Rust-1.70.0-GCCcore-12.3.0.eb
  • SUCCESS SPAdes-3.15.2-GCC-10.2.0-Python-2.7.18.eb
  • SUCCESS SRA-Toolkit-3.0.5-gompi-2022b.eb
  • SUCCESS SentencePiece-0.2.0-GCC-13.2.0.eb
  • SUCCESS SeqAn-2.4.0-GCCcore-9.3.0.eb
  • SUCCESS scikit-build-core-0.5.0-GCCcore-12.3.0.eb
  • SUCCESS tmap-20220502-GCC-11.2.0.eb
  • SUCCESS UShER-0.5.0-gompi-2021b.eb
  • SUCCESS vcflib-1.0.9-gfbf-2023a-R-4.3.2.eb
  • SUCCESS xtensor-0.24.7-foss-2023a.eb
  • SUCCESS Z3-4.13.0-GCCcore-13.3.0.eb
  • SUCCESS zlib-ng-2.2.1-GCCcore-13.3.0.eb
  • SUCCESS manta-1.6.0-GCC-10.2.0-Python-2.7.18.eb
  • SUCCESS pybind11-2.7.1-GCCcore-11.2.0-Python-2.7.18.eb

Build succeeded for 50 out of 52 (52 easyconfigs in total)
node3580.doduo.os - Linux RHEL 8.8, x86_64, AMD EPYC 7552 48-Core Processor, Python 3.6.8
See https://gist.github.com/boegel/78d5a4c84c02603bbb22184a55e13e92 for a full test report.

@boegel
Copy link
Member

boegel commented Oct 13, 2024

cppyy-3.1.2-GCCcore-13.2.0.eb failed due to an existing lock, I'll upload a new test report for that.

For OpenCV-4.8.1-foss-2023a-contrib.eb it seems like we have a regression introduced by the changes here though:

== 2024-10-12 22:42:02,342 cmakemake.py:354 INFO Checking Python paths
== 2024-10-12 22:42:02,343 cmakemake.py:365 DEBUG Python related CMake cache line found: PYTHON2_EXECUTABLE:FILEPATH=
== 2024-10-12 22:42:02,343 cmakemake.py:368 INFO Python executable path: 
== 2024-10-12 22:42:02,344 cmakemake.py:365 DEBUG Python related CMake cache line found: PYTHON2_INCLUDE_DIR:PATH=
== 2024-10-12 22:42:02,344 cmakemake.py:368 INFO Python include_dir path: 
== 2024-10-12 22:42:02,344 cmakemake.py:365 DEBUG Python related CMake cache line found: PYTHON2_INCLUDE_DIR2:PATH=
== 2024-10-12 22:42:02,344 cmakemake.py:368 INFO Python include_dir path: 
== 2024-10-12 22:42:02,344 cmakemake.py:365 DEBUG Python related CMake cache line found: PYTHON2_LIBRARY:FILEPATH=
== 2024-10-12 22:42:02,344 cmakemake.py:368 INFO Python library path: 
== 2024-10-12 22:42:02,344 cmakemake.py:365 DEBUG Python related CMake cache line found: PYTHON2_LIBRARY_DEBUG:FILEPATH=
== 2024-10-12 22:42:02,344 cmakemake.py:368 INFO Python library path: 
== 2024-10-12 22:42:02,344 cmakemake.py:365 DEBUG Python related CMake cache line found: PYTHON3_EXECUTABLE:FILEPATH=/user/gent/400/vsc40023/eb_arcaninescratch/RHEL8-eb5dev-rpath/zen2-ib/software/Python/3.11.3-GCCcore-12.3.0/bin/python3
== 2024-10-12 22:42:02,344 cmakemake.py:368 INFO Python executable path: /user/gent/400/vsc40023/eb_arcaninescratch/RHEL8-eb5dev-rpath/zen2-ib/software/Python/3.11.3-GCCcore-12.3.0/bin/python3
== 2024-10-12 22:42:02,344 cmakemake.py:365 DEBUG Python related CMake cache line found: PYTHON3_INCLUDE_DIR:PATH=/user/gent/400/vsc40023/eb_arcaninescratch/RHEL8-eb5dev-rpath/zen2-ib/software/Python/3.11.3-GCCcore-12.3.0/include/python3.11
== 2024-10-12 22:42:02,344 cmakemake.py:368 INFO Python include_dir path: /user/gent/400/vsc40023/eb_arcaninescratch/RHEL8-eb5dev-rpath/zen2-ib/software/Python/3.11.3-GCCcore-12.3.0/include/python3.11
== 2024-10-12 22:42:02,344 cmakemake.py:365 DEBUG Python related CMake cache line found: PYTHON3_INCLUDE_DIR2:PATH=
== 2024-10-12 22:42:02,344 cmakemake.py:368 INFO Python include_dir path: 
== 2024-10-12 22:42:02,344 cmakemake.py:365 DEBUG Python related CMake cache line found: PYTHON3_LIBRARY:FILEPATH=/user/gent/400/vsc40023/eb_arcaninescratch/RHEL8-eb5dev-rpath/zen2-ib/software/Python/3.11.3-GCCcore-12.3.0/lib/libpython3.11.so
== 2024-10-12 22:42:02,344 cmakemake.py:368 INFO Python library path: /user/gent/400/vsc40023/eb_arcaninescratch/RHEL8-eb5dev-rpath/zen2-ib/software/Python/3.11.3-GCCcore-12.3.0/lib/libpython3.11.so
== 2024-10-12 22:42:02,344 cmakemake.py:365 DEBUG Python related CMake cache line found: PYTHON3_LIBRARY_DEBUG:FILEPATH=
== 2024-10-12 22:42:02,345 cmakemake.py:368 INFO Python library path: 
== 2024-10-12 22:42:02,346 cmakemake.py:365 DEBUG Python related CMake cache line found: _Python2_EXECUTABLE:INTERNAL=_Python2_EXECUTABLE-NOTFOUND
== 2024-10-12 22:42:02,346 cmakemake.py:368 INFO Python executable path: _Python2_EXECUTABLE-NOTFOUND
== 2024-10-12 22:42:02,353 build_log.py:226 ERROR EasyBuild encountered an error (at easybuild/eb5/easybuild-framework/easybuild/base/exceptions.py:126 in __init__): Python path errors:
Python executable path does not exist: 
Python include_dir path does not exist: 
Python include_dir path does not exist: 
Python include_dir path does not exist: 
Python library path does not exist: 
Python library path does not exist: 
Python library path does not exist:  (at easybuild/easyblocks/generic/cmakemake.py:393 in check_python_paths)

@boegel
Copy link
Member

boegel commented Oct 13, 2024

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS cppyy-3.1.2-GCCcore-13.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node3580.doduo.os - Linux RHEL 8.8, x86_64, AMD EPYC 7552 48-Core Processor, Python 3.6.8
See https://gist.github.com/boegel/cbae8ec769b48e72e80aa0e3b0344587 for a full test report.

Various expressions are considered "False" by CMake (matching case insensitively) including especially the empty string which is matched by the `.*` value capture group.
Exclude all such values when checking paths.
Do the exclusion during the collection which avoids reporting those paths.

Use named capture groups for easier understanding which group is considered.
@Flamefire
Copy link
Contributor Author

I didn't account for empty cache entries as they usually won't appear. The find-modules would set them to *-NOT-FOUND, not to empty.

It seems that OpenCV sets those cache entries even if they are empty.

I went full way and excluded any value considered "false" by CMake which should make it reasonably robust.

We could be more restrictive and use (Python\d?|PYTHON) to only include prefixes from known CMake find-modules (FindPythonInterp only uses PYTHON_EXECUTABLE without any version, the Python2/3 are the newer FindPython2/3) That would exclude the PYTHON2_EXECUTABLE set by OpenCV but I guess we do want to check those too (when non-False).

@boegel
Copy link
Member

boegel commented Oct 16, 2024

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS ALAMODE-1.4.2-foss-2022b.eb
  • SUCCESS ANTs-2.5.0-foss-2022b.eb
  • SUCCESS AOMP-13.0-2-GCCcore-10.2.0.eb
  • SUCCESS Arcade-Learning-Environment-0.8.1-foss-2023a.eb
  • SUCCESS Arrow-16.1.0-gfbf-2023b.eb
  • SUCCESS awscli-2.11.21-GCCcore-11.3.0.eb
  • SUCCESS btllib-1.7.0-GCC-12.3.0.eb
  • SUCCESS CVXPY-1.4.2-foss-2023a.eb
  • SUCCESS CheckM2-1.0.2-foss-2022b.eb
  • SUCCESS cppyy-3.1.2-GCCcore-13.2.0.eb
  • SUCCESS DIRAC-23.0-intel-2023a.eb
  • SUCCESS Doxygen-1.9.1-GCCcore-10.3.0.eb
  • SUCCESS EGTtools-0.1.11-foss-2022a.eb
  • SUCCESS FLINT-3.1.1-gfbf-2023b.eb
  • SUCCESS FlexiBLAS-3.3.1-GCC-13.2.0.eb
  • SUCCESS flatbuffers-23.1.4-GCCcore-12.2.0.eb
  • SUCCESS fumi_tools-0.18.2-GCC-11.2.0.eb
  • SUCCESS Gymnasium-0.29.1-foss-2023a.eb
  • SUCCESS HepMC3-3.2.6-GCC-12.3.0.eb
  • SUCCESS HiGHS-1.7.0-gfbf-2023b.eb
  • SUCCESS kineto-0.4.0-GCC-12.3.0.eb
  • SUCCESS LLVM-18.1.8-GCCcore-13.3.0.eb
  • SUCCESS MDI-1.4.29-gompi-2023b.eb
  • SUCCESS MEGAHIT-1.2.9-GCCcore-9.3.0.eb
  • SUCCESS MetaBAT-2.15-gompi-2021b.eb
  • SUCCESS ncbi-vdb-3.1.1-gompi-2023b.eb
  • SUCCESS ONNX-1.15.0-gfbf-2023a.eb
  • SUCCESS OpenAI-Gym-0.26.2-foss-2022a.eb
  • SUCCESS OpenCV-4.8.1-foss-2023a-contrib.eb
  • SUCCESS OpenMEEG-2.5.7-foss-2023a.eb
  • SUCCESS OpenMM-8.0.0-foss-2022a.eb
  • SUCCESS PICI-LIGGGHTS-3.8.1-foss-2022a.eb
  • SUCCESS PyTorch-bundle-2.1.2-foss-2023a.eb
  • SUCCESS pFUnit-4.7.3-gompi-2022a.eb
  • SUCCESS poppler-24.04.0-GCC-13.2.0.eb
  • SUCCESS pybind11-2.9.2-GCCcore-11.3.0.eb
  • SUCCESS pypmt-1.2.0-gfbf-2023a.eb
  • SUCCESS python-mujoco-3.1.4-foss-2023a.eb
  • SUCCESS Rust-1.70.0-GCCcore-12.3.0.eb
  • SUCCESS SPAdes-3.15.2-GCC-10.2.0-Python-2.7.18.eb
  • SUCCESS SRA-Toolkit-3.0.5-gompi-2022b.eb
  • SUCCESS SentencePiece-0.2.0-GCC-13.2.0.eb
  • SUCCESS SeqAn-2.4.0-GCCcore-9.3.0.eb
  • SUCCESS scikit-build-core-0.5.0-GCCcore-12.3.0.eb
  • SUCCESS tmap-20220502-GCC-11.2.0.eb
  • SUCCESS UShER-0.5.0-gompi-2021b.eb
  • SUCCESS vcflib-1.0.9-gfbf-2023a-R-4.3.2.eb
  • SUCCESS xtensor-0.24.7-foss-2023a.eb
  • SUCCESS Z3-4.13.0-GCCcore-13.3.0.eb
  • SUCCESS zlib-ng-2.2.1-GCCcore-13.3.0.eb
  • SUCCESS manta-1.6.0-GCC-10.2.0-Python-2.7.18.eb
  • SUCCESS pybind11-2.7.1-GCCcore-11.2.0-Python-2.7.18.eb

Build succeeded for 52 out of 52 (52 easyconfigs in total)
node3508.doduo.os - Linux RHEL 8.8, x86_64, AMD EPYC 7552 48-Core Processor, Python 3.6.8
See https://gist.github.com/boegel/d9cf61969f78c4c90ae98271f3c80b2e for a full test report.

Copy link
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@boegel boegel dismissed Micket’s stale review October 16, 2024 06:05

requested changes made

@boegel boegel merged commit 72b4653 into easybuilders:5.0.x Oct 16, 2024
19 checks passed
@Flamefire Flamefire deleted the cmake-checks branch October 16, 2024 07:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants