feat: new FindPython support

[henryiii]

This refactors the targets and tooling to make this cleaner, simpler, more modular, and (of course, given the title) adds support for FindPython in CMake 3.12+. Minimum versions of CMake now explicitly fixed, listed, and tested (3.4+ Linux, 3.7 macOS, 3.8 Windows for building pybind11's own tests, probably 3.4+ for submodule/config mode on all platforms).

Modular targets

There are now a collection of modular targets, and you can mix and match. The modules beyond pybind11::module and pybind::embed are considered advanced, but provide better support for integrating pybind11 into an existing or complex build than before. Targets (other than headers) are generated at run/configure import time, which allows better customization for CMake version or generator (MSVC + CMake < 3.11, for example).

• pybind11::headers - Just the PyBind11 headers and minimum compile requirements
• pybind11::pybind11 - Python headers too
• pybind11::module - for extension modules
• pybind11::embed - for embedding the Python interpreter
• pybind11::python_link_helper - Just the "linking" part of pybind11:module
• pybind11::python2_no_register - Quiets the warning/error when mixing C++14+ and Python 2, also included in pybind11::module
• pybind11::lto / pybind11::thin_lto - An alternative to INTERPROCEDURAL_OPTIMIZATION
• pybind11::windows_extras - Bigobj and mp for MSVC.

The stripping feature has been made available as pybind11_strip(target), as well. The extension feature is available as pybind11_extension(target) (like everything else, this works with both modes).

These targets are now created in a common place, pybind11Common.cmake, which is shared across Config/CMakeLists, so both modes share the same initialization, with the single exception of headers, which is a normal, exported target - as it's the only one that contains non-"imported" information. All classic Python setup is in pybind11Tools.cmake and all new Python setup is in pybind11NewTools.cmake.

INTERPROCEDURAL_OPTIMIZATION

This has been refactored (mostly due to a bug in CMake 3.15-3.18's FindPython) to be simpler: If CMAKE_INTERPROCEDURAL_OPTIMIZATION is defined, use that (on or off). If not, add pybind11::lto or pybind11::thin_lto to the created target. This provides a way to turn it off (which was missing before), and a way to use the built-in CMake support if you know it supports your situation (it is affected by bugs due to CMake polices). If you target MSVC, INTERPROCEDURAL_OPTIMIZATION uses incremental IPO, which needs all targets to have it enabled, so it is better to use the CMAKE variable instead of the property - but advanced users can always set the variable to OFF and then add it to each target. (The bug is fixed for CMake 3.18.2)

FindPython Mode

To use the new FindPython support, simply use find_package(Python COMPONENTS Interpreter Development) before adding PyBind11 - this way, you can control the hints, run multiple times, etc. You can also use -DPYBIND11_FINDPYTHON=ON to force the search internally (very useful for building tests, etc).

Example:

find_package(Python COMPONENTS Interpreter Development)
find_package(pybind11 CONFIG)

# pybind11 method:
pybind11_add_module(MyModule1 src1.cpp)

# Pure method:
Python_add_library(MyModule2 src2.cpp)
target_link_libraries(MyModule2 PUBLIC pybind11::headers pybind11::lto)
set_target_properties(MyModule2 PROPERTIES
                                CXX__VISIBILITY_PRESET hidden
                                VISIBLITY_INLINES_HIDDEN ON)
pybind11_strip(MyModule2)

NOPYTHON mode

Manually selectable PYBIND11_NOPYTHON=ON (also activated by calling FindPython2 and FindPython3 without FindPython), ideal for multi-target support, Scikit-Build PythonExtensions integration, OpenCV interaction, and more. Disables Python-specific features but still provides pybind11::headers, pybind11::python2_no_register, pybind11::lto, pybind11::thin_lto, pybind11::windows_extras, and pybind11_strip.

Classic Python mode

Support for venv, virtualenv, and conda added. GitHub Action's setup-python activation, too.

---

TODO:

• THIN_LTO is not respected yet
• PyPy is currently not included in the search - fixed, but due to bug, doesn't produce the correct extensions. Bug will be fixed for PyPy2 in 3.18.2.
• Verify COMPILER_LANGUAGE doesn't leak into MSVC on CMake < 3.12
• Add docs

• Closes Pybind11 on Windows might need a CMake version newer than 3.10.1? #2357 - Supports older CMake's on Windows
• Closes CMake warning about INTERPROCEDURAL_OPTIMIZATION from vcpkg.cmake #2355, closes GCC 4.8 LTO Crashes #1415 - IPO can be deactivated or changed to CMake built-in
• Closes CMake 3.12+ Python Find Routines, IPO/LTO #2201 - Adds FindPython support
• Helps with FindPythonLibsNew.cmake finds wrong python interpreter when using pyenv #2154 - FindPython supports venv/virtualenv/conda/GHA; added support for those to FindPythonLibsNew too, for parity (and easier testing). Not sure about vanilla pyenv though.
• Closes FindPythonLibsNew.cmake module is not suitable for cross-compilation environment #2139 - FindPython can be used in cross-compilation
• Closes Inconsistent detection of python version in pybind11 and default cmake #1090 - Detection mismatch
• Closes Targeting multiple Python versions at once #748 - Multiple Pythons should be targetable via FindPython 2/3 or multiple calls + pybind11::headers (probably with NOPYTHON mode)
• Closes pybind11_add_module unexpected error after PR #2368 #2373 by modifying a check to be sensitive to this sort of mistake
• Helps with Recent CMake changes prefer pybind11 installed in /usr/local/include to development files [macOS] #2381 if CMake is less than 3.12 by correctly making targets or reusing targets in all cases (in tests)
• Closes CMake's 'find_host_package(PythonInterp 2.7)' founds wrong Python version: #2386 by allowing NOPYTHON mode to avoid interaction with OpenCV
• Closes arm cross compilation issue when need to compile to different python #1718 by allowing either a vanilla search for FindPython, or NOPYTHON mode and manual setup.
• Closes [BUG] CMP0069 #2406 - IPO + old policies does not change

Bugs reported and fixed in CMake 3.18.2 (upcoming) as a result of this work:

• INTERPROCEDURAL_OPTIMIZATION can't be used with Python_add_library
• wrong pypy extensions produced
• pypy virtual environments may be unfindable

[rwgk]

Wow, that's an impressive amount of work!

I don't have the background to meaningfully review the details, but one high-level observation: when seeing PYBIND11_NEW_PYTHON for the first time I had no clue what it might be doing. Looking through your changes I found:

PYBIND11_NEW_PYTHON "Force new findPython"

Aha! :-)
How about: PYBIND11_FORCE_NEW_FINDPYTHON?
It's only a little bit longer but gives the right clues even at first sight.

[henryiii]

How about PYBIND11_FINDPYTHON?

[rwgk]

How about PYBIND11_FINDPYTHON?

Looks good to me. 👍

[henryiii]

Merging if this passes! Made very minor (mostly docs) improvements after one more read over.

[Adrimeov]

Hi,
We have recently start working with Pybind11 and we find it extremely useful. However, the only problem we encounter now is that building the .cpp file with the following command :
c++ -O3 -Wall -shared -std=c++11 -fPIC python3 -m pybind11 --includes example.cpp -o examplepython3-config --extension-suffix

seems to generate a .o file that cannot be imported on a different machine (Linux and Mac Os x86) than the host build machine. Reading the documentation and you exchange on git made us a bit confuse on how to do this . Could you refer us to an example of such a use case ?

Thank you.

[henryiii]

I'm no expert in cross-compilation, but combined with Python, I don't know if it would be a good idea. The standard way to do this would be to setup a CI job (using GitHub Actions, Azure, GitLab CI, Travis, AppVeyor, or CircleIO, maybe DroneIO) that runs on each of the major OSs (such as https://scikit-hep.org/developer/gha_wheels) that you can then download binaries for each platform you support.

I would use setuptools (if you have a simple build) or CMake (more complex build) to get the compilation right on each platform, as the commands will vary.

I don't think this has anything to do with the new FindPython support, though. ;)

[krburkhart]

Hello, we have a large source tree with a few pybind11 targets. We use multiple python versions which requires us to build the entire tree multiple times. I am very interested in building pybind11 so targets against multiple versions of python in the same run. I have not been able to find any examples of doing this, but I see comments to the effect that it is possible. I have cmake 3.21 and pybind11 2.6.1. Can someone point me in the right direction? Sample CMakeLists.txt:

cmake_minimum_required(VERSION 3.21)
set(CMAKE_CXX_STANDARD 20)           # c++ standard version
set(CMAKE_CXX_STANDARD_REQUIRED ON)  # disable fallback behavior
set(CMAKE_CXX_EXTENSIONS OFF)        # no compiler extensions

project(tst)

set(pybind11_ROOT /opt/pb11)
set(PYTHON_ROOT /opt/conda/envs/py3.9.6)
#set(PYTHON_EXECUTABLE /opt/conda/envs/py3.9.6/bin/python3)
find_package(Python COMPONENTS Interpreter Development)
find_package(pybind11 CONFIG)
pybind11_add_module(tst-3.9.6 tst.cpp)

set(PYTHON_ROOT /opt/conda/envs/py3.9.7)
#set(PYTHON_EXECUTABLE /opt/conda/envs/py3.9.7/bin/python3)
find_package(Python COMPONENTS Interpreter Development)
find_package(pybind11 CONFIG)
pybind11_add_module(tst-3.9.7 tst.cpp)

Thank you.

[henryiii]

Could you please use 2.8.1 instead of 2.6.1? CMake support is likely better. Also, I'd recommend trying to use the native tools more heavily (like python_add_library), and just use the light-weight pybind11 ones (like pybind11::headers targets). You might be able to do this with pybind11_add_module, but there might be issues. What are you seeing?

Also, are you really sensitive to patch releases? A build should work on all patch releases of Python, unless you statically link or something like that.

[krburkhart]

I will download 2.8.1. I'm not really sensitive to patch releases, I just created two conda environments to try to get this working.

Also, what is "native tools"? I will look at pybind11_add_module.

Thank you Henry

[henryiii]

I would build for two different minor releases of Python, as otherwise you'll be making exactly matching extensions, so there's no difference in the output, one will override the other. You'll make cp39 both times.

By native tools, I mean the ones provided by FindPython, rather than our wrappers. We haven't put careful thought into building multiple Python extensions at once (normally, you are inside a Python environment and building for that environment, there aren't a lot of reasons to build against several external Pythons), but I think CMake's FindPython has. We do provide lots of granular targets to make this easy to do, though.

[henryiii]

Also, be careful about building against Anaconda Python. They have custom compilers for anaconda, so you have to use those, not your system compilers, like normal copies of Python.

[tuoyuanl]

@henryiii

normally, you are inside a Python environment and building for that environment

This is true when people are just developing Python modules consist of native extensions, for which native source code are just bundled in a pypi package. Simply running pip install would invoke cmake to build it for the current running Python environment. All looks easy and good.

However, there is another situation that people are working on an existing native codebase and just want to provide a Python binding library as side-products to distribute. Instead of filling the build configuration in setup.py and then entering virtualenvs and running commands to build the wheel for different Python version, directly compile multiple extension.so in single cmake command would be more comfortable and familiar for us C++ guys.

People have proposed many workaround for such case, like #748 and ROCm/AMDMIGraphX#655 . But an official solution would be more favourable.

[henryiii]

One of the reasons for FindPython was to support multiple Pythons at once. We might already support it when using FindPython; we just haven't done much with setting things up to make sure we also support it.

You have a typo, it's Python_ROOT, not PYTHON_ROOT:

cmake_minimum_required(VERSION 3.21)
set(CMAKE_CXX_STANDARD 20)           # c++ standard version
set(CMAKE_CXX_STANDARD_REQUIRED ON)  # disable fallback behavior
set(CMAKE_CXX_EXTENSIONS OFF)        # no compiler extensions

project(tst)

set(pybind11_ROOT /opt/pb11)
set(Python_ROOT /opt/conda/envs/py3.9.6)
find_package(Python COMPONENTS Interpreter Development)
find_package(pybind11 CONFIG)
pybind11_add_module(tst-3.9.6 tst.cpp)

set(Python_ROOT /opt/conda/envs/py3.9.7)
find_package(Python COMPONENTS Interpreter Development)
find_package(pybind11 CONFIG) # maybe optional?
pybind11_add_module(tst-3.9.7 tst.cpp)

But if that doesn't work (and if it doesn't, I'd be willing to help work fixing whatever doesn't work if shown), then try the native tools:

cmake_minimum_required(VERSION 3.21)
set(CMAKE_CXX_STANDARD 20)           # c++ standard version
set(CMAKE_CXX_STANDARD_REQUIRED ON)  # disable fallback behavior
set(CMAKE_CXX_EXTENSIONS OFF)        # no compiler extensions

project(tst)

set(pybind11_ROOT /opt/pb11)
set(pybind11_NOPYTHON TRUE)
find_package(pybind11 CONFIG)

set(Python_ROOT /opt/conda/envs/py3.9.6)
find_package(Python COMPONENTS Interpreter Development)

Python_add_library(tst-3.9.6 MODULE tst.cpp)
target_link_libraries(tst-3.9.6 PUBLIC pybind11::module)

set(Python_ROOT /opt/conda/envs/py3.9.7)
find_package(Python COMPONENTS Interpreter Development)
Python_add_library(tst-3.9.7 MODULE tst.cpp)
target_link_libraries(tst-3.9.7 PUBLIC pybind11::module)

[darshak7195]

I am following these steps and not able to compile it for two different versions of python. Here is my cmake file:

cmake_minimum_required(VERSION 3.21)
set(CMAKE_CXX_EXTENSIONS OFF)
include(FetchContent)
project(pybind11-example)
cmake_policy(SET CMP0074 NEW)

set(PYBIND11_NOPYTHON ON)

set(Python_ROOT /usr/bin/)
set(PYBIND11_PYTHON_VERSION 3.6 CACHE STRING "")
find_package(Python ${PYBIND11_PYTHON_VERSION} COMPONENTS Interpreter Development)
FetchContent_Declare(pybind11
    GIT_REPOSITORY https://github.com/pybind/pybind11.git
    GIT_TAG        v2.8.1
  )

FetchContent_MakeAvailable(pybind11)
Python_add_library(example example.cpp)
target_link_libraries(example PUBLIC pybind11::module)


set(Python_ROOT /opt/rh/rh-python38/root/usr/bin/)
set(PYBIND11_PYTHON_VERSION 3.8 CACHE STRING "" FORCE)
find_package(Python ${PYBIND11_PYTHON_VERSION} COMPONENTS Interpreter Development)

Python_add_library(example38 example.cpp)
target_link_libraries(example38 PUBLIC pybind11::module)

two libraries are generated, but one of them is corrupted. And I am getting the following error in cmake:

-- Found Python: /usr/bin/python3.6 (found suitable version "3.6.8", minimum required is "3.6") found components: Interpreter Development Development.Module Development.Embed
-- pybind11 v2.8.1
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Could NOT find Python: Found unsuitable version "3.6.8", but required is at least "3.8" (found /usr/bin/python3.6, found components: Interpreter Development Development.Module Development.Embed)

[henryiii]

set(Python_ROOT /opt/rh/rh-python38/root/usr/bin/) should not include "bin/". You don't need to include bin in /usr/bin either, I think. You should also set Development.Module instead, just in case the static libs are not present, you don't actually need static libs. Also pretty sure CMP0074 is covered by min version you are setting (minor cleanup). I'd also not set a cached variable - with FORCE, there's no benefit to it over a normal variable here.

cmake_minimum_required(VERSION 3.21)
project(pybind11-example LANGUAGES CXX)

set(CMAKE_CXX_EXTENSIONS OFF)
include(FetchContent)

set(PYBIND11_NOPYTHON ON)
FetchContent_Declare(pybind11
    GIT_REPOSITORY https://github.com/pybind/pybind11.git
    GIT_TAG        v2.8.1
)
FetchContent_MakeAvailable(pybind11)

set(Python_ROOT_DIR /usr)
find_package(Python 3.6 COMPONENTS Interpreter Development.Module)
Python_add_library(example example.cpp)
target_link_libraries(example PUBLIC pybind11::module)

set(Python_ROOT_DIR /opt/rh/rh-python38/root/usr)
find_package(Python 3.8 COMPONENTS Interpreter Development.Module)
Python_add_library(example38 example.cpp)
target_link_libraries(example38 PUBLIC pybind11::module)

[henryiii]

Also, it's Python_ROOT_DIR, that might have been most / all of the problem, though I think it's still not supposed to have "bin" at the end (since libs and such are also "in" the root).

https://cmake.org/cmake/help/latest/module/FindPython.html#hints

[darshak7195]

I copied this cmake and tested it, it still shows the same error:

-- Found Python: /usr/bin/python3.6 (found suitable version "3.6.8", minimum required is "3.6") found components: Interpreter Development.Module
-- Could NOT find Python: Found unsuitable version "3.6.8", but required is at least "3.8" (found /usr/bin/python3.6, found components: Interpreter Development.Module)

Also when I try to import example38, it throws the below error:

>>> import example38
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dynamic module does not define module export function (PyInit_example38)

[henryiii]

If you switch the order (3.8 first), does that find 3.8 or is it still stuck on 3.6?

[darshak7195]

If I switch the order, it finds 3.8 for both:

-- Found Python: /opt/rh/rh-python38/root/usr/bin/python3.8 (found suitable version "3.8.11", minimum required is "3.8") found components: Interpreter Development.Module
-- Found Python: /opt/rh/rh-python38/root/usr/bin/python3.8 (found suitable version "3.8.11", minimum required is "3.6") found components: Interpreter Development.Module

And if I write EXACT in find package it throws error:

set(Python_ROOT_DIR /usr)
find_package(Python 3.6 EXACT COMPONENTS Interpreter Development.Module)

-- Found Python: /opt/rh/rh-python38/root/usr/bin/python3.8 (found suitable version "3.8.11", minimum required is "3.8") found components: Interpreter Development.Module
-- Could NOT find Python: Found unsuitable version "3.8.11", but required is exact version "3.6" (found /opt/rh/rh-python38/root/usr/bin/python3.8, found components: Interpreter Development.Module)