BIG CHANGE: pybind11 -> nanobind

Please report problems when you find them.

notes and changes
-----------------

Protocol Buffer is currently not supported by nanobind.
Adding GridBase.__array__() for NumPy interoperability
in many cases works as seemless replacement of the Protocol Buffer.

Implicit list->ndarray conversion is also not supported, but is welcomed:
wjakob/nanobind#327

None is not accepted by default, an annotation must be added.
I may have missed it in some functions - let me know.
In case of const char* arg, None is not supported at all; I proposed it
upstream: wjakob/nanobind#683

For default nullptr, using nb::arg()=nb::none() instead of =nullptr.
Both work, but the latter seems to be more canonical in nanobind.

In nanobind 2.x using make_iterator and bind_vector+__getitem__ returns
copies, not references, by default. We usually prefer to avoid copying.
Added a helper function usual_iterator() that uses old rv_policy.

gemmi.ValueSigmaAsuData.value_array: previously had so-called
structured data type, now it has shape (N,2) with dtype=float32.

Added pickling support for Structure, Model, Chain, Residue, Atom, UnitCell.

Notes about pickling of SpaceGroup:
wjakob/nanobind#670
In a follow up, picking of SpaceGroup was optimized, replacing xhm()
with an index. The gain was modest (pickling and unpickling a list of all
SpaceGroups from the built-in table was reduced from ~1ms, to ~0.5ms).
It might not be safe - multiple copies of the table may exist.

Notes about constructors for PdbWriteOptions and MmcifOutputGroups:
see wjakob/nanobind#664

doctests: many changes, mostly because __repr__ of enums
and of bind_vector-ed classes in nanobind is different than in pybind11

added alignas(8) to HklValue: in nanobind array bindings,
stride is given as a number of elements, not bytes.
This was a problem for ComplexAsuData.value_array:
sizeof(HklValue) was 20 (3*4+2*4) which is not a multiple of 8.

Functions IT92Coef.calculate_sf() and IT92Coef.calculate_density_iso()
used to be vectorized (could take numpy array as arg), now they aren't,
let me know if you need a vectorized version.

Not yet done: *.pyi files with type annotations

Planned: unified logging of warnings/errors from various gemmi functions

.appveyor.yml

@@ -13,18 +13,11 @@ environment:
     - arch: x64
       CMAKE_GENERATOR: Visual Studio 16 2019
       CMAKE_CONFIG: RelWithDebInfo
-      CMAKE_ARGS: -DCMAKE_CXX_STANDARD=11 -DPython_EXECUTABLE="C:\\Python310-x64\\python.exe" -DBUILD_SHARED_LIBS=OFF -DVCPKG_TARGET_TRIPLET=x64-windows-static -DCMAKE_TOOLCHAIN_FILE=C:/Tools/vcpkg/scripts/buildsystems/vcpkg.cmake
+      CMAKE_ARGS: -DCMAKE_CXX_STANDARD=17 -DPython_EXECUTABLE="C:\\Python310-x64\\python.exe" -DBUILD_SHARED_LIBS=OFF -DVCPKG_TARGET_TRIPLET=x64-windows-static -DCMAKE_TOOLCHAIN_FILE=C:/Tools/vcpkg/scripts/buildsystems/vcpkg.cmake
       APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2019
       VCPKG_BUILD_TYPE: "release"
       PY_PYTHON: 3.10
 
-    - arch: x86
-      CMAKE_GENERATOR: Visual Studio 15 2017
-      CMAKE_CONFIG: MinSizeRel
-      CMAKE_ARGS: -DPython_EXECUTABLE="C:\\Python38\\python.exe"
-      APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2017
-      PY_PYTHON: 3.8-32
-
     - arch: x64
       CMAKE_GENERATOR: MSYS Makefiles
       COMPILER: MinGW64-gcc-8.1.0
@@ -43,7 +36,7 @@ for:
   build_script:
   - path
   - cmake --version
-  - git clone --depth=1 -b stable https://github.com/pybind/pybind11.git
+  - git clone --depth=1 --recursive https://github.com/wjakob/nanobind.git
   - cmake -G "%CMAKE_GENERATOR%" -DUSE_PYTHON=1 %CMAKE_ARGS% .
   - cmake --build . --config %CMAKE_CONFIG%
   - cmake --build . --config %CMAKE_CONFIG% --target check

.github/workflows/ci.yml

@@ -22,10 +22,13 @@ jobs:
     - name: Python Tests
       run: |
         source pyenv/bin/activate
-        python3 -m unittest discover -v -s tests/
         python3 -m pip install sphinx
-        cd docs
-        sphinx-build -M doctest . _build -n -E
+        # run tests twice: without and with numpy
+        python3 -m unittest discover -s tests/
+        (cd docs && sphinx-build -M doctest . _build -n -E)
+        python3 -m pip install numpy
+        python3 -m unittest discover -v -s tests/
+        (cd docs && sphinx-build -M doctest . _build -n -E)
     - uses: actions/upload-artifact@v4
       with:
         name: gemmi-macos14
@@ -96,14 +99,15 @@ jobs:
     - name: apt-get
       run: |
         sudo apt-get update
-        sudo apt-get install python3-full g++ gfortran python3-numpy
+        sudo apt-get install python3-full g++ gfortran
     - name: build and test
       run: |
         g++ --version
         python3 -m venv pyenv
         source pyenv/bin/activate
         export SKBUILD_CMAKE_ARGS='-DFETCH_ZLIB_NG=ON;-DCMAKE_BUILD_TYPE=None'
         python3 -m pip install -v .
+        python3 -m pip install numpy
         python3 -m unittest discover -v -s tests/
     - name: run doctest
       run: |
@@ -122,13 +126,9 @@ jobs:
       run: |
         sudo apt-get update
         sudo apt-get install libz-dev python3-pip g++ gfortran python3-numpy
-    - name: install pybind11
+    - name: install nanobind
       run: |
-        git clone --depth=1 https://github.com/pybind/pybind11.git
-        #cd pybind11
-        #cmake . -Wno-dev -DPYTHON_EXECUTABLE=/usr/bin/python3 -DPYBIND11_TEST=OFF
-        #make
-        #sudo make install
+        git clone --recursive --depth=1 https://github.com/wjakob/nanobind.git
     - name: build and test
       run: |
         g++ --version
@@ -158,9 +158,9 @@ jobs:
       run: |
         sudo apt-get update
         sudo apt-get install libz-dev python3-pip g++ gfortran python3-numpy valgrind
-    - name: install pybind11
+    - name: install nanobind
       run: |
-        python3 -m pip install pybind11
+        python3 -m pip install nanobind
     - name: build and test
       run: |
         g++ --version
@@ -181,21 +181,21 @@ jobs:
     - name: run tests under valgrind
       run: PYTHONMALLOC=malloc valgrind python3 -m unittest discover -v -s tests/
 
-  ubuntu2004_clang6:
-    name: "Ubuntu 20.04 with Clang 6.0"
+  ubuntu2004_clang9:
+    name: "Ubuntu 20.04 with Clang 9"
     runs-on: ubuntu-20.04
     if: "!contains(github.event.head_commit.message, '[skip ci]')"
     env:
-      CC: clang-6.0
-      CXX: clang++-6.0
+      CC: clang-9
+      CXX: clang++-9
       SKBUILD_CMAKE_ARGS: "-DCMAKE_CXX_STANDARD=11;-DEXTRA_WARNINGS=ON;-DSTANDALONE_PYTHON_MODULE=OFF"
       SKBUILD_CMAKE_TARGETS: "all;check"
     steps:
     - uses: actions/checkout@v4
-    - run: sudo apt-get install clang-6.0 libz-dev python3-pip python3-numpy
-    - name: install pybind11
+    - run: sudo apt-get install clang-9 libz-dev python3-pip python3-numpy
+    - name: install nanobind
       run: |
-        sudo /usr/bin/python3 -m pip install "pybind11[global]" build
+        sudo /usr/bin/python3 -m pip install nanobind build
     - name: build and test
       run: |
         $CXX --version
@@ -224,11 +224,11 @@ jobs:
         python3.8 --version
         cmake --version
         type python3.8
-    - name: install pybind11
+    - name: install nanobind
       run: |
-        git clone --branch stable --depth=1 https://github.com/pybind/pybind11.git
-        cd pybind11
-        cmake . -Wno-dev -DPYTHON_EXECUTABLE=/usr/bin/python3.8 -DPYBIND11_TEST=OFF
+        git clone --recursive --depth=1 https://github.com/wjakob/nanobind.git
+        cd nanobind
+        cmake . -Wno-dev -DPYTHON_EXECUTABLE=/usr/bin/python3.8 -DNB_TEST=OFF
         make
         make install
     - name: build and test

.gitignore

@@ -22,8 +22,8 @@ tags
 /CTestTestfile.cmake
 /Makefile
 
-# local pybind11 is used for testing new versions, don't add to the repo
-third_party/pybind11
+# local nanobind can be used for testing new versions, don't add to the repo
+/nanobind
 
 # ignore random data files in the top and source directory
 /*.cif

CMakeLists.txt

@@ -67,9 +67,8 @@ endif()
 set(CMAKE_POLICY_DEFAULT_CMP0069 NEW)
 
 # Unless it's explicitly specifed, we want LTO for non-debug builds only.
-# pybind11Common.cmake has its own logic to set LTO options.
-# Having different options in pybind11_add_module than for gemmi_cpp
-# may cause problems -- we may need to handle it at some point.
+# TODO: check that it's compatible with nanobind_add_module, which has
+# its own logic of setting LTO options.
 if (NOT DEFINED CMAKE_INTERPROCEDURAL_OPTIMIZATION)
   include(CheckIPOSupported)
   check_ipo_supported(RESULT is_ipo_supported OUTPUT ipo_error)
@@ -108,10 +107,9 @@ if (DEFINED ENV{CXXFLAGS})
 endif()
 
 # Set default build mode (based on CMake FAQ)
-if (NOT CMAKE_BUILD_TYPE AND NOT USING_ENV_CXXFLAGS)
-  set(CMAKE_BUILD_TYPE Release CACHE STRING
-      "Choose the type of build, options are: None Debug Release RelWithDebInfo MinSizeRel."
-      FORCE)
+if (NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES AND NOT USING_ENV_CXXFLAGS)
+  set(CMAKE_BUILD_TYPE Release CACHE STRING "Choose the type of build." FORCE)
+  set_property(CACHE CMAKE_BUILD_TYPE PROPERTY STRINGS "Debug" "Release" "MinSizeRel" "RelWithDebInfo")
 endif()
 message(STATUS "Build type: ${CMAKE_BUILD_TYPE}")
 
@@ -458,29 +456,28 @@ if (USE_PYTHON)
   else()
     find_package(Python ${PYTHON_VERSION} REQUIRED COMPONENTS Interpreter Development.Module)
   endif()
-  # pybind11 macros use PYTHON_EXECUTABLE
-  if (DEFINED Python_EXECUTABLE AND NOT DEFINED PYTHON_EXECUTABLE)
-    set(PYTHON_EXECUTABLE "${Python_EXECUTABLE}")
-  endif()
-  if (EXISTS "${CMAKE_HOME_DIRECTORY}/pybind11")
-    message(STATUS "Using ${CMAKE_HOME_DIRECTORY}/pybind11 (internal copy).")
-    add_subdirectory(pybind11)
+  if (EXISTS "${CMAKE_HOME_DIRECTORY}/nanobind")
+    message(STATUS "Using ${CMAKE_HOME_DIRECTORY}/nanobind (internal copy).")
+    add_subdirectory(nanobind)
   else()
-    # use pybind11-config (if available) to determine pybind11_DIR
-    execute_process(COMMAND pybind11-config --cmakedir OUTPUT_VARIABLE pybind11_DIR)
-    string(STRIP "${pybind11_DIR}" pybind11_DIR)
-    find_package(pybind11 2.6 CONFIG REQUIRED)
-    message(STATUS "Found pybind11 ${pybind11_VERSION}: ${pybind11_INCLUDE_DIRS}")
+    # Detect the installed nanobind package and import it into CMake
+    execute_process(
+      COMMAND "${Python_EXECUTABLE}" -m nanobind --cmake_dir
+      OUTPUT_STRIP_TRAILING_WHITESPACE OUTPUT_VARIABLE NB_DIR)
+    list(APPEND CMAKE_PREFIX_PATH "${NB_DIR}")
+    find_package(nanobind 2.1.0 CONFIG REQUIRED)
+    message(STATUS "Found nanobind ${nanobind_VERSION}: ${NB_DIR}")
   endif()
-  pybind11_add_module(gemmi_py
-          python/mol.cpp python/gemmi.cpp python/align.cpp
+  nanobind_add_module(gemmi_py NOMINSIZE
+          python/gemmi.cpp python/align.cpp
           python/ccp4.cpp python/chemcomp.cpp python/cif.cpp python/custom.cpp
           python/elem.cpp python/grid.cpp python/hkl.cpp
-          python/meta.cpp python/monlib.cpp
+          python/meta.cpp python/mol.cpp python/monlib.cpp
           python/mtz.cpp python/read.cpp python/recgrid.cpp
           python/scaling.cpp python/search.cpp
           python/sf.cpp python/sym.cpp python/topo.cpp
           python/unitcell.cpp python/write.cpp)
+
   if (STANDALONE_PYTHON_MODULE)
     target_sources(gemmi_py PRIVATE $<TARGET_OBJECTS:gemmi_cpp>)
     get_target_property(_gemmi_cpp_libs gemmi_cpp LINK_LIBRARIES)
@@ -489,9 +486,25 @@ if (USE_PYTHON)
     target_link_libraries(gemmi_py PRIVATE gemmi_cpp)
   endif()
   set_property(TARGET gemmi_py PROPERTY OUTPUT_NAME gemmi)
+
+  # nanobind gives warnings with -Wpedantic and -Wshadow
   if(CMAKE_CXX_FLAGS MATCHES "-Wshadow")
+    if (TARGET nanobind-static)
+      target_compile_options(nanobind-static PRIVATE "-Wno-shadow")
+    endif()
     target_compile_options(gemmi_py PRIVATE "-Wno-shadow")
   endif()
+  if (CMAKE_CXX_FLAGS MATCHES "-Wpedantic")
+    if (TARGET nanobind-static)
+      target_compile_options(nanobind-static PRIVATE "-Wno-pedantic")
+    endif()
+    target_compile_options(gemmi_py PRIVATE "-Wno-pedantic")
+  endif()
+  if (CMAKE_CXX_FLAGS MATCHES "-Wredundant-decls")
+    if (TARGET nanobind-static)
+      target_compile_options(nanobind-static PRIVATE "-Wno-redundant-decls")
+    endif()
+  endif()
 
   if (USE_NLOPT)
     message(STATUS "Using NLopt.")
@@ -503,22 +516,14 @@ if (USE_PYTHON)
   if (GENERATE_STUBS AND CMAKE_CROSSCOMPILING)
     message(WARNING "Stubs cannot be generated when cross-compiling - skipping.")
   elseif (GENERATE_STUBS)
-    set(sep ":")
-    if (WIN32)
-      set(sep ";")
-    endif()
-    add_custom_command(TARGET gemmi_py POST_BUILD
-    COMMAND ${CMAKE_COMMAND} -E env
-         "PYTHONPATH=$<TARGET_FILE_DIR:gemmi_py>${sep}$ENV{PYTHONPATH}"
-       ${Python_EXECUTABLE} -m pybind11_stubgen gemmi
-         --enum-class-locations=Ignore:gemmi.ContactSearch
-         --enum-class-locations=.+:gemmi
-         --numpy-array-remove-parameters
-         --output-dir stubs
-     COMMAND ${CMAKE_COMMAND} -E touch stubs/gemmi/py.typed
-     COMMENT "Generate type stubs with pybind11-stubgen"
-     BYPRODUCTS stubs/gemmi/__init__.pyi stubs/gemmi/cif.pyi stubs/gemmi/py.typed
-     VERBATIM)
+    #nanobind_add_stub(
+    #  gemmi_py_stub
+    #  MODULE gemmi_py
+    #  OUTPUT stubs/gemmi/__init__.pyi
+    #  PYTHON_PATH $<TARGET_FILE_DIR:gemmi_py>
+    #  DEPENDS gemmi_py
+    #  MARKER_FILE stubs/py.typed
+    #)
   endif()
 else()
   message(STATUS "Skipping Python module. Add -D USE_PYTHON=1 to build it.")
@@ -579,8 +584,8 @@ if (USE_PYTHON)
   install(DIRECTORY examples DESTINATION "${Python_SITEARCH}/gemmi" COMPONENT py
           FILES_MATCHING PATTERN "*.py"
           PATTERN "[._]*" EXCLUDE)
-  if (GENERATE_STUBS AND NOT CMAKE_CROSSCOMPILING)
-    install(DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/stubs/gemmi"
-            DESTINATION "${Python_SITEARCH}" COMPONENT py)
-  endif()
+  #if (GENERATE_STUBS AND NOT CMAKE_CROSSCOMPILING)
+  #  install(DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/stubs/gemmi"
+  #          DESTINATION "${Python_SITEARCH}" COMPONENT py)
+  #endif()
 endif()

docs/cif.rst

@@ -225,9 +225,6 @@ element found in mmCIF:
 
 More complex examples are shown in the :ref:`cif_examples` section.
 
-Internally, Python bindings use
-`pybind11 <https://github.com/pybind/pybind11>`_.
-
 
 DOM and SAX
 ===========

docs/conf.py

@@ -54,7 +54,6 @@
 doctest_global_setup = '''
 import os
 import sys
-assert sys.version_info[0] > 2, "Tests in docs are for Python 3 only"
 disabled_features = []
 try:
     import numpy

docs/grid.rst

@@ -504,7 +504,7 @@ The parameters of SolventMasker can be inspected and changed:
 .. doctest::
 
   >>> masker.atomic_radii_set
-  <AtomicRadiiSet.Cctbx: 1>
+  AtomicRadiiSet.Cctbx
   >>> masker.rprobe
   1.1
   >>> masker.rshrink

docs/hkl.rst

@@ -114,7 +114,7 @@ Datasets are stored in the variable `datasets`::
 .. doctest::
 
   >>> mtz.datasets
-  MtzDatasets[<gemmi.Mtz.Dataset 0 HKL_base/HKL_base/HKL_base>, <gemmi.Mtz.Dataset 1 5e5z/5e5z/1>]
+  gemmi.MtzDatasets([<gemmi.Mtz.Dataset 0 HKL_base/HKL_base/HKL_base>, <gemmi.Mtz.Dataset 1 5e5z/5e5z/1>])
 
 In the MTZ file, each dataset is identified internally by an integer
 "dataset ID". To get dataset with the specified ID use function::
@@ -820,7 +820,7 @@ Blocks are moved from the Document to the new list:
   >>> doc
   <gemmi.cif.Document with 0 blocks ()>
   >>> rblocks
-  ReflnBlocks[<gemmi.ReflnBlock r5wkdsf with 17 x 406 loop>]
+  gemmi.ReflnBlocks([<gemmi.ReflnBlock r5wkdsf with 17 x 406 loop>])
 
 When ReflnBlock is created some of the mmCIF tags are interpreted
 to initialize the following properties:
@@ -1209,7 +1209,7 @@ The unit cell and the upper bin boundaries (in Å\ :sup:`--2`) are stored intern
   >>> binner.cell
   <gemmi.UnitCell(50.347, 4.777, 14.746, 90, 101.73, 90)>
   >>> binner.limits  # doctest: +ELLIPSIS
-  [0.12224713046942721, 0.19395414269012246, 0.254107666379415..., inf]
+  [0.122247130469427..., 0.193954142690122..., 0.254107666379415..., inf]
   >>> binner.size  # number of bins
   4
 

docs/install.rst

@@ -90,8 +90,10 @@ To install the gemmi module do::
 We have binary wheels for several Python versions (for all supported CPython
 versions and one PyPy version), so the command usually downloads binaries.
 If a matching wheel is not available,
-the module is compiled from source -- it takes several minutes
-and requires a C++ compiler.
+the module is compiled from source -- it takes a few minutes
+and requires a C++ compiler that supports C++17.
+
+Gemmi 0.7+ supports only Python 3.8+.
 
 Other binaries
 ~~~~~~~~~~~~~~
@@ -136,10 +138,11 @@ for details.
 If gemmi is already installed, uninstall the old version first
 (`pip uninstall`) or add option `--upgrade`.
 
-Alternatively, you can build a cloned project directly with CMake::
+Alternatively, you can manually install nanobind and cmake (using pip)
+and build a cloned project directly with CMake::
 
     cmake -D USE_PYTHON=1 .
-    make -j4 py
+    make -j4 gemmi_py
 
 Fortran and C bindings
 ----------------------
@@ -246,6 +249,8 @@ Code derived from the following projects is used in the library:
 Projects included under `third_party/` that are not used in the library
 itself, but are used in command-line utilities, python bindings or tests:
 
+* `zpp serializer <https://github.com/eyalz800/serializer>`_ --
+  serialization framework. License: MIT.
 * `The Lean Mean C++ Option Parser <http://optionparser.sourceforge.net/>`_ --
   command-line option parser. License: MIT.
 * `doctest <https://github.com/onqtam/doctest>`_ -- testing framework.
@@ -258,7 +263,7 @@ itself, but are used in command-line utilities, python bindings or tests:
 
 Not distributed with Gemmi:
 
-* `pybind11 <https://github.com/pybind/pybind11>`_ -- used for creating
+* `nanobind <https://github.com/wjakob/nanobind>`_ -- used for creating
   Python bindings. License: 3-clause BSD.
 * `zlib-ng <https://github.com/zlib-ng/zlib-ng>`_ -- optional, can be used
   instead of zlib for faster reading of gzipped files.