Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wheels: 0.15.0.post2 #1406

Merged
merged 4 commits into from
Mar 31, 2023
Merged

Wheels: 0.15.0.post2 #1406

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5ce4e14
Wheels: 0.15.0.post2
ax3l Mar 28, 2023
5bc62b9
[Hack] Win w/ only HDF5 w/ static Zlib
ax3l Mar 29, 2023
7523314
Win: Activate ADIOS2, too
ax3l Mar 30, 2023
2058077
Reactivate All Builds
ax3l Mar 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,12 +113,12 @@ jobs:
url: "https://github.com/ax3l/openPMD-api/commit/43c4bf4327616e9cf7e7992d91d9b8ccd4ee3a83.patch"
target: src/.patch/

# 0.15.0.post1 bump
# 0.15.0.post2 bump
- name: Download Patch 4/4
uses: suisei-cn/actions-download-file@v1
id: setupversion
with:
url: "https://gist.githubusercontent.com/ax3l/5e83edefe4b05cf6aa2a971649285fe0/raw/b510a363aac6b68651ec8081efbb40cb3bda1d4b/0001-Bump-setup.py.patch"
url: "https://gist.githubusercontent.com/ax3l/5e83edefe4b05cf6aa2a971649285fe0/raw/023cbd69e06715e5191ece741bb1de56560a9f96/0001-Bump-setup.py.patch"
target: src/.patch/

- name: Apply Patches
Expand All @@ -143,8 +143,8 @@ jobs:
CIBW_BEFORE_BUILD_WINDOWS: 'cmd /E:ON /V:ON /C .github\library_builders.bat'
# for the openPMD-api build, CMake shall search for
# static dependencies of HDF5 and ADIOS1 (see setup.py)
CIBW_ENVIRONMENT: HDF5_USE_STATIC_LIBRARIES='ON' ADIOS_USE_STATIC_LIBS='ON' openPMD_CMAKE_openPMD_USE_HDF5='ON' openPMD_CMAKE_openPMD_USE_ADIOS2='ON'
CIBW_ENVIRONMENT_WINDOWS: HDF5_USE_STATIC_LIBRARIES='ON' openPMD_CMAKE_openPMD_USE_HDF5='ON' openPMD_CMAKE_openPMD_USE_ADIOS2='ON' CMAKE_PREFIX_PATH='C:/Program Files (x86)/ADIOS2;C:/Program Files (x86)/blosc;C:/Program Files (x86)/HDF5;C:/Program Files (x86)/ZFP;C:/Program Files (x86)/zlib'
CIBW_ENVIRONMENT: HDF5_USE_STATIC_LIBRARIES='ON' ZLIB_USE_STATIC_LIBS='ON' ADIOS_USE_STATIC_LIBS='ON' openPMD_CMAKE_openPMD_USE_HDF5='ON' openPMD_CMAKE_openPMD_USE_ADIOS2='ON'
CIBW_ENVIRONMENT_WINDOWS: HDF5_USE_STATIC_LIBRARIES='ON' ZLIB_USE_STATIC_LIBS='ON' openPMD_CMAKE_openPMD_USE_HDF5='ON' openPMD_CMAKE_openPMD_USE_ADIOS2='ON' CMAKE_PREFIX_PATH='C:/Program Files (x86)/ADIOS2;C:/Program Files (x86)/blosc;C:/Program Files (x86)/HDF5;C:/Program Files (x86)/ZFP;C:/Program Files (x86)/zlib'
Comment on lines +146 to +147
Copy link
Member Author

@ax3l ax3l Mar 30, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ZLIB_USE_STATIC_LIBS='ON' env var here has probably no effect: not added to setup.py in #1410

We mostly need to pass this in library_builders.sh/bat when we build dependencies that directly pick up ZLIB.

# C++17 support in macOS 10.13+ (partial) and 10.14+ (std::visit) and 10.15+ (std::filesystem::path)
# https://cibuildwheel.readthedocs.io/en/stable/cpp_standards/#macos-and-deployment-target-versions
# arm64 Python interpreters are built with 11.0
Expand Down
2 changes: 1 addition & 1 deletion .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ before_script:
- curl -sOL https://github.com/ax3l/openPMD-api/commit/b622cc5ea770f866c1e373185a9e389c04bdb54c.patch
- curl -sOL https://github.com/ax3l/openPMD-api/commit/a1aac530dbc8261656cf6a07900374bc6ac6fcab.patch
- curl -sOL https://github.com/ax3l/openPMD-api/commit/43c4bf4327616e9cf7e7992d91d9b8ccd4ee3a83.patch
- curl -sOL https://gist.githubusercontent.com/ax3l/5e83edefe4b05cf6aa2a971649285fe0/raw/b510a363aac6b68651ec8081efbb40cb3bda1d4b/0001-Bump-setup.py.patch
- curl -sOL https://gist.githubusercontent.com/ax3l/5e83edefe4b05cf6aa2a971649285fe0/raw/023cbd69e06715e5191ece741bb1de56560a9f96/0001-Bump-setup.py.patch
- cd ..
- python3 -m patch .patch/b622cc5ea770f866c1e373185a9e389c04bdb54c.patch
- python3 -m patch .patch/a1aac530dbc8261656cf6a07900374bc6ac6fcab.patch
Expand Down
33 changes: 28 additions & 5 deletions library_builders.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,9 @@ exit /b 0
cmake --build build-adios2 --target install --config Release
if errorlevel 1 exit 1

rmdir /s /q build-adios2
if errorlevel 1 exit 1

break > adios2-stamp
if errorlevel 1 exit 1
exit /b 0
Expand All @@ -68,6 +71,7 @@ exit /b 0
-DBUILD_SHARED=OFF ^
-DBUILD_STATIC=ON ^
-DBUILD_TESTS=OFF ^
-DZLIB_USE_STATIC_LIBS=ON ^
-DDEACTIVATE_SNAPPY=ON
if errorlevel 1 exit 1

Expand All @@ -77,6 +81,9 @@ exit /b 0
cmake --build build-blosc --target install --config Release
if errorlevel 1 exit 1

rmdir /s /q build-blosc
if errorlevel 1 exit 1

break > blosc-stamp
if errorlevel 1 exit 1
exit /b 0
Expand All @@ -93,6 +100,7 @@ exit /b 0
-DCMAKE_VERBOSE_MAKEFILE=ON ^
-DBUILD_SHARED_LIBS=OFF ^
-DBUILD_TESTING=OFF ^
-DTEST_SHELL_SCRIPTS=OFF ^
-DHDF5_BUILD_CPP_LIB=OFF ^
-DHDF5_BUILD_EXAMPLES=OFF ^
-DHDF5_BUILD_FORTRAN=OFF ^
Expand All @@ -101,6 +109,7 @@ exit /b 0
-DHDF5_ENABLE_PARALLEL=OFF ^
-DHDF5_ENABLE_SZIP_SUPPORT=OFF ^
-DHDF5_ENABLE_Z_LIB_SUPPORT=ON ^
-DZLIB_USE_STATIC_LIBS=ON ^
-DCMAKE_INSTALL_PREFIX=%BUILD_PREFIX%/HDF5
if errorlevel 1 exit 1

Expand All @@ -110,6 +119,9 @@ exit /b 0
cmake --build build-hdf5 --target install --config Release
if errorlevel 1 exit 1

rmdir /s /q build-hdf5
if errorlevel 1 exit 1

break > hdf5-stamp
if errorlevel 1 exit 1
exit /b 0
Expand Down Expand Up @@ -137,19 +149,22 @@ exit /b 0
cmake --build build-zfp --target install --config Release
if errorlevel 1 exit 1

rmdir /s /q build-zfp
if errorlevel 1 exit 1

break > zfp-stamp
if errorlevel 1 exit 1
exit /b 0

:build_zlib
if exist zlib-stamp exit /b 0

curl -sLo zlib-1.2.12.zip ^
https://github.com/madler/zlib/archive/v1.2.12.zip
powershell Expand-Archive zlib-1.2.12.zip -DestinationPath dep-zlib
curl -sLo zlib-1.2.13.zip ^
https://github.com/madler/zlib/archive/v1.2.13.zip
powershell Expand-Archive zlib-1.2.13.zip -DestinationPath dep-zlib

cmake -S dep-zlib/zlib-1.2.12 -B build-zlib ^
-DBUILD_SHARED_LIBS=OFF ^
cmake -S dep-zlib/zlib-1.2.13 -B build-zlib ^
-DBUILD_SHARED_LIBS=ON ^
-DCMAKE_BUILD_TYPE=Release
if errorlevel 1 exit 1
:: Manually-specified variables were not used by the project:
Expand All @@ -161,6 +176,14 @@ exit /b 0
cmake --build build-zlib --target install --config Release
if errorlevel 1 exit 1

set "zlib_dll=%BUILD_PREFIX:~1,-1%/zlib/bin/zlib1.dll"
set "zlib_dll=%zlib_dll:/=\%"
del "%zlib_dll%"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Windows builds still pick up a zlib.dll somewhere, probably from the system...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HDF5:

2023-03-28T19:36:25.6925189Z -- Could NOT find ZLIB (missing: ZLIB_DIR)
2023-03-28T19:36:25.8188114Z -- Found ZLIB: C:/Program Files (x86)/zlib/lib/zlib.lib (found version "1.2.12")

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compiled without any backends, the .dll dependencies are reduced to only allowed ones:

$ objdump -p openpmd_api_cxx.cp311-win_amd64.pyd | grep dll
	DLL Name: python311.dll
	DLL Name: KERNEL32.dll
	DLL Name: MSVCP140.dll
	DLL Name: VCRUNTIME140_1.dll
	DLL Name: VCRUNTIME140.dll
	DLL Name: api-ms-win-crt-string-l1-1-0.dll
	DLL Name: api-ms-win-crt-heap-l1-1-0.dll
	DLL Name: api-ms-win-crt-runtime-l1-1-0.dll
	211e44	   63  _seh_filter_dll
	DLL Name: api-ms-win-crt-convert-l1-1-0.dll
	DLL Name: api-ms-win-crt-math-l1-1-0.dll
	DLL Name: api-ms-win-crt-stdio-l1-1-0.dll
	DLL Name: api-ms-win-crt-locale-l1-1-0.dll
	DLL Name: api-ms-win-crt-filesystem-l1-1-0.dll
	DLL Name: api-ms-win-crt-environment-l1-1-0.dll
	DLL Name: api-ms-win-crt-time-l1-1-0.dll

Copy link
Member Author

@ax3l ax3l Mar 29, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Building only zlib & HDF5 already pulls in a zlib.dll again, even with zlib.dll removed in the zlib install path.

Unrelated: also found a PREFER_EXTERNAL_ZLIB option in blosc to avoid building an internal one.

Copy link
Member Author

@ax3l ax3l Mar 29, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • setting the HDF5 hints to our zlib explicitly does not help.
  • building HDF5 w/o zlib support also pulls in zlib.dll :-o (zlib built)
  • building HDF5 w/o zlib support (zlib not built) - this one does not pull a dep in!
  • test CMake 3.24+ option: ZLIB_USE_STATIC_LIBS - this one does not pull a dep in!
    • also, activates BUILD_SHARED_LIBS=ON for zlib, because it cannot be turned off conditionally with current zlib's CMake target logic and zlib1.dll is the correct target to build (and then remove)
  • also set zlib options for c-blosc and activate ADIOS2 again

if errorlevel 1 exit 1

rmdir /s /q build-zlib
if errorlevel 1 exit 1

break > zlib-stamp
if errorlevel 1 exit 1
exit /b 0
Expand Down
19 changes: 14 additions & 5 deletions library_builders.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,8 +118,8 @@ function build_adios2 {
ADIOS2_USE_SST=OFF
fi

mkdir build-ADIOS2
cd build-ADIOS2
mkdir build-adios2
cd build-adios2
PY_BIN=$(which python3)
CMAKE_BIN="$(${PY_BIN} -m pip show cmake 2>/dev/null | grep Location | cut -d' ' -f2)/cmake/data/bin/"
if [ "$(uname -s)" = "Linux" ]
Expand Down Expand Up @@ -153,6 +153,8 @@ function build_adios2 {
make install
cd -

rm -rf build-adios2

touch adios2-stamp
}

Expand All @@ -178,8 +180,8 @@ function build_blosc {
DEACTIVATE_SSE2=ON
fi

mkdir build-c-blosc
cd build-c-blosc
mkdir build-blosc
cd build-blosc
PY_BIN=$(which python3)
CMAKE_BIN="$(${PY_BIN} -m pip show cmake 2>/dev/null | grep Location | cut -d' ' -f2)/cmake/data/bin/"
PATH=${CMAKE_BIN}:${PATH} cmake \
Expand All @@ -190,11 +192,14 @@ function build_blosc {
-DBUILD_BENCHMARKS=OFF \
-DCMAKE_VERBOSE_MAKEFILE=ON \
-DCMAKE_INSTALL_PREFIX=${BUILD_PREFIX} \
-DZLIB_USE_STATIC_LIBS=ON \
../c-blosc-*
make -j${CPU_COUNT}
make install
cd -

rm -rf build-blosc

touch blosc-stamp
}

Expand All @@ -221,13 +226,15 @@ function build_zfp {
make install
cd -

rm -rf build-zfp

touch zfp-stamp
}

function build_zlib {
if [ -e zlib-stamp ]; then return; fi

ZLIB_VERSION="1.2.12"
ZLIB_VERSION="1.2.13"

curl -sLO https://zlib.net/fossils/zlib-$ZLIB_VERSION.tar.gz
file zlib*.tar.gz
Expand All @@ -247,6 +254,8 @@ function build_zlib {
PATH=${CMAKE_BIN}:${PATH} cmake --build build-zlib --target install
rm -rf ${BUILD_PREFIX}/lib/libz.*dylib ${BUILD_PREFIX}/lib/libz.*so

rm -rf build-zlib

touch zlib-stamp
}

Expand Down