Build TileDB-VCF-Py on Windows without conda

[jdblischak]

Goal: Run Windows CI job without using conda

Motivation: We haven't been able to build the TileDB-VCF conda binary for months due to problems linking to fmt/spdlog from conda-forge. Ideally these sort of conda-specific problems should be handled in the feedstock repo, not here. It wasn't a pressing concern until recently when @gspowley started work to migrate to numpy 2, which included updating pyarrow (#762 (comment)). Now the Windows CI build is failing due to similar fmt/spdlog linking errors.

Status: I can build libtiledbvcf and tiledbvcf-py on Windows without conda. However, I can't import tiledbvcf because it can't find the libtiledbvcf DLL (strangely, since it is installed in the Python package).

More details:

• Ideally, since we are using scikit-build-core, the build would be as straightforward as python -m pip install -v apis\python[test,dev]. However, this failed because the following wouldn't execute on Windows:

TileDB-VCF/apis/python/CMakeLists.txt

Lines 55 to 61 in b9fd438

	execute_process(
	COMMAND bash -c "\
	${CMAKE_COMMAND} .. && \
	${CMAKE_COMMAND} --build . && \
	${CMAKE_COMMAND} --build . --target install-libtiledbvcf"
	WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}/dist_links/libtiledbvcf/build"
	)

• The Python CMake expects TileDB to be in apis/python/dist_links/libtiledbvcf/build/externals/install. This caused me headaches when I tried building libtiledbvcf from the root of the directory. I tried a few different ways to build from the symlinked subdirectory apis/python/dist_links/libtiledbvcf, but symlinks are different on Windows, so I abandoned this

TileDB-VCF/apis/python/CMakeLists.txt

Lines 104 to 107 in b9fd438

	# Search for TileDB in the EP install directory
	list(PREPEND CMAKE_PREFIX_PATH
	"${CMAKE_CURRENT_SOURCE_DIR}/dist_links/libtiledbvcf/build/externals/install"
	)

• I finally got everything to build by setting TileDB_DIR=.\libtiledbvcf\build\externals\install\lib\cmake\TileDB\, so that the Python CMake routine could fine TileDB

  *** Installing project into wheel...
  -- Installing: C:\Users\RUNNER~1\AppData\Local\Temp\tmpc7c23i0p\wheel\platlib/tiledbvcf/libtiledbvcf.cp311-win_amd64.pyd
  -- Installing: C:\Users\RUNNER~1\AppData\Local\Temp\tmpc7c23i0p\wheel\platlib/tiledbvcf/tiledb.lib
  -- Installing: C:\Users\RUNNER~1\AppData\Local\Temp\tmpc7c23i0p\wheel\platlib/tiledbvcf/tiledbvcf.lib
  -- Installing: C:\Users\RUNNER~1\AppData\Local\Temp\tmpc7c23i0p\wheel\platlib/tiledbvcf/hts-3.lib
  -- Installing: C:/Users/RUNNER~1/AppData/Local/Temp/tmpc7c23i0p/wheel/scripts/tiledbvcf.dll
  -- Installing: C:/Users/RUNNER~1/AppData/Local/Temp/tmpc7c23i0p/wheel/scripts/tiledbvcf.exe
  -- Installing: C:/Users/RUNNER~1/AppData/Local/Temp/tmpc7c23i0p/wheel/scripts/tiledbvcf4test.dll

• However, I can't run the tests because I can't even import tiledbvcf-py. Unclear what I can do about this. Maybe the problem is that only tiledb.lib is copied to the Python package and not tiledb.dll

python -c "import tiledbvcf; print(tiledbvcf.version)"
## Traceback (most recent call last):
##   File "<string>", line 1, in <module>
##   File "C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\tiledbvcf\__init__.py", line 9, in <module>
##     import tiledbvcf.libtiledbvcf
## ImportError: DLL load failed while importing libtiledbvcf: The specified module could not be found.

TileDB-VCF/apis/python/src/tiledbvcf/__init__.py

Line 9 in b9fd438

import tiledbvcf.libtiledbvcf

• Note that I fixed the htslib download by updating the sha, which I learned from the PR Draft: Add initial build wheels CI/CD pipeline #763 from @dudoslav

[dudoslav]

@jdblischak I also put some effort in fixing the wheel production here: #763 (as you mentioned)

In my case I realized that some parts of VCF build logic needs to be rewritten. I think that we are both working on the same thing.

[jdblischak]

I tried a few things on a separate branch, but still failing.

• Tried an editable install with -e since sometimes this can help with resolving finding shared libraries. This didn't help.
• Using the same editable install, I skipped the simple import test and ran pytest, with the hope that this scenario might be able to find the shared libraries. Unfortunately not. Still failed with the same import error

[jdblischak]

Reminder of our options for fixing the Windows CI builds:

• Fix the libfmt linking issues when building against more recent versions of libtiledb that link to conda-installed versions of fmt/spdlog (Build with NumPy 2 #762 (comment)). This would allow us to continue to use the conda-based build setup we are currently using
• Fix the non-conda Windows build that I have attempted in this PR. I can build everything, but I can't import tiledbvcf-py (and thus also can't run its test suite). Potentially some of the updates to the Windows build in Draft: Add initial build wheels CI/CD pipeline #763 might be able to help with this