CMake and Windows support #512
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With this patch, sysbench can be built (on Linux, macOS, or FreeBSD at least) using cmake. Packaging using cpack works, too.
Add just basic build and unit test, nothing fancy
This build actually runs some 30 seconds OLTP smoke tests against MySQL 8.0
__builtin_strncpy specified bound 4096 equals destination size [-Werror=stringop-truncation] strncpy wants output buffer to have place for terminating 0
sysbench/src/tests/memory/sb_memory.c:199:18: error: ‘buffer’ may be used uninitialized in this function [-Werror=maybe-uninitialized] 199 | buffers[i] = buffer;
…ssert. This fixes warning about unused typedef - SB_ATTRIBUTE_UNUSED does not help
The original URL is https://locklessinc.com/downloads/winpthreads.h mingw64 winpthreads is based on this library. Compared to other pthread implementations, this one does support pthread cancellation, which turns out to be important for sysbench.
Functions used in sysbench were ported : alarm() ,clock_gettime(), basename(), pread(), pwrite(), fdatasync() It also contains an open() portability wrapper that supports O_SYNC and O_DIRECT
Port some bits of ConcurrencyKit used by sysbench to MSVC compiler.
Use native compiler intrinsics for ck_pr_{inc/dec/store} family functions.
Copy ck_ring code from concurrency kit, as it is not possible to use
the original header.
Use instructions from Luajit documentation to build the library. Allow FindLuajit.cmake to find luajit if it is built by vcpkg, if system luajit library is requested. Also fix BuildLuajit.cmake to work with Ninja
…om source As Windows does not have system libraries for MySQL or MariaDB client, for the sake of simplicity, allow to build libmariadb as external project during sysbench build. That is, if WITH_LIBMARIADB is defined, libmariadb will be built from source https://github.com/mariadb-corporation/mariadb-connector-c, courtesy of CMake's ExternalProject_Add libmariadb is chosen because it is easy to build standalone (as opposed to libmysqlclient) and works well with named pipes, which libmysqlclient has problems with. WITH_LIBMARIADB defaults to ON on Windows, as to OFF elsewhere. It is built as shared library on Windows, as static elsewhere( to avoid linker warning about mix of C runtimes)
MSVC does not allow to use declspec(align) after the variable definition. So CK_CC_CACHELINE is moved before variable declaration.
- Do not use #ifdef within macro
- Use do{}while(0) instead of unportable GCC braces in xfree() macro
Use _aligned_malloc() for aligned allocation, and _aligned_free() for deallocation. Add sb_free_memaligned() instead of free() where appropriate.
Implement cancellable sleeps. "Cancellable" property is important here, otherwise sysbench is going to hang, when used with --report-interval
Internally, those are translated to FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH
Emulate search path logic on Windows from luajit code Search paths always relative to current executable, which is great for ZIP type distribution. Do not use Unixy path with "/usr/" etc on Windows.
The list of symbols is extracted from ffi.cdef sections in internal lua files into "module definition file" sysbench.def, which is used by the linker to export symbols. Alternative to that could be WINDOWS_EXPORT_ALL_SYMBOLS in recent cmake, but that does not work with LTO/PGO, unfortunately.
long is 4 bytes , also on 64bit Windows, so that does not work as intended
…efined. HAVE_MMAP was not ported so far, and sb_get_allocation_granularity() is only defined with HAVE_MMAP Also, sb_next_event() does not exist.
…te_event() MSVC compiler understands that the function has no side-effects, and optimizes the function away, completely. Also, clang-16 complains about set-but-unused variable. To fix both, assign calculated prime count in cpu_execute_event() to a thread local variable, at the end of function This way, cpu_execute_event() function has _some_ side-effect and will not be optimized away.
- Copy luajit and libmariadb DLLs next to sysbench executable, during build. This allows to test without packaging - Do not install Cram based test suite on Windows, it does not work - Install DLLs that sysbench depends on next to executable (luajit DLL, libmariadb, postgres client, and their dependencies such as openssl or zlib. Also install VC runtime DLLs, and ASAN runtime DLL, when built WITH_ASAN=1 - Default to ZIP for CPack.
LuaJit opens source files using C runtime functions, and in sysbench it does so simultaneously in different threads. To avoid "Too many open files" error, use _setmaxstdio() with maximum possible value for open handles, which is 8K
Without this call, Sleep(1) can take as much as 14-15ms
The build utilizes vcpkg package manager in order to build with Postgres driver. A ZIP sysbench package is built and stored as build artifact, so if desired, someone can use extract it, and produce official binary release. The tests are smoke OLTP tests that run against installed MariaDB and Postgres. As Cram unit test does not work on Windows, so this is the only test option
Combining flags in file-extra-flags in fileio should be allowed O_SYNC|O_DIRECT, or O_DSYNC|O_DIRECT is actually reasonable combination.
On CI, the error code of the message "Unknown MySQL server host 'non-existing' " is -3, not 0.
Remove repetive copy-and-paste workflows, create a reusable one. Now the build and test, and ASAN, and building with non-default compilers runs on all supported (by CI) OSes (Windows, Linux, and macOS, as much uniform as possible. "make test" is not very portable in sysbench It does not work on Windows, does not work with either MariaDB client or server, does not work with recent poostgres Therefore on all combinations or OS(Windows, Linux, macOS)/database(MariaDB, MySQL, Postgres) some smoke tests (oltp) are also run. "make test" will run, where it is known to have worked before (non-Windows, server and client library are both recent MySQL) ASAN builds run on Windows and macOS. Due to a bug in ASAN on Ubuntu's gcc, google/sanitizers#1010 , we do not run it on ubuntu Ninja CMake generator is run on all OSes, and "alternative" compilers (clang on Linux, mingw64 gcc on Windows) are also tested.
Also link MySQL-specific plugins statically.
Also disable Windows ASAN - something bad happened, it can't run on CI Reenable ASAN on Linux though
There are some problems with PG client library, so it does not build as it should
The new libmariadb defaults to enabling SSL, which introduces backward compatibility issues. Moreover, certificate verification may be enforced, together with SSL, potentially breaking connections with MySQL and older MariaDB versions. Thus, disable both certificate validation (apparently there is no option for it at the moment) and SSL.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This request implements CMake build, and Windows port on top of it.
CMake build should work on all supported OSes (in addition to Linux, Windows, macOS, which are tested on CI, I tried FreeBSD myself).
On Windows, most of native compilers will work out of the box - MSVC, clang-cl,mingw64 gcc. Performance-wise oltp should be as good as client libraries are. Named pipes for communication are supported, via
--mysql-host=., or--mysql-socket=named-pipe-nameThe request is split into many small-ish commits, to ease the reviewer's job.
I added CI workflow, to test builds on the supported platforms and database combinations
resolves #325