-
Notifications
You must be signed in to change notification settings - Fork 6.4k
Building on Windows
This is a simple explanation of how to build RocksDB by either:
- Method 1 - Windows 10, and Visual Studio 2019.
- Method 2 - Windows 11, and Visual Studio 2022.
- Method 3 - vcpkg, Windows 10, and Visual Studio 2017/2019.
- Microsoft Visual Studio 2019 (Community) with "Desktop development with C++" installed
- CMake - I used version 3.20 installed from the 64bit MSI installer
- Git - I used the Windows Git Bash.
- wget
- Java 8 - I used BellSoft Liberica JDK 8 https://bell-sw.com/pages/downloads/
Create a directory somewhere on your machine that will be used as a container for both the RocksDB source code and that of its 3rd-party dependencies. On my machine I used C:\Users\aretter\code
, from hereon in I will just refer to it as %CODE_HOME%
; which can be set as an environment variable, i.e. SET CODE_HOME=C:\Users\aretter\code
.
All of the following is executed from the "Developer Command Prompt for VS2019":
cd %CODE_HOME%
wget https://github.com/gflags/gflags/archive/refs/tags/v2.2.2.zip
unzip v2.2.2.zip
cd gflags-2.2.2
mkdir target
cd target
cmake -G "Visual Studio 16 2019" -A x64 ..
msbuild gflags.sln /p:Configuration=Debug /p:Platform=x64
msbuild gflags.sln /p:Configuration=Release /p:Platform=x64
The resultant static library can be found in %CODE_HOME%\gflags-2.2.2\target\lib\Debug\gflags_static.lib
or %CODE_HOME%\gflags-2.2.2\target\lib\Release\gflags_static.lib
.
cd %CODE_HOME%
wget https://github.com/google/snappy/archive/refs/tags/1.1.9.zip
unzip 1.1.9.zip
cd snappy-1.1.9
mkdir build
cd build
cmake -G "Visual Studio 16 2019" -A x64 -DCMAKE_GENERATOR_PLATFORM=x64 -DSNAPPY_BUILD_TESTS=OFF -DSNAPPY_BUILD_BENCHMARKS=OFF ..
msbuild Snappy.sln /p:Configuration=Debug /p:Platform=x64
msbuild Snappy.sln /p:Configuration=Release /p:Platform=x64
The resultant static library can be found in %CODE_HOME%\snappy-1.1.9\build\Debug\snappy.lib
or %CODE_HOME%\snappy-1.1.9\build\Release\snappy.lib
.
Requires Windows 8.1 SDK from - https://go.microsoft.com/fwlink/p/?LinkId=323507
cd %CODE_HOME%
wget https://github.com/lz4/lz4/archive/refs/tags/v1.9.4.zip
unzip v1.9.4.zip
cd lz4-1.9.4
cd visual\VS2017
devenv lz4.sln /upgrade
msbuild lz4.sln /p:Configuration=Debug /p:Platform=x64
msbuild lz4.sln /p:Configuration=Release /p:Platform=x64
The resultant static library can be found in %CODE_HOME%\lz4-1.9.4\visual\VS2017\bin\x64_Debug\liblz4_static.lib
or %CODE_HOME%\lz4-1.9.4\visual\VS2017\bin\x64_Release\liblz4_static.lib
.
cd %CODE_HOME%
wget https://zlib.net/fossils/zlib-1.3.1.tar.gz
tar zxvf zlib-1.3.1.tar.gz
cd zlib-1.3.1\contrib\vstudio\vc14
devenv zlibvc.sln /upgrade
msbuild zlibvc.sln /p:Configuration=Debug /p:Platform=x64
msbuild zlibvc.sln /p:Configuration=Release /p:Platform=x64
copy x64\ZlibDllDebug\zlibwapi.lib x64\ZlibStatDebug\
copy x64\ZlibDllRelease\zlibwapi.lib x64\ZlibStatRelease\
The resultant static library can be found in %CODE_HOME%\zlib-1.3.1\contrib\vstudio\vc14\x64\ZlibStatDebug\zlibstat.lib
or %CODE_HOME%\zlib-1.3.1\contrib\vstudio\vc14\x64\ZlibStatRelease\zlibstat.lib
.
wget https://github.com/facebook/zstd/archive/v1.5.5.zip
unzip v1.5.5.zip
cd zstd-1.5.5\build\VS2010
devenv zstd.sln /upgrade
msbuild zstd.sln /p:Configuration=Debug /p:Platform=x64
msbuild zstd.sln /p:Configuration=Release /p:Platform=x64
The resultant static library can be found in %CODE_HOME%\zstd-1.5.5\build\VS2010\bin\x64_Debug\libzstd_static.lib
or %CODE_HOME%\zstd-1.5.5\build\VS2010\bin\x64_Release\libzstd_static.lib
.
cd %CODE_HOME%
git clone https://github.com/facebook/rocksdb.git
cd rocksdb
Edit the file %CODE_HOME%\rocksdb\thirdparty.inc
to have these changes:
set(GFLAGS_HOME $ENV{THIRDPARTY_HOME}/gflags-2.2.2)
set(GFLAGS_INCLUDE ${GFLAGS_HOME}/target/include)
set(GFLAGS_LIB_DEBUG ${GFLAGS_HOME}/target/lib/Debug/gflags_static.lib)
set(GFLAGS_LIB_RELEASE ${GFLAGS_HOME}/target/lib/Release/gflags_static.lib)
set(SNAPPY_HOME $ENV{THIRDPARTY_HOME}/snappy-1.1.9)
set(SNAPPY_INCLUDE ${SNAPPY_HOME} ${SNAPPY_HOME}/build)
set(SNAPPY_LIB_DEBUG ${SNAPPY_HOME}/build/Debug/snappy.lib)
set(SNAPPY_LIB_RELEASE ${SNAPPY_HOME}/build/Release/snappy.lib)
set(LZ4_HOME $ENV{THIRDPARTY_HOME}/lz4-1.9.4)
set(LZ4_INCLUDE ${LZ4_HOME}/lib)
set(LZ4_LIB_DEBUG ${LZ4_HOME}/visual/VS2017/bin/x64_Debug/liblz4_static.lib)
set(LZ4_LIB_RELEASE ${LZ4_HOME}/visual/VS2017/bin/x64_Release/liblz4_static.lib)
set(ZLIB_HOME $ENV{THIRDPARTY_HOME}/zlib-1.3.1)
set(ZLIB_INCLUDE ${ZLIB_HOME})
set(ZLIB_LIB_DEBUG ${ZLIB_HOME}/contrib/vstudio/vc14/x64/ZlibStatDebug/zlibstat.lib)
set(ZLIB_LIB_RELEASE ${ZLIB_HOME}/contrib/vstudio/vc14/x64/ZlibStatRelease/zlibstat.lib)
set(ZSTD_HOME $ENV{THIRDPARTY_HOME}/zstd-1.5.5)
set(ZSTD_INCLUDE ${ZSTD_HOME}/lib ${ZSTD_HOME}/lib/dictBuilder)
set(ZSTD_LIB_DEBUG ${ZSTD_HOME}/build/VS2010/bin/x64_Debug/libzstd_static.lib)
set(ZSTD_LIB_RELEASE ${ZSTD_HOME}/build/VS2010/bin/x64_Release/libzstd_static.lib)
And then finally to compile RocksDB:
-
NOTE: The default CMake build will generate MSBuild project files which include the
/arch:AVX2
flag. If you have this CPU extension instruction set, then the generated binaries will also only work on other CPU's with AVX2. If you want to create a build which has no specific CPU extensions, then you should also pass the-DPORTABLE=1
flag in thecmake
arguments below. -
NOTE: The build options below include
-DXPRESS=1
which enables Microsoft XPRESS compression. This requires Windows 10 or newer to work reliably and is not backwards compatible with older versions of Windows. At present we build RocksJava releases without XPRESS. -
NOTE: Optionally RocksDBJava can also be built with the option
-DWITH_JNI_EXPORT_ROCKSDB=ON
. This causes the RockDB C API to also be exported in the build JNI binaries. Using the option requires the MSVCdumpbin
tool to be present on the path or same directory as the msvc compiler.
mkdir build
cd build
set JAVA_HOME="C:\Program Files\BellSoft\LibericaJDK-8-Full"
set THIRDPARTY_HOME=C:/Users/aretter/code
cmake -G "Visual Studio 16 2019" -A x64 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=20 -DJNI=1 -DGFLAGS=1 -DSNAPPY=1 -DSNAPPY_BUILD_TESTS=OFF -DSNAPPY_BUILD_BENCHMARKS=OFF -DSNAPPY_HAVE_BMI2=0 -DLZ4=1 -DZLIB=1 -DZSTD=1 -DXPRESS=1 ..
msbuild rocksdb.sln /p:Configuration=Release
The easiest way to run the tests for RocksDB Java is via ctest
from build/java
directory:
cd build\java
ctest -C Release -j <number of CPU cores>
This is a simple step-by-step explanation of how I was able to build RocksDB (or RocksJava) and all of the 3rd-party libraries on Microsoft Windows 11. The Windows build system was already in place, however it took some trial-and-error for me to be able to build the 3rd-party libraries and incorporate them into the build.
- Microsoft Visual Studio 2022 (Community) with "Desktop development with C++" installed
- CMake - I used version 3.27.3 installed from the 64bit MSI installer
- Java 8 - I used BellSoft Liberica JDK 8 https://bell-sw.com/pages/downloads/
- Power Shell
Create a directory somewhere on your machine that will be used as a container for both the RocksDB source code and that of its 3rd-party dependencies. On my machine I used C:\Users\aretter\code
, from hereon in I will just refer to it as CODE_HOME
; which can be set as an environment variable (in Powershell), i.e. $Env:CODE_HOME = "C:\Users\aretter\code"
.
All of the following is executed from the "Developer PowerShell for VS2022":
cd $Env:CODE_HOME
wget https://github.com/gflags/gflags/archive/refs/tags/v2.2.2.zip -OutFile gflags.zip
Expand-Archive gflags.zip -DestinationPath .
cd gflags-2.2.2
mkdir target
cd target
cmake -G "Visual Studio 17 2022" -A x64 ..
msbuild gflags.sln /p:Configuration=Debug /p:Platform=x64
msbuild gflags.sln /p:Configuration=Release /p:Platform=x64
The resultant static library can be found in $Env:CODE_HOME\gflags-2.2.2\target\lib\Debug\gflags_static_debug.lib
or $Env:CODE_HOME\gflags-2.2.2\target\lib\Release\gflags_static.lib
.
cd $Env:CODE_HOME
wget https://github.com/google/snappy/archive/refs/tags/1.1.9.zip -OutFile snappy.zip
Expand-Archive snappy.zip -DestinationPath .
cd snappy-1.1.9
mkdir target
cd target
cmake -G "Visual Studio 17 2022" -A x64 -DCMAKE_GENERATOR_PLATFORM=x64 -DSNAPPY_BUILD_TESTS=OFF -DSNAPPY_BUILD_BENCHMARKS=OFF ..
msbuild Snappy.sln /p:Configuration=Debug /p:Platform=x64
msbuild Snappy.sln /p:Configuration=Release /p:Platform=x64
The resultant static library can be found in $Env:CODE_HOME\snappy-1.1.9\build\Debug\snappy.lib
or $Env:CODE_HOME\snappy-1.1.9\build\Release\snappy.lib
.
cd $Env:CODE_HOME
wget https://github.com/lz4/lz4/archive/refs/tags/v1.9.4.zip -OutFile lz4.zip
Expand-Archive lz4.zip -DestinationPath .
cd lz4-1.9.4
mkdir target
cd target
cmake -G "Visual Studio 17 2022" -A x64 -S ../build/cmake
msbuild LZ4.sln /p:Configuration=Debug /p:Platform=x64
msbuild LZ4.sln /p:Configuration=Release /p:Platform=x64
The resultant static library can be found in $Env:CODE_HOME\lz4-1.9.4\target\Debug
or $Env:CODE_HOME\lz4-1.9.4\target\Release
.
cd $Env:CODE_HOME
wget https://github.com/madler/zlib/releases/download/v1.3.1/zlib131.zip -OutFile zlib.zip
Expand-Archive zlib.zip -DestinationPath .
cd zlib-1.3.1
mkdir target
cd target
cmake -G "Visual Studio 17 2022" -A x64 -S ..
msbuild zlib.sln /p:Configuration=Debug /p:Platform=x64
msbuild zlib.sln /p:Configuration=Release /p:Platform=x64
The resultant static library can be found in $Env:CODE_HOME\zlib-1.3.1\target\Debug\zlibstaticd.lib
or $Env:CODE_HOME\zlib-1.3.1\target\Release\zlibstatic.lib
.
cd $Env:CODE_HOME
wget https://github.com/facebook/zstd/archive/refs/tags/v1.5.5.zip -OutFile zstd.zip
Expand-Archive zstd.zip -DestinationPath .
cd zstd-1.5.5
mkdir target
cd target
cmake -G "Visual Studio 17 2022" -A x64 -S ../build/cmake
msbuild zstd.sln /p:Configuration=Debug /p:Platform=x64
msbuild zstd.sln /p:Configuration=Release /p:Platform=x64
The resultant static library can be found in $Env:CODE_HOME\zstd-1.5.5\target\lib\Debug\zstd_static.lib
or $Env:CODE_HOME\zstd-1.5.5\target\lib\Release\zstd_static.lib
.
cd $Env:CODE_HOME
wget https://github.com/facebook/rocksdb/archive/refs/heads/main.zip -OutFile rocksdb.zip
Expand-Archive rocksdb.zip -DestinationPath .
cd rocksdb-main
Edit the file %CODE_HOME%\rocksdb-main\thirdparty.inc
to have these changes:
set(GFLAGS_HOME $ENV{THIRDPARTY_HOME}/gflags-2.2.2)
set(GFLAGS_INCLUDE ${GFLAGS_HOME}/target/include)
set(GFLAGS_LIB_DEBUG ${GFLAGS_HOME}/target/lib/Debug/gflags_static.lib)
set(GFLAGS_LIB_RELEASE ${GFLAGS_HOME}/target/lib/Release/gflags_static.lib)
set(SNAPPY_HOME $ENV{THIRDPARTY_HOME}/snappy-1.1.9)
set(SNAPPY_INCLUDE ${SNAPPY_HOME} ${SNAPPY_HOME}/target)
set(SNAPPY_LIB_DEBUG ${SNAPPY_HOME}/target/Debug/snappy.lib)
set(SNAPPY_LIB_RELEASE ${SNAPPY_HOME}/target/Release/snappy.lib)
set(LZ4_HOME $ENV{THIRDPARTY_HOME}/lz4-1.9.4)
set(LZ4_INCLUDE ${LZ4_HOME}/lib)
set(LZ4_LIB_DEBUG ${LZ4_HOME}/target/Debug/lz4.lib)
set(LZ4_LIB_RELEASE ${LZ4_HOME}/target/Release/lz4.lib)
set(ZLIB_HOME $ENV{THIRDPARTY_HOME}/zlib-1.3.1)
set(ZLIB_INCLUDE ${ZLIB_HOME} ${ZLIB_HOME}/target )
set(ZLIB_LIB_DEBUG ${ZLIB_HOME}/target/Debug/zlibstatic.lib)
set(ZLIB_LIB_RELEASE ${ZLIB_HOME}/target/Release/zlibstatic.lib)
set(ZSTD_HOME $ENV{THIRDPARTY_HOME}/zstd-1.5.5)
set(ZSTD_INCLUDE ${ZSTD_HOME}/lib ${ZSTD_HOME}/lib/dictBuilder)
set(ZSTD_LIB_DEBUG ${ZSTD_HOME}/target/lib/Debug/zstd_static.lib)
set(ZSTD_LIB_RELEASE ${ZSTD_HOME}/target/lib/Release/zstd_static.lib)
And then finally to compile RocksDB:
-
NOTE: The default CMake build will generate MSBuild project files which include the
/arch:AVX2
flag. If you have this CPU extension instruction set, then the generated binaries will also only work on other CPU's with AVX2. If you want to create a build which has no specific CPU extensions, then you should also pass the-DPORTABLE=1
flag in thecmake
arguments below. -
NOTE: The build options below include
-DXPRESS=1
which enables Microsoft XPRESS compression. This requires Windows 10 or newer to work reliably and is not backwards compatible with older versions of Windows. At present we build RocksJava releases without XPRESS. -
NOTE: Optionally RocksDBJava can also be built with the option
-DWITH_JNI_EXPORT_ROCKSDB=ON
. This causes the RockDB C API to also be exported in the build JNI binaries. Using the option requires the MSVCdumpbin
tool to be present on the path or same directory as the msvc compiler.
mkdir build
cd build
$Env:JAVA_HOME="C:\Program Files\BellSoft\LibericaJDK-8-Full"
$Env:THIRDPARTY_HOME="C:\Users\aretter\code"
$env:Path = $env:JAVA_HOME + ";" + $env:Path #CMake find(JNI) have a bug and doesn't support JAVA_HOME
cmake -G "Visual Studio 17 2022" -A x64 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=20 -DJNI=1 -DGFLAGS=1 -DSNAPPY=1 -DSNAPPY_BUILD_TESTS=OFF -DSNAPPY_BUILD_BENCHMARKS=OFF -DSNAPPY_HAVE_BMI2=0 -DLZ4=1 -DZLIB=1 -DZSTD=1 -DXPRESS=1 ..
msbuild rocksdb.sln /p:Configuration=Release -maxCpuCount -property:Platform=x64 -target:rocksdb-shared`;rocksdbjni_test_classes`;rocksdbjni`;rocksdbjni_test_classes
Then the resulting libraries can be found in :
- Static library :
build/Release/rocksdb.lib
- Dynamic library:
rocksdb-shared.dll
- Java jar archive:
build/java/rocksdbjni_classes.jar
- Java native library:
build/java/Release/rocksdbjni.dll
The easiest way to run the tests for RocksDB Java is via ctest
from build/java
directory:
cd build\java
ctest -C Release -j <number of CPU cores>
-
cd %home%\vcpkg\ports\rocksdb
, %home% is the path where you installed your vcpkg -
set(VCPKG_LIBRARY_LINKAGE static)
at the top ofportfile.cmake
and then runvcpkg install rocksdb:x64-windows
. If you have installed RocksDB as a static library, then runvcpkg remove rocksdb:x64-windows
before the install command. The resultant static library can be found in%home%\vcpkg\packages\rocksdb_x64-windows\lib\rocksdb.lib
rather than%home%\vcpkg\packages\rocksdb_x64-windows\lib\rocksdb-shared.lib
.
- include
Shlwapi.lib
andRpcrt4.lib
as the input of your project linker manually.
Contents
- RocksDB Wiki
- Overview
- RocksDB FAQ
- Terminology
- Requirements
- Contributors' Guide
- Release Methodology
- RocksDB Users and Use Cases
- RocksDB Public Communication and Information Channels
-
Basic Operations
- Iterator
- Prefix seek
- SeekForPrev
- Tailing Iterator
- Compaction Filter
- Multi Column Family Iterator
- Read-Modify-Write (Merge) Operator
- Column Families
- Creating and Ingesting SST files
- Single Delete
- Low Priority Write
- Time to Live (TTL) Support
- Transactions
- Snapshot
- DeleteRange
- Atomic flush
- Read-only and Secondary instances
- Approximate Size
- User-defined Timestamp
- Wide Columns
- BlobDB
- Online Verification
- Options
- MemTable
- Journal
- Cache
- Write Buffer Manager
- Compaction
- SST File Formats
- IO
- Compression
- Full File Checksum and Checksum Handoff
- Background Error Handling
- Huge Page TLB Support
- Tiered Storage (Experimental)
- Logging and Monitoring
- Known Issues
- Troubleshooting Guide
- Tests
- Tools / Utilities
-
Implementation Details
- Delete Stale Files
- Partitioned Index/Filters
- WritePrepared-Transactions
- WriteUnprepared-Transactions
- How we keep track of live SST files
- How we index SST
- Merge Operator Implementation
- RocksDB Repairer
- Write Batch With Index
- Two Phase Commit
- Iterator's Implementation
- Simulation Cache
- [To Be Deprecated] Persistent Read Cache
- DeleteRange Implementation
- unordered_write
- Extending RocksDB
- RocksJava
- Lua
- Performance
- Projects Being Developed
- Misc