Skip to content

Building Couchbase

Jump to bottom
aborkar-ibm edited this page Sep 6, 2019 · 52 revisions

Building Couchbase

The instructions provided below specify the steps to build Couchbase version 6.0.0 on Linux on IBM Z for following distributions:

  • Ubuntu (16.04, 18.04)
  • RHEL (7.5, 7.6, 7.7)
  • SLES (12 SP4)

General Notes:

  • When following the steps below please use a standard permission user unless otherwise specified.

  • A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Step 1 : Install Dependencies

export SOURCE_ROOT=/<source_root>/
  • Ubuntu (16.04)
sudo apt-get update && sudo apt-get install autoconf automake check cmake curl flex gcc-5 git g++-5 libcurl4-openssl-dev libevent-dev libglib2.0-dev libncurses5-dev libsnappy-dev libssl-dev libtool libxml2-utils make openssl pkg-config python python-dev ruby sqlite3 subversion tar unixodbc unixodbc-dev vim wget xsltproc golang-1.10 patch xinetd
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 40
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-5 40
export PATH=/usr/lib/go-1.10/bin:$PATH
  • Ubuntu (18.04)
sudo apt-get update && sudo apt-get install autoconf automake check cmake curl flex gcc-5 git g++-5 libcurl4-gnutls-dev libevent-dev libglib2.0-dev libncurses5-dev libsnappy-dev libssl1.0-dev libtool libxml2-utils make openssl pkg-config python python-dev ruby sqlite3 subversion tar unixodbc unixodbc-dev vim wget xsltproc golang-1.10 patch xinetd
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 40
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-5 40
export PATH=/usr/lib/go-1.10/bin:$PATH
  • RHEL (7.5, 7.6, 7.7)
export LD_LIBRARY_PATH=/usr/local/lib64/:/usr/lib64:/usr/local/lib/:/usr/lib/
sudo yum install autoconf automake bzip2 bzip2-devel check check-devel cmake curl flex git java-1.8.0-ibm java-1.8.0-ibm-devel libcurl-devel libevent-devel libtool libxml2-devel libxslt-devel make ncurses-devel pkgconfig python python-devel ruby ruby-devel snappy-devel sqlite-devel subversion tar unixODBC vim wget xinetd golang patch gcc gcc-c++
  • SLES (12 SP4)
export LD_LIBRARY_PATH=/usr/lib64:/usr/local/lib64:/usr/local/lib/:/usr/lib/:$SOURCE_ROOT/couchbase/install/lib/:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/root/couchbase/couchbase/build/kv_engine
sudo zypper install autoconf automake cmake curl flex git-core libevent-2_0-5 libtool libxml2-tools libxslt-tools make libncurses6 ncurses-devel pkg-config libsnappy1 sqlite3 python python-xml ruby tar unixODBC vim wget xinetd libz1 zlib-devel gcc gcc-c++ patch

Note: python -V should be 2.7.x

  • Install Go (For SLES only)

    Instructions to install Go can be found here.

  • For RHEL and SLES only:

    • gcc(5.5.0)
    cd $SOURCE_ROOT
wget http://ftp.gnu.org/gnu/gcc/gcc-5.5.0/gcc-5.5.0.tar.gz
tar -xzf gcc-5.5.0.tar.gz && cd gcc-5.5.0
./contrib/download_prerequisites
mkdir build && cd build/
../configure --disable-multilib --disable-checking --enable-languages=c,c++ --enable-multiarch --enable-shared --enable-threads=posix --without-included-gettext --with-system-zlib --prefix=/usr/local
make && sudo make install
    • For RHEL
    sudo mv /usr/bin/gcc /usr/bin/gcc-4.8.5
sudo mv /usr/bin/g++ /usr/bin/g++-4.8.5
sudo mv /usr/bin/c++ /usr/bin/c++-4.8.5
sudo rm /usr/bin/cc
    • For SLES
    sudo rm /usr/bin/gcc
sudo rm /usr/bin/g++
sudo rm /usr/bin/c++
sudo rm /usr/bin/cc
    • For RHEL and SLES
    sudo update-alternatives --install /usr/bin/cc cc /usr/local/bin/gcc 40
sudo update-alternatives --install /usr/bin/gcc gcc /usr/local/bin/gcc 40
sudo update-alternatives --install /usr/bin/g++ g++ /usr/local/bin/g++ 40
sudo update-alternatives --install /usr/bin/c++ c++ /usr/local/bin/c++ 40
export CC=/usr/local/bin/s390x-ibm-linux-gnu-gcc
export CXX=/usr/local/bin/s390x-ibm-linux-gnu-g++
sudo /sbin/ldconfig
gcc --version

    Note: gcc --version should be 5.5.0

    • check(0.12.0)(For SLES only)
    wget https://github.com/libcheck/check/releases/download/0.12.0/check-0.12.0.tar.gz
tar -xvf check-0.12.0.tar.gz
autoreconf --install
./configure
make
make check
sudo make install
hash -r
sudo ldconfig

  • Build openssl (Ubuntu 18.04, RHEL and SLES only)

 cd $SOURCE_ROOT
 wget https://ftp.openssl.org/source/old/1.0.2/openssl-1.0.2h.tar.gz
 tar zxf openssl-1.0.2h.tar.gz
 cd openssl-1.0.2h
 ./config --prefix=/usr --openssldir=/etc/ssl --libdir=lib shared zlib-dynamic
 make depend
 make && sudo make install
  • cmake(3.14.5)(For RHEL only)
  cd $SOURCE_ROOT
  wget https://github.com/Kitware/CMake/releases/download/v3.14.5/cmake-3.14.5.tar.gz
  tar -xvf cmake-3.14.5.tar.gz
  cd cmake-3.14.5
  ./bootstrap
  make && sudo make install
  hash -r

  • For SLES:

    • lz4(1.8.1.2)
      cd $SOURCE_ROOT
  wget https://github.com/lz4/lz4/archive/v1.8.1.2.tar.gz
  tar -xvf v1.8.1.2.tar.gz
  cd v1.8.1.2
  make
  sudo make install
  hash -r
    • snappy(1.1.5)
      cd $SOURCE_ROOT
  git clone https://github.com/google/snappy.git
  cd snappy
  git checkout 1.1.5
  mkdir build
  cd build && cmake ../ &&. make
  sudo make install
  hash -r
    • libevent(2.1.8)
      cd $SOURCE_ROOT
  wget https://github.com/libevent/libevent/releases/download/release-2.1.8-stable/libevent-2.1.8-stable.tar.gz
  tar -xvf libevent-2.1.8-stable.tar.gz
  cd libevent-2.1.8-stable
  ./configure
  make
  sudo make install
    • curl(7.60.0)
      cd $SOURCE_ROOT
  git clone https://github.com/curl/curl.git
  cd curl
  git checkout curl-7_60_0
  cmake -G "Unix Makefiles" .
  make
  sudo make install
  hash -r

  • Install Java (Ubuntu 16.04, 18.04 and SLES 12 SP4 Only)

cd $SOURCE_ROOT
cat <<EOF > installer.properties
LICENSE_ACCEPTED=TRUE
USER_INSTALL_DIR=/opt/ibm/java-s390x-80
-fileOverwrite_/opt/ibm/java-s390x-80/_uninstall/uninstall.lax=Yes
EOF
wget http://public.dhe.ibm.com/ibmdl/export/pub/systems/cloud/runtimes/java/8.0.5.37/linux/s390x/ibm-java-s390x-sdk-8.0-5.37.bin
chmod +x ibm-java-s390x-sdk-8.0-5.37.bin
sudo ./ibm-java-s390x-sdk-8.0-5.37.bin -i silent
export JAVA_HOME=/opt/ibm/java-s390x-80
export PATH=$JAVA_HOME/bin:$PATH

Note: java -version should be 1.8.0_211

  • Install json
	cd $SOURCE_ROOT
	git clone https://github.com/nlohmann/json
	cd json
	export PATH=$PATH:`pwd`
  • Install Boost 1.69.0
	cd $SOURCE_ROOT
	wget https://dl.bintray.com/boostorg/release/1.69.0/source/boost_1_69_0.tar.gz
	tar -xvzf boost_1_69_0.tar.gz
	sudo ln -s $SOURCE_ROOT/boost_1_69_0/boost /usr/include/boost
  • Erlang couchbase-watson
	cd $SOURCE_ROOT
	git clone https://github.com/couchbasedeps/erlang.git
	cd erlang && git checkout couchbase-watson
	./otp_build autoconf
	touch lib/debugger/SKIP lib/megaco/SKIP lib/observer/SKIP lib/wx/SKIP
  • For (Ubuntu 16.04, RHEL, and SLES Only)
	./configure --prefix=/usr/local --enable-smp-support --disable-hipe --disable-fp-exceptions CFLAGS="-fno-strict-aliasing -O3 -ggdb3 -DOPENSSL_NO_EC=1"
	make -j$(cat /proc/cpuinfo | grep processor | wc -l) && sudo make install
	hash -r
  • For Ubuntu 18.04 Only
	./configure --prefix=/usr/local --enable-smp-support --disable-hipe --disable-fp-exceptions CFLAGS="-fno-strict-aliasing -O3 -ggdb3"
	make -j$(cat /proc/cpuinfo | grep processor | wc -l) && sudo make install
	hash -r

  • Install Flatbuffers

    • Download the source code
    	cd $SOURCE_ROOT
	git clone https://github.com/google/flatbuffers
	cd flatbuffers && git checkout v1.5.0
    • Create a patch for the file include/flatbuffers/flatbuffers.h 
    cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ flatbuffers/include/flatbuffers/flatbuffers.h
@@ -69,6 +69,7 @@

 // The wire format uses a little endian encoding (since that's efficient for
 // the common platforms).
+#define FLATBUFFERS_LITTLEENDIAN 0
 #if !defined(FLATBUFFERS_LITTLEENDIAN)
	#if defined(__GNUC__) || defined(__clang__)
 #ifdef __BIG_ENDIAN__
EOF
    • Build and Install flatbuffers
    	cd flatbuffers
	cmake -G "Unix Makefiles"
	make && export PATH=$PATH:`pwd`
	sudo make install
	hash -r

  • Install ICU 54.1

	cd $SOURCE_ROOT
	git clone https://github.com/couchbasedeps/icu4c.git
	cd icu4c/source && git checkout r54.1
	./configure --prefix=/usr/local --disable-extras --disable-layout --disable-tests --disable-samples
	make && sudo make install
	hash -r

  • Install Jemalloc 4.3.1

    	cd $SOURCE_ROOT
	git clone https://github.com/couchbasedeps/jemalloc.git
	cd jemalloc && git checkout 4.3.1
	autoconf configure.ac > configure
	chmod u+x configure
	CPPFLAGS=-I/usr/local/include ./configure --prefix=/usr/local --with-jemalloc-prefix=je_ --disable-cache-oblivious --disable-zone-allocator --enable-prof
	make build_lib_shared
	sudo make install_lib_shared install_include
    • For SLES:

    Note: ls /usr/lib64/libjemalloc.so.2 should show No such file or directory. If /usr/lib64/libjemalloc.so.2 exists, then perform the following:

    	mv /usr/lib64/libjemalloc.so.2 ~/badlibjemalloc

  • Install V8 5.9

	cd $SOURCE_ROOT
	git clone https://github.com/couchbasedeps/v8.git
	cd v8
	git checkout 5.9-couchbase
	make -j$(cat /proc/cpuinfo | grep processor | wc -l) s390x.release GYPFLAGS+="-Dcomponent=shared_library -Dv8_enable_backtrace=1 -Dv8_use_snapshot='true' -Dclang=0 -Dv8_use_external_startup_data=0 -Dv8_enable_i18n_support=0 -Dtest_isolation_mode=noop" PYTHONPATH=`pwd`/third_party/argparse-1.4.0
	sudo cp -vR include/* /usr/local/include/
	sudo chmod 644 /usr/local/include/libplatform/libplatform.h
	sudo chmod 644 /usr/local/include/v8*h
	sudo cp -v out/s390x.release/lib.target/libv8*.so /usr/local/lib/
	sudo chmod -f 755 /usr/local/lib/libv8*.so

Step 2 : Download the Repo tool

  • Download the Repo tool and ensure that it has execute permissions
	cd $SOURCE_ROOT
	curl https://storage.googleapis.com/git-repo-downloads/repo > repo
	chmod a+x repo

Step 3 : Build and test Couchbase

3.1) Clone Couchbase 6.0.0

  • Clone the Couchbase 6.0.0 release using the the manifest file via Repo tool with the init and sync commands
	mkdir -p $SOURCE_ROOT/couchbase
	cd $SOURCE_ROOT/couchbase
	git config --global user.email your@email.addr
	git config --global user.name  your_name
	../repo init -u git://github.com/couchbase/manifest -m released/6.0.0.xml
	../repo sync

3.2) Patch Files

  • Patch the following files using in-line scripts as shown
  1. $SOURCE_ROOT/couchbase/tlm/CMakeLists.txt
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/tlm/CMakeLists.txt
@@ -73,7 +73,7 @@

 # Try to download the prebuilt 3rd-party dependencies by default
 IF (NOT DEFINED CB_DOWNLOAD_DEPS)
-   SET(CB_DOWNLOAD_DEPS True
+   SET(CB_DOWNLOAD_DEPS False
        CACHE BOOL "Download prebuilt dependencies by default")
 ENDIF (NOT DEFINED CB_DOWNLOAD_DEPS)

@@ -120,7 +120,7 @@
 INCLUDE(FindCouchbaseErlang)
 INCLUDE(FindCouchbaseDtrace)
 INCLUDE(FindCouchbaseGo)
-INCLUDE(FindCouchbaseBreakpad)
+#INCLUDE(FindCouchbaseBreakpad)
 INCLUDE(FindCouchbaseFlatbuffers)
 INCLUDE(FindCouchbaseValgrind)
 INCLUDE(FindCouchbaseV8)
EOF
  1. $SOURCE_ROOT/couchbase/forestdb/src/arch.h
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/forestdb/src/arch.h
@@ -330,6 +330,10 @@
         #define spin_unlock(arg) pthread_spin_unlock(arg)
         #define spin_destroy(arg) pthread_spin_destroy(arg)
         #define SPIN_INITIALIZER (spin_t)(1)
+        #if defined(__s390x__)
+            #undef SPIN_INITIALIZER
+            #define SPIN_INITIALIZER (spin_t)(0)
+        #endif
     #endif
     #ifndef mutex_t
         // mutex
EOF
  1. $SOURCE_ROOT/couchbase/couchstore/src/views/bin/couch_view_file_merger.cc
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/couchstore/src/views/bin/couch_view_file_merger.cc
@@ -37,8 +37,7 @@
     MERGE_FILE_TYPE_ID_BTREE = 'i',
     MERGE_FILE_TYPE_MAPREDUCE_VIEW = 'v',
     MERGE_FILE_TYPE_SPATIAL = 's'
-} merge_file_type_t;
-
+}; typedef unsigned char merge_file_type_t;

 int main(int argc, char *argv[])
 {
EOF
  1. $SOURCE_ROOT/couchbase/forestdb/utils/debug.cc
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/forestdb/utils/debug.cc
@@ -89,6 +89,8 @@
     ucontext_t *u = (ucontext_t *)context;
 #ifdef REG_RIP // Test if the Program Counter is 64 bits
     unsigned char *pc = (unsigned char *)u->uc_mcontext.gregs[REG_RIP];
+#elif __s390x__
+    unsigned char *pc = (unsigned char *)u->uc_mcontext.psw.addr;
 #else // 32 bit machine, PC is stored in %eip register
     unsigned char *pc = (unsigned char *)u->uc_mcontext.gregs[REG_EIP];
 #endif // REG_RIP for 64-bit machines
EOF
  1. $SOURCE_ROOT/couchbase/platform/CMakeLists.txt
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/platform/CMakeLists.txt
@@ -187,7 +187,6 @@
                             src/cbassert.c
                             src/checked_snprintf.cc
                             src/crc32c.cc
-                            src/crc32c_sse4_2.cc
                             src/crc32c_private.h
                             src/getopt.cc
                             src/global_new_replacement.cc
EOF
  1. $SOURCE_ROOT/couchbase/platform/include/platform/crc32c.h
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/platform/include/platform/crc32c.h
@@ -39,7 +39,7 @@
 // To fix will require refactoring to hide the X86 dependencies when
 // built on another platform.
 //
-#if !defined(__x86_64__) && !defined(_M_X64) && !defined(_M_IX86)
+#if !defined(__x86_64__) && !defined(_M_X64) && !defined(_M_IX86) && !defined(__s390x__)
 #error "crc32c requires X86 SSE4.2 for hardware acceleration"
 #endif
EOF
  1. $SOURCE_ROOT/couchbase/platform/src/crc32c.cc
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/platform/src/crc32c.cc
@@ -52,6 +52,7 @@
 //

 #include "platform/crc32c.h"
+#include <crc32-s390x.h>
 #include "crc32c_private.h"

 #include <stdint.h>
@@ -60,7 +61,7 @@
 // select header file for cpuid.
 #if defined(WIN32)
 #include <intrin.h>
-#elif defined(__clang__) || defined(__GNUC__)
+#elif defined(__clang__) || defined(__GNUC__) && !defined(__s390x__)
 #include <cpuid.h>
 #endif

@@ -368,10 +369,10 @@
 // If SSE4.2 is available then hardware acceleration is used.
 //
 crc32c_function setup_crc32c() {
-    const uint32_t SSE42 = 0x00100000;
+//    const uint32_t SSE42 = 0x00100000;
 
     crc32c_function f = crc32c_sw;
-
+/*
 #if defined(WIN32)
     std::array<int, 4> registers = {{0,0,0,0}};
     __cpuid(registers.data(), 1);
@@ -383,7 +384,7 @@
     if (registers[2] & SSE42) {
         f = crc32c_hw;
     }
-
+*/
     return f;
 }
 

EOF
  1. $SOURCE_ROOT/couchbase/platform/tests/CMakeLists.txt
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/platform/tests/CMakeLists.txt
@@ -5,7 +5,7 @@
 ADD_SUBDIRECTORY(checked_snprintf)
 ADD_SUBDIRECTORY(cjson)
 ADD_SUBDIRECTORY(corestore)
-ADD_SUBDIRECTORY(crc32)
+#ADD_SUBDIRECTORY(crc32)
 ADD_SUBDIRECTORY(dirutils)
 ADD_SUBDIRECTORY(gethrtime)
 ADD_SUBDIRECTORY(gettimeofday)
EOF
  1. $SOURCE_ROOT/couchbase/goproj/src/github.com/couchbase/indexing/secondary/memdb/skiplist/skiplist.go 
bind 'set disable-completion on'
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/goproj/src/github.com/couchbase/indexing/secondary/memdb/skiplist/skiplist.go
@@ -74,12 +74,13 @@
 	}
 
 	if cfg.UseMemoryMgmt {
-		s.freeNode = func(n *Node) {
-			if Debug {
-				debugMarkFree(n)
-			}
-			cfg.Free(unsafe.Pointer(n))
-		}
+		s.freeNode = func(*Node) {}
+		//s.freeNode = func(n *Node) {
+		//	if Debug {
+		//		debugMarkFree(n)
+		//	}
+		//	cfg.Free(unsafe.Pointer(n))
+		//}
 	} else {
 		s.freeNode = func(*Node) {}
 	}
EOF
bind 'set disable-completion off'
  1. $SOURCE_ROOT/couchbase/kv_engine/daemon/subdocument_validators.cc
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/kv_engine/daemon/subdocument_validators.cc
@@ -435,7 +435,7 @@
 
     if ((req->request.magic != PROTOCOL_BINARY_REQ) ||
         (req->request.keylen == 0) ||
-        (req->request.bodylen < minimum_body_len) ||
+        (htonl(__bswap_32(req->request.bodylen)) < minimum_body_len) ||
         (req->request.datatype != PROTOCOL_BINARY_RAW_BYTES)) {
         return PROTOCOL_BINARY_RESPONSE_EINVAL;
     }
EOF
  1. $SOURCE_ROOT/couchbase/benchmark/src/cycleclock.h
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/benchmark/src/cycleclock.h
@@ -82,6 +82,11 @@
   uint64_t low, high;
   __asm__ volatile("rdtsc" : "=a"(low), "=d"(high));
   return (high << 32) | low;
+#elif defined(__s390__) // Covers both s390 and s390x.
+  // Return the CPU clock.
+  uint64_t tsc;
+  asm("stck %0" : "=Q"(tsc) : : "cc");
+  return tsc;
 #elif defined(__powerpc__) || defined(__ppc__)
   // This returns a time-base, which is not always precisely a cycle-count.
   int64_t tbl, tbu0, tbu1;
EOF
  1. $SOURCE_ROOT/couchbase/platform/include/platform/cacheline_padded.h
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/platform/include/platform/cacheline_padded.h
@@ -20,7 +20,7 @@
 // expect this to be a single cache line (64B on x86-64), but on
 // Sandybridge (at least) it has been observed that pairs of
 // cachelines can interfere with each other.
-#define FALSE_SHARING_RANGE 128
+#define FALSE_SHARING_RANGE 64

 namespace cb {
EOF
  1. $SOURCE_ROOT/couchbase/kv_engine/engines/ep/src/ep_engine.cc
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/kv_engine/engines/ep/src/ep_engine.cc
@@ -4353,7 +4353,7 @@
             vb_id);

         // Get key stats
-        uint16_t keystatus = 0;
+        uint8_t keystatus = 0;
         struct key_stats kstats;
         memset(&kstats, 0, sizeof(key_stats));
         ENGINE_ERROR_CODE rv = kvBucket->getKeyStats(
EOF
  1. $SOURCE_ROOT/couchbase/kv_engine/engines/ep/src/tagged_ptr.h
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/kv_engine/engines/ep/src/tagged_ptr.h
@@ -19,7 +19,7 @@

 #include <memory>

-#if !__x86_64__ && !_M_X64
+#if !__x86_64__ && !_M_X64 && !__s390x__
 #error "TaggedPtr is x64 specific code.  Not tested on other architectures"
 #endif
EOF
  1. $SOURCE_ROOT/couchbase/tlm/cmake/Modules/CBDownloadDeps.cmake
cd $SOURCE_ROOT
cat <<'EOF' | patch -p0
+++ couchbase/tlm/cmake/Modules/CBDownloadDeps.cmake
@@ -269,7 +269,11 @@
       SET (_gofile "go${GOVERSION}.freebsd-amd64.tar.gz")
     ELSE ()
       # Presumed Linux
-      SET (_gofile "go${GOVERSION}.linux-amd64.tar.gz")
+      IF (_arch STREQUAL "s390x")
+        SET (_gofile "go${GOVERSION}.linux-s390x.tar.gz")
+      ELSE ()
+        SET (_gofile "go${GOVERSION}.linux-amd64.tar.gz")
+      ENDIF ()
     ENDIF ()
     SET (_cachefile "${CB_DOWNLOAD_DEPS_CACHE}/${_gofile}")
     IF (NOT EXISTS "${_cachefile}")
EOF
  1. $SOURCE_ROOT/couchbase/kv_engine/engines/ep/tests/ep_testsuite_xdcr.cc
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/kv_engine/engines/ep/tests/ep_testsuite_xdcr.cc
@@ -2384,7 +2384,7 @@
     itemMeta.flags = 0xdeadbeef;

     // Watson (4.6) accepts valid encodings, but ignores them
-    std::vector<char> junkMeta = {-2,-1,2,3};
+    std::vector<char> junkMeta = {char(-2),char(-1),2,3};

     int force = 0;
EOF
  1. $SOURCE_ROOT/couchbase/platform/include/platform/n_byte_integer.h
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/platform/include/platform/n_byte_integer.h
@@ -98,12 +98,20 @@
     /// @return the current value as a uint64_t
     uint64_t load() const {
         uint64_t value = 0;
+#if __s390x__
+	std::copy_n(counter.begin(), N, ((8-N)+reinterpret_cast<uint8_t*>(&value)));
+#else
         std::copy_n(counter.begin(), N, reinterpret_cast<uint8_t*>(&value));
+#endif
         return value;
     }
 
     void store(uint64_t value) {
+#if __s390x__
+        std::copy_n((reinterpret_cast<uint8_t*>(&value)+(8-N)), N, counter.begin());
+#else
         std::copy_n(reinterpret_cast<uint8_t*>(&value), N, counter.begin());
+#endif
     }
 
     /// The current value of the n-byte integer
EOF
  1. $SOURCE_ROOT/couchbase/couchstore/tests/views/index_headers.cc
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/couchstore/tests/views/index_headers.cc
@@ -343,7 +343,7 @@
     size_t header_bin2_size = 0;
     char *header_bin3 = NULL;
     size_t header_bin3_size = 0;
-
+#if !__s390x__
     unsigned char header_bin[] = {
         5,226,251,160,170,107,207,39,248,218,139,62,137,58,95,46,204,10,12,1,0,64,0,
         254,1,0,218,1,0,0,136,5,1,4,0,136,254,127,0,218,127,0,8,0,83,119,9,1,254,128,
@@ -361,7 +361,28 @@
         0,254,154,0,46,154,0,112,1,0,4,0,5,0,10,0,60,0,62,0,2,0,11,0,15,0,2,0,58,0,
         61,0,2,0,15,0,58,105,173,44,0,0,4,197,0,63,0,0,0,0,4,197
     };
-
+#else
+    unsigned char header_bin[] = {
+        5,226,251,160,170,107,207,39,248,218,139,62,137,58,95,46,204,10,12,1,0,64,0,
+        254,1,0,218,1,0,0,136,5,1,4,0,136,254,127,0,218,127,0,8,0,83,119,9,1,254,128,
+        0,222,128,0,0,36,5,121,20,136,0,0,58,0,1,1,11,12,4,197,0,2,13,8,0,3,13,
+        8,0,4,13,8,0,5,13,8,0,6,13,8,0,7,13,8,0,8,13,8,0,9,13,8,0,10,
+        13,8,0,12,13,8,0,13,13,8,0,14,13,8,0,16,13,8,0,17,13,8,0,18,
+        13,8,0,19,13,8,0,20,13,8,0,21,13,8,0,22,13,8,0,23,13,8,0,24,
+        13,8,0,25,13,8,0,26,13,8,0,27,13,8,0,28,13,8,0,29,13,8,0,30,
+        13,8,0,31,13,8,0,32,13,8,0,33,13,8,0,34,13,8,0,35,13,8,37,19,
+        12,4,197,0,37,13,16,0,38,13,8,0,39,13,8,0,40,13,8,0,41,13,8,0,
+        42,13,8,0,43,13,8,0,44,13,8,0,45,13,8,0,46,13,8,0,47,13,8,0,
+        48,13,8,0,49,13,8,0,50,13,8,0,51,13,8,0,52,13,8,0,53,13,8,0,
+        54,13,8,0,55,13,8,0,56,13,8,0,57,13,8,0,59,13,8,0,60,13,8,0,
+        62,13,8,64,145,0,0,0,24,174,99,0,0,0,19,159,140,0,0,1,49,254,101,3,
+        226,101,3,0,255,13,1,32,2,0,152,0,0,0,44,71,93,1,148,4,152,106,254,148,
+        0,254,148,0,5,148,24,0,5,55,56,49,52,52,5,154,8,63,200,207,1,154,4,129,
+        243,254,154,0,254,154,0,46,154,0,112,1,0,4,0,5,0,10,0,60,0,62,0,2,
+        0,11,0,15,0,2,0,58,0,61,0,2,0,15,0,58,105,173,44,0,0,4,197,0,
+        63,0,0,0,0,4,197
+    };
+#endif
     fprintf(stderr, "Decoding an index header v1...\n");
     header = test_index_header_decoding_v1((const char*)header_bin, sizeof(header_bin));
 
@@ -395,7 +416,7 @@
     size_t header_bin2_size = 0;
     char *header_bin3 = NULL;
     size_t header_bin3_size = 0;
-
+#if !__s390x__
     unsigned char header_bin[] = {
         0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,237,14,12,2,0,32,0,254,1,0,234,1,0,12,1,32,0,
         24,254,127,0,234,127,0,16,0,144,0,64,34,254,128,0,238,128,0,12,0,0,32,4,61,
@@ -421,7 +442,38 @@
         92,62,36,0,0,27,58,36,0,129,120,62,36,0,0,28,54,36,0,133,148,62,36,0,0,29,58,
         36,0,129,176,62,36,0,0,30,58,36,0,129,204,62,36,0,0,31,58,36,0,0,193
     };
-
+#else
+    unsigned char header_bin[] = {
+        0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,237,14,12,2,0,32,0,254,
+        1,0,234,1,0,12,1,32,0,24,254,127,0,234,127,0,16,0,144,0,64,34,254,128,
+        0,238,128,0,12,0,0,32,4,61,130,0,1,5,140,8,1,0,2,5,8,8,4,0,
+        3,5,8,8,9,0,4,5,8,8,16,0,5,5,8,8,25,0,6,5,8,8,36,0,
+        7,5,8,8,49,0,8,5,8,8,64,0,9,5,8,8,81,0,10,5,8,8,100,0,
+        11,5,8,8,121,0,12,5,8,8,144,0,13,5,8,8,169,0,14,5,8,8,196,0,
+        15,5,8,8,225,0,16,1,8,12,1,0,0,17,5,8,8,33,0,18,5,8,8,68,
+        0,19,5,8,8,105,0,20,5,8,8,144,0,21,5,8,8,185,0,22,5,8,8,228,
+        0,23,1,8,4,2,17,41,196,12,2,64,0,25,5,16,8,113,0,26,5,8,8,164,
+        0,27,5,8,8,217,0,28,1,8,12,3,16,0,29,5,8,8,73,0,30,5,8,8,
+        132,0,31,5,8,0,193,9,112,4,0,123,1,14,32,2,55,114,101,100,118,97,108,2,
+        9,125,4,9,41,1,21,4,3,21,9,21,0,50,9,21,4,13,128,1,21,4,34,197,
+        9,21,52,51,0,0,1,0,5,0,2,0,14,0,28,0,1,37,62,16,1,0,28,0,
+        2,53,62,17,102,65,234,32,2,97,117,117,105,100,49,50,51,1,66,1,1,28,97,110,
+        111,116,104,101,114,49,1,12,5,1,0,1,66,36,0,0,1,58,36,0,33,160,62,72,
+        0,0,2,58,36,0,33,188,62,36,0,0,3,58,36,0,33,216,62,36,0,0,4,58,
+        36,0,33,244,62,36,0,0,5,58,36,0,65,16,62,36,0,0,6,58,36,0,65,44,
+        62,36,0,0,7,58,36,0,65,72,62,36,0,0,8,58,36,0,65,100,62,36,0,0,
+        9,58,36,0,65,128,62,36,0,0,10,58,36,0,65,156,62,36,0,0,11,58,36,0,
+        8,121,0,12,66,36,0,0,12,58,36,0,65,212,62,72,0,0,13,58,36,0,65,240,
+        62,36,0,0,14,58,36,0,8,196,0,15,66,36,0,0,15,58,36,0,97,40,62,72,
+        0,0,16,54,36,0,101,68,62,36,0,0,17,58,36,0,97,96,62,36,0,0,18,58,
+        36,0,97,124,62,36,0,0,19,58,36,0,97,152,62,36,0,0,20,58,36,0,97,180,
+        62,36,0,0,21,58,36,0,97,208,62,36,0,0,22,58,36,0,8,228,0,23,66,36,
+        0,0,23,54,36,0,0,2,129,8,62,72,0,0,24,58,36,0,129,36,62,36,0,0,
+        25,58,36,0,129,64,62,36,0,0,26,58,36,0,129,92,62,36,0,0,27,58,36,0,
+        129,120,62,36,0,0,28,54,36,0,133,148,62,36,0,0,29,58,36,0,129,176,62,36,
+        0,0,30,58,36,0,129,204,62,36,0,0,31,58,36,0,0,193
+    };
+#endif
     fprintf(stderr, "Decoding an index header v2...\n");
     header = test_index_header_decoding_v2(
         (const char*)header_bin, sizeof(header_bin));
EOF
  1. $SOURCE_ROOT/couchbase/couchstore/tests/views/spatial.cc
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/couchstore/tests/views/spatial.cc
@@ -280,6 +280,7 @@
     char encoded[66];
     double mbb[] = {6.3, 18.7};
     double mbb2[] = {1.0, 3.0, 30.33, 31.33, 15.4, 138.7, 7.8, 7.8};
+#if !__s390x__
     unsigned char expected[] = {
         0x00, 0x02, 0x33, 0x33, 0x33, 0x33, 0x33, 0x33, 0x19, 0x40,
         0x33, 0x33, 0x33, 0x33, 0x33, 0xb3, 0x32, 0x40
@@ -293,7 +294,22 @@
         0x33, 0x33, 0x33, 0x33, 0x33, 0x33, 0x1f, 0x40, 0x33, 0x33,
         0x33, 0x33, 0x33, 0x33, 0x1f, 0x40
     };
-
+#else
+    unsigned char expected[] = {
+        0x00, 0x02, 0x40, 0x19, 0x33, 0x33, 0x33, 0x33,
+        0x33, 0x33, 0x40, 0x32, 0xb3, 0x33, 0x33, 0x33, 0x33, 0x33,
+    };
+    unsigned char expected2[] = {
+        0x00, 0x08, 0x3f, 0xf0, 0x00, 0x00, 0x00, 0x00,
+        0x00, 0x00, 0x40, 0x08, 0x00, 0x00, 0x00, 0x00,
+        0x00, 0x00, 0x40, 0x3e, 0x54, 0x7a, 0xe1, 0x47,
+        0xae, 0x14, 0x40, 0x3f, 0x54, 0x7a, 0xe1, 0x47,
+        0xae, 0x14, 0x40, 0x2e, 0xcc, 0xcc, 0xcc, 0xcc,
+        0xcc, 0xcd, 0x40, 0x61, 0x56, 0x66, 0x66, 0x66,
+        0x66, 0x66, 0x40, 0x1f, 0x33, 0x33, 0x33, 0x33,
+        0x33, 0x33, 0x40, 0x1f, 0x33, 0x33, 0x33, 0x33, 0x33, 0x33
+    };
+#endif
     fprintf(stderr, "Running encode spatial key tests\n");
 
     mbb_struct.mbb = mbb;
@@ -325,6 +341,7 @@
 void test_decode_spatial_key()
 {
     sized_mbb_t decoded;
+#if !__s390x__
     unsigned char mbb[] = {
         0x00, 0x02, 0x33, 0x33, 0x33, 0x33, 0x33, 0x33, 0x19, 0x40,
         0x33, 0x33, 0x33, 0x33, 0x33, 0xb3, 0x32, 0x40
@@ -338,6 +355,24 @@
         0x33, 0x33, 0x33, 0x33, 0x33, 0x33, 0x1f, 0x40, 0x33, 0x33,
         0x33, 0x33, 0x33, 0x33, 0x1f, 0x40
     };
+#else
+    unsigned char mbb[] = {
+       0x00, 0x02, 0x40, 0x19, 0x33, 0x33, 0x33, 0x33,
+       0x33, 0x33, 0x40, 0x32, 0xb3, 0x33, 0x33, 0x33, 0x33, 0x33
+    };
+    unsigned char mbb2[] = {
+       0x00, 0x08, 0x3f, 0xf0, 0x00, 0x00, 0x00, 0x00,
+       0x00, 0x00, 0x40, 0x08, 0x00, 0x00, 0x00, 0x00,
+       0x00, 0x00, 0x40, 0x3e, 0x54, 0x7a, 0xe1, 0x47,
+       0xae, 0x14, 0x40, 0x3f, 0x54, 0x7a, 0xe1, 0x47,
+       0xae, 0x14, 0x40, 0x2e, 0xcc, 0xcc, 0xcc, 0xcc,
+       0xcc, 0xcd, 0x40, 0x61, 0x56, 0x66, 0x66, 0x66,
+       0x66, 0x66, 0x40, 0x1f, 0x33, 0x33, 0x33, 0x33,
+       0x33, 0x33, 0x40, 0x1f, 0x33, 0x33, 0x33, 0x33,
+       0x33, 0x33
+    };
+
+#endif
     double expected[] = {6.3, 18.7};
     double expected2[] = {1.0, 3.0, 30.33, 31.33, 15.4, 138.7, 7.8, 7.8};
EOF
  • Patch the following file manually as shown in the diff output
  1. $SOURCE_ROOT/couchbase/tlm/cmake/Modules/FindCouchbaseNlohmannJson.cmake
  • In the following, substitute the actual directory for <source_root>
@@ -3,7 +3,7 @@
 #
 #  NLOHMANN_JSON_INCLUDE_DIR

-SET(_nhlomann_json_exploded ${CMAKE_BINARY_DIR}/tlm/deps/json.exploded)
+SET(_nhlomann_json_exploded /<source_root>/json/)
  • Patch the following file using the in-line script as shown (Ubuntu 16.04 and 18.04 only)
  1. $SOURCE_ROOT/couchbase/tlm/cmake/Modules/FindCouchbaseLibevent.cmake
cd $SOURCE_ROOT
cat <<EOF | patch -p0
+++ couchbase/tlm/cmake/Modules/FindCouchbaseLibevent.cmake
@@ -44,7 +44,10 @@
     get_filename_component(${_dirname} ${resolved} DIRECTORY)
   endif ()
 endmacro(get_directory _dirname filename)
-
+set(LIBEVENT_INCLUDE_DIR /usr/include)
+set(LIBEVENT_EXTRA_LIB /usr/lib/s390x-linux-gnu/libevent_extra.so)
+set(LIBEVENT_CORE_LIB /usr/lib/s390x-linux-gnu/libevent_core.so)
+set(LIBEVENT_THREAD_LIB /usr/lib/s390x-linux-gnu/libevent_pthreads.so)
 set(_libevent_exploded ${CMAKE_BINARY_DIR}/tlm/deps/libevent.exploded)
 set(_libevent_library_dir ${CMAKE_INSTALL_PREFIX})

3.3) Replace Boltdb package

	cd $SOURCE_ROOT/couchbase/godeps/src/github.com/
	mv boltdb boltdb_ORIG
	mkdir boltdb
	cd boltdb
	git clone https://github.com/boltdb/bolt.git
	cd bolt/
	git checkout v1.3.0

3.4) Add s390x crc32 support

	cd $SOURCE_ROOT
	git clone https://github.com/linux-on-ibm-z/crc32-s390x.git
	cd crc32-s390x
	export CC=/usr/bin/gcc
  • Patch the following file using the in-line script as shown (RHEL and SLES only)

$SOURCE_ROOT/crc32-s390x/Makefile

bind 'set disable-completion on'
cd $SOURCE_ROOT
cat <<'EOF' | patch -p0
+++ crc32-s390x/Makefile
@@ -1,4 +1,4 @@
-CFLAGS=-Icrc32-vpmsum
+CFLAGS=-Icrc32-vpmsum -march=z13
 
 CRC32_POLY=0x04C11DB7
 CRC32C_POLY=0x1EDC6F41
@@ -22,10 +22,10 @@
 all: $(PROGS)
 
 crc32be-vx.o: crc32be-vx.S
-	$(CC) -c crc32be-vx.S
+	$(CC) -march=z13 -c crc32be-vx.S
 
 crc32le-vx.o: crc32le-vx.S
-	$(CC) -c crc32le-vx.S
+	$(CC) -march=z13 -c crc32le-vx.S
 
 libcrc32_s390x.a: crc32-s390x.o crc32be-vx.o crc32le-vx.o
 	ar rcs $@ $^ 
EOF

bind 'set disable-completion off'
cd $SOURCE_ROOT/crc32-s390x
make && sudo cp crc32-s390x.h /usr/local/include/
  • For RHEL
sudo cp libcrc32_s390x.a /usr/local/lib64/
  • For Ubuntu
sudo cp libcrc32_s390x.a /usr/local/lib/
  • For SLES
sudo cp libcrc32_s390x.a /usr/local/lib/
sudo cp libcrc32_s390x.a /usr/local/lib64/

3.5) Replace the sys package

cd $SOURCE_ROOT/couchbase/godeps/src/golang.org/x/
mv sys sys_ORIG
git clone https://github.com/golang/sys.git

3.6) Set environment variables

	export CB_MULTI_GO=0
  • Ubuntu 16.04 only
	export LD_LIBRARY_PATH=/usr/local/lib64/:/usr/lib:/usr/lib64:/usr/local/lib/
	export LD_LIBRARY_PATH=$SOURCE_ROOT/couchbase/build/kv_engine:$LD_LIBRARY_PATH

3.7) Install goyacc tool

	export GOPATH=$HOME/go
	go get -u golang.org/x/tools/cmd/goyacc
  • RHEL
	sudo ln -s $GOPATH/bin/goyacc /usr/lib/golang/pkg/tool/linux_s390x/
	sudo ln -sf /usr/lib/golang/pkg/tool/linux_s390x/goyacc /usr/lib/golang/pkg/tool/linux_s390x/yacc
	sudo chmod -f 755 /usr/lib/golang/pkg/tool/linux_s390x/goyacc
  • Ubuntu
	sudo cp $HOME/go/bin/goyacc /usr/lib/go-1.10/pkg/tool/linux_s390x/
	sudo chown root:root /usr/lib/go-1.10/pkg/tool/linux_s390x/goyacc
	sudo chmod -f 755 /usr/lib/go-1.10/pkg/tool/linux_s390x/goyacc
	sudo ln -sf /usr/lib/go-1.10/pkg/tool/linux_s390x/goyacc /usr/lib/go-1.10/pkg/tool/linux_s390x/yacc
  • SLES
	sudo cp $HOME/go/bin/goyacc  /usr/lib64/go/1.11/pkg/tool/linux_s390x/
	sudo chown root:root  /usr/lib64/go/1.11/pkg/tool/linux_s390x/goyacc
	sudo chmod -f 755  /usr/lib64/go/1.11/pkg/tool/linux_s390x/goyacc
	sudo ln -sf /usr/lib64/go/1.11/pkg/tool/linux_s390x/goyacc /usr/lib64/go/1.11/pkg/tool/linux_s390x/yacc

Step 4 : Build Couchbase

	cd $SOURCE_ROOT/couchbase/
	make

Step 5 : Run test cases

	sudo ldconfig
	cd $SOURCE_ROOT/couchbase/build/
	ctest

Note:The following errors occur exclusive to Z. These are only problems with the written test cases, and do not indicate problems in couch-base itself.

  • memcached-timestamp-test: fails if the user is in a timezone other than EDT.
  • The following tests fail because of incorrect copying of their results into the returned value. The results are copied as little-endian values into a big-endian 64-bit word.
	ep_testsuite.value_eviction.comp_off
	ep_testsuite.value_eviction.comp_passive
	ep_testsuite.value_eviction.comp_active
	ep_testsuite.full_eviction.comp_off
	ep_testsuite.full_eviction.comp_passive
	ep_testsuite.full_eviction.comp_active
	ep_testsuite.ephemeral.comp_off
	ep_testsuite.ephemeral.comp_passive
	ep_testsuite.ephemeral.comp_active
  • cbsasl-password-database fails on Ubuntu 18.04 as the cyrpto algorithm being tested -- HMAC (SHA512) -- is not distributed in Ubuntu 18.04.
  • memcached-auditconfig-test fails when json does not throw an exception when creating a string on a non-existant component.
  • memcached_testapp.default.TransportProtocols/LegacyUsersTest fails on Ubuntu 18.04, as SCRAM-SH1 is no longer distributed.
  • memcached_testapp.(default & ep).TransportProtocols/XattrTest fails because they compare a little-endian crc32c result to the system crc32c, which is big-endian.
  • Below test cases might fail in case IPv6 is not configured on SLES 12 SP4:
	memcached_testapp.default.TransportProtocols/ArithmeticTest
	memcached_testapp.default.TransportProtocols/ArithmeticXattrOnTest
	memcached_testapp.default.TransportProtocols/BucketTest
	memcached_testapp.default.TransportProtocols/ClusterConfigTest
	memcached_testapp.default.TransportProtocols/DcpTest
	memcached_testapp.default.TransportProtocols/ErrmapTest
	memcached_testapp.default.TransportProtocols/GetSetTest
	memcached_testapp.default.TransportProtocols/GetSetSnappyOnOffTest
	memcached_testapp.default.TransportProtocols/LockTest
	memcached_testapp.default.TransportProtocols/NoAutoselectDefaultBucketTest
	memcached_testapp.default.TransportProtocols/RemoveTest
	memcached_testapp.default.TransportProtocols/SaslTest
	memcached_testapp.default.TransportProtocols/StatsTest
	memcached_testapp.default.TransportProtocols/NoBucketStatsTest
	memcached_testapp.default.TransportProtocols/TouchTest
	memcached_testapp.default.TransportProtocols/TuneMcbpSla
	memcached_testapp.default.TransportProtocols/WithMetaTest
	memcached_testapp.default.TransportProtocols/XattrTest
	memcached_testapp.default.TransportProtocols/SaslauthdTest
	memcached_testapp.ep.TransportProtocols/ArithmeticTest
	memcached_testapp.ep.TransportProtocols/ArithmeticXattrOnTest
	memcached_testapp.ep.TransportProtocols/BucketTest
	memcached_testapp.ep.TransportProtocols/ClusterConfigTest
	memcached_testapp.ep.TransportProtocols/DcpTest
	memcached_testapp.ep.TransportProtocols/ErrmapTest
	memcached_testapp.ep.TransportProtocols/GetSetTest
	memcached_testapp.ep.TransportProtocols/GetSetSnappyOnOffTest
	memcached_testapp.ep.TransportProtocols/LockTest
	memcached_testapp.ep.TransportProtocols/MiscTest
	memcached_testapp.ep.TransportProtocols/NoAutoselectDefaultBucketTest
	memcached_testapp.ep.TransportProtocols/RemoveTest
	memcached_testapp.ep.TransportProtocols/SaslTest
	memcached_testapp.ep.TransportProtocols/StatsTest
	memcached_testapp.ep.TransportProtocols/NoBucketStatsTest
	memcached_testapp.ep.TransportProtocols/TouchTest
	memcached_testapp.ep.TransportProtocols/TuneMcbpSla
	memcached_testapp.ep.TransportProtocols/WithMetaTest
	memcached_testapp.ep.TransportProtocols/XattrTest
	memcached_testapp.ep.TransportProtocols/SaslauthdTest

Step 6 : Start the server

  • Use the below command to start Couchbase Server

    $SOURCE_ROOT/couchbase/install/bin/couchbase-server -- -noinput &

    You can view the Couchbase UI at http://hostname:8091

References:

http://www.couchbase.com/nosql-databases/couchbase-server

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Clone this wiki locally