OAP-Installation-Guide.md

OAP Installation Guide

This document introduces how to install OAP and its dependencies on your cluster nodes by Conda. Follow steps below on every node of your cluster to set right environment for each machine.

Contents

• Prerequisites
• Installing OAP
• Configuration

Prerequisites

• OS Requirements
  We have tested OAP on Fedora 29, CentOS 7.6 (kernel 4.18.16) and Ubuntu 20.04 (kernel 5.4.0-65-generic).

• Conda Requirements
  Install Conda on your cluster nodes with below commands and follow the prompts on the installer screens.:

$ wget -c https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
$ chmod +x Miniconda2-latest-Linux-x86_64.sh 
$ bash Miniconda2-latest-Linux-x86_64.sh 

For changes to take effect, close and re-open your current shell. To test your installation, run the command conda list in your terminal window. A list of installed packages appears if it has been installed correctly.

Installing OAP

Create a Conda environment and install OAP Conda package.

$ conda create -n oapenv -c conda-forge -c intel -y oap=1.5.0.spark32

Once finished steps above, you have completed OAP dependencies installation and OAP building, and will find built OAP jars under $HOME/miniconda2/envs/oapenv/oap_jars

Configuration

Once finished steps above, make sure libraries installed by Conda can be linked by Spark, please add the following configuration settings to $SPARK_HOME/conf/spark-defaults.conf.

spark.executorEnv.LD_LIBRARY_PATH   $HOME/miniconda2/envs/oapenv/lib
spark.executor.extraLibraryPath     $HOME/miniconda2/envs/oapenv/lib
spark.driver.extraLibraryPath       $HOME/miniconda2/envs/oapenv/lib
spark.executorEnv.CC                $HOME/miniconda2/envs/oapenv/bin/gcc
spark.executor.extraClassPath       $HOME/miniconda2/envs/oapenv/oap_jars/$OAP_FEATURE.jar
spark.driver.extraClassPath         $HOME/miniconda2/envs/oapenv/oap_jars/$OAP_FEATURE.jar

Then you can follow the corresponding feature documents for more details to use them.