Skip to content

Latest commit

 

History

History
256 lines (164 loc) · 9.58 KB

README.md

File metadata and controls

256 lines (164 loc) · 9.58 KB
Raw

Sgx-Spark

This is Apache Spark with modifications to run security sensitive code inside Intel SGX enclaves. The implementation leverages sgx-lkl, a library OS that allows to run Java-based applications inside SGX enclaves.

Docker quick start

This guide shows how to run Sgx-Spark in a few simple steps using Docker. Most parts of the setup and deployment are wrapped within Docker containers. Compliation and deployment should thus be smooth.

Preparing the Sgx-Spark Docker environment

  • Clone this Sgx-Spark repository

  • Build the Sgx-Spark base image. The name of the resulting Docker image is sgxpsark. This process might take a while (30-60 mins):

      sgx-spark/dockerfiles$ docker build -t sgxspark .

  • Prepare the disk image that will be required by sgx-lkl. Due to restrictions of Docker, this step can currently not be implemented as part of the above Docker build process. Thus, this step is platform-dependent. The process has been successfully tested on Ubuntu 16.04 and Arch Linux:

      sgx-spark/lkl$ make prepare-image

  • Create a Docker network device that will be used for communication by the Docker containers. Note that by creating a user-defined network, Docker will create an embedded DNS server so that workers can find the Spark master by name.

      sgx-spark$ docker network create sgxsparknet

Running Sgx-Spark jobs using Docker

From within directory sgx-spark/dockerfiles, run the Sgx-Spark master node, the Sgx-Spark worker node, as well as the actual Sgx-Spark job as follows.

  • Run the Sgx-Spark master node:

      sgx-spark/dockerfiles$ docker run \
  --user user \
  --env-file $(pwd)/docker-env \
  --net sgxsparknet \
  --name sgxspark-docker-master \
  -p 7077:7077 \
  -p 8082:8082 \
  -ti sgxspark /sgx-spark/master.sh

  • Run the Sgx-Spark worker node:

      sgx-spark/dockerfiles$ docker run \
  --user user \
  --memory="4g" \
  --shm-size="8g" \
  --env-file $(pwd)/docker-env \
  --net sgxsparknet \
  --privileged \
  -v $(pwd)/../lkl:/spark-image:ro \
  -ti sgxspark /sgx-spark/worker-and-enclave.sh

  • Run the Sgx-Spark job as follows.

    As of writing, the three jobs below are known to be fully supported:

    • WordCount

        sgx-spark/dockerfiles$ docker run \
  --user user \
  --memory="4g" \
  --shm-size="8g" \
  --env-file $(pwd)/docker-env \
  --net sgxsparknet \
  --privileged \
  -v $(pwd)/../lkl:/spark-image:ro \
  -e SPARK_JOB_CLASS=org.apache.spark.examples.MyWordCount \
  -e SPARK_JOB_NAME=WordCount \
  -e SPARK_JOB_ARG0=README.md \
  -e SPARK_JOB_ARG1=output \
  -ti sgxspark /sgx-spark/driver-and-enclave.sh

    • KMeans

        sgx-spark/dockerfiles$ docker run \
  --user user \
  --memory="4g" \
  --shm-size="8g" \
  --env-file $(pwd)/docker-env \
  --net sgxsparknet \
  --privileged \
  -v $(pwd)/../lkl:/spark-image:ro \
  -e SPARK_JOB_CLASS=org.apache.spark.examples.mllib.KMeansExample \
  -e SPARK_JOB_NAME=KMeans \
  -e SPARK_JOB_ARG0=data/mllib/kmeans_data.txt \
  -ti sgxspark /sgx-spark/driver-and-enclave.sh

    • LineCount

        sgx-spark/dockerfiles$ docker run \
  --user user \
  --memory="4g" \
  --shm-size="8g" \
  --env-file $(pwd)/docker-env \
  --net sgxsparknet \
  --privileged \
  -v $(pwd)/../lkl:/spark-image:ro \
  -e SPARK_JOB_CLASS=org.apache.spark.examples.LineCount \
  -e SPARK_JOB_NAME=LineCount \
  -e SPARK_JOB_ARG0=SgxREADME.md \
  -ti sgxspark /sgx-spark/driver-and-enclave.sh

Native compilation, installation and deployment

To run Sgx-Spark natively, proceed as detailed in the following.

Install package dependencies

Install all required dependencies. For Ubuntu 16.04, these can be installed as follows:

$ sudo apt-get update
$ sudo apt-get install -y --no-install-recommends scala libtool autoconf curl xutils-dev git build-essential maven openjdk-8-jdk ssh bc python autogen wget autotools-dev sudo automake

Compile and install Google Protocol Buffer 2.5.0

Hadoop, and thus Spark, depends on Google Protocol Buffers (GPB) in version 2.5.0:

  • Make sure to uninstall any other versions of GPB

  • Install GPB v2.5.0. Instructions for Ubuntu 16.04 are as follows:

      $ cd /tmp
  /tmp$ wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
  /tmp$ tar xvf protobuf-2.5.0.tar.gz
  /tmp$ cd protobuf-2.5.0
  /tmp/protobuf-2.5.0$ ./autogen.sh && ./configure && make && sudo make install
  /tmp/protobuf-2.5.0$ sudo apt-get install -y --no-install-recommends libprotoc-dev

    Instructions for Arch Linux are available at https://stackoverflow.com/a/29799354/2273470.

Compile sgx-lkl

As Sgx-Spark uses sgx-lkl, the latter must have been downloaded and compiled successfully. As of writing (June 14, 2018), sgx-lkl should be compiled using branch cleanup-musl. Please follow the documentation of sgx-lkl and ensure that your installation of sgx-lkl executes simple Java applications successfully.

Compile Sgx-Spark

sgx-spark$ build/mvn -DskipTests package

  • As part of this compilation process, a modified Hadoop library has been compiled. Copy the Hadoop JAR file into the Sgx-Spark jars directory:

      sgx-spark$ cp hadoop-2.6.5-src/hadoop-common-project/hadoop-common/target/hadoop-common-2.6.5.jar assembly/target/scala-2.11/jars/

  • Sgx-Spark ships with a native C library (libringbuff.so) that enables shared-memory-based communication between two JVMs. Compile as follows:

      sgx-spark/C$ make install

Prepare the Sgx-Spark disk images that will be run using sgx-lkl

  • Adjust file spark-sgx/lkl/Makefile for your environment:

    Variable SGX_LKL must point to your sgx-lkl directory (see Prerequisites).

  • Build the Sgx-Spark disk image required for sgx-lkl:

      sgx-spark/lkl$ make clean all

Run Sgx-Spark using sgx-lkl

Finally, we are ready to run (i) the Sgx-Spark master node, (ii) the Sgx-Spark worker node, (iii) the worker's enclave, (iv) the Sgx-Spark client, and (v) the client's enclave. In the following commands, replace: <hostname> with the master node's actual hostname; <sgx-lkl> with the path to your sgx-lkl installation.

Note: After running each example, make sure to (i) restart all processes, (ii) delete all shared memory files in /dev/shm.

  • If you run all the nodes locally, you need to add the following line to variables.sh:

      export SPARK_LOCAL_IP=127.0.0.1

  • Run the Master node

      sgx-spark$ ./master.sh

  • Run the Worker node

      sgx-spark$ ./worker.sh

  • Run the enclave for the Worker node

      sgx-spark$ ./worker-enclave.sh

  • Run the enclave for the driver program. This is the process that will output the job results!

      sgx-spark$ ./driver-enclave.sh

  • Finally, submit a Spark job. The result will be output in the process we started just before.

    • WordCount

        sgx-spark$ ./submitwordcount.sh

    • KMeans

        sgx-spark$ ./submitkmeans.sh

    • LineCount

        sgx-spark$ ./submitlinecount.sh

Native execution of the same Spark installation

In order to run the above installation without SGX, start your environment as follows:

  • Start the Master node as above

  • Start the Worker node as above, but change environment variable SGX_ENABLED=true to SGX_ENABLED=false

  • Do not start the enclaves

  • Submit the Spark job as above, but change evironment variable SGX_ENABLED=true to SGX_ENABLED=false

Important developer notes

Code changes and recompilation

There are a few important things to keep in mind when developing Sgx-Spark:

  • Whenever you change parts of the code, obviously, you must recompile the Spark code

      sgx-spark$ mvn package -DskipTests

    There have been (not clearly definable) situations in which the above command did not compile all of the changed files. In this case, issue:

      sgx-spark$ mvn clean package -DskipTests

  • After making changes to the Sgx-Spark code and after compiling the Java/Scala code (see above), you always need to rebuild the lkl image that will be used by sgx-lkl:

      sgx-spark/lkl$ make clean all

  • If you changed parts of the Hadoop code (in directory hadoop-2.6.5-src), you will also need to copy the resulting *jar file:

      sgx-spark$ cp hadoop-2.6.5-src/hadoop-common-project/hadoop-common/target/hadoop-common-2.6.5.jar assembly/target/scala-2.11/jars/

  • Lastly, do not forget to remove all related shared memory files in /dev/shm/ before running your next experiment!

Running without sgx-lkl

Development with sgx-lkl can be tedious. For development purposes, a special flag allows to run the enclave-side of Sgx-Spark in a regular JVM rather than on top of sgx-lkl. To make use of this feature, run the enclave JVMs using scripts worker-enclave-nosgx.sh and driver-enclave-nosgx.sh.

Under the hood, these scripts set environment variable DEBUG_IS_ENCLAVE_REAL=false (defaults to true) and provide the JVM with a value for environment variable SGXLKL_SHMEM_FILE. Note that the value of SGXLKL_SHMEM_FILE must be the same as the one provided for the corresponding Worker (worker.sh) and Driver (driver.sh).