Skip to content
This repository has been archived by the owner on Dec 20, 2022. It is now read-only.

Commit

Permalink
Merge pull request #9 from Mellanox/release-2.0
Browse files Browse the repository at this point in the history
Release 2.0
  • Loading branch information
petro-rudenko committed Mar 30, 2018
2 parents d95ce3e + 090f075 commit 5519dd9
Show file tree
Hide file tree
Showing 32 changed files with 1,889 additions and 1,761 deletions.
29 changes: 15 additions & 14 deletions README.md
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -5,24 +5,24 @@ performing Shuffle data transfers in Spark jobs.
This open-source project is developed, maintained and supported by [Mellanox Technologies](http://www.mellanox.com).

## Performance results
Example performance speedup for HiBench TeraSort:
![Alt text](https://user-images.githubusercontent.com/20062725/28947340-30d45c6a-7864-11e7-96ea-ca3cf505ce7a.png)
Example performance speedup for [HiBench](https://github.com/intel-hadoop/HiBench) TeraSort:
![Alt text](https://user-images.githubusercontent.com/1121987/37930913-cdc3591c-314c-11e8-9794-d30db5900751.png)

Running TeraSort with SparkRDMA is x1.41 faster than standard Spark (runtime in seconds)
Running TeraSort with SparkRDMA is x1.53 faster than standard Spark (runtime in seconds)

Testbed:

175GB Workload

15 Workers, 2x Intel Xeon E5-2697 v3 @ 2.60GHz, 28 cores per Worker, 256GB RAM, non-flash storage (HDD)
16 Workers, 2x Intel Xeon E5-2697 v3 @ 2.60GHz, 30 cores per Worker, 256GB RAM, non-flash storage (HDD)

Mellanox ConnectX-4 network adapter with 100GbE RoCE fabric, connected with a Mellanox Spectrum switch

## Wiki pages
For more information on configuration, performance tuning and troubleshooting, please visit the [SparkRDMA GitHub Wiki](https://github.com/Mellanox/SparkRDMA/wiki)

## Runtime requirements
* Apache Spark 2.0.0/2.1.0/2.2.0
* Apache Spark 2.0.0/2.1.0/2.2.0/2.3.0
* Java 8
* An RDMA-supported network, e.g. RoCE or Infiniband

Expand All @@ -33,20 +33,21 @@ Please use the ["Releases"](https://github.com/Mellanox/SparkRDMA/releases) page
<br>If you would like to build the project yourself, please refer to the ["Build"](https://github.com/Mellanox/SparkRDMA#build) section below.

The pre-built binaries are packed as an archive that contains the following files:
* spark-rdma-1.0-for-spark-2.0.0-jar-with-dependencies.jar
* spark-rdma-1.0-for-spark-2.1.0-jar-with-dependencies.jar
* spark-rdma-1.0-for-spark-2.2.0-jar-with-dependencies.jar
* spark-rdma-2.0-for-spark-2.0.0-jar-with-dependencies.jar
* spark-rdma-2.0-for-spark-2.1.0-jar-with-dependencies.jar
* spark-rdma-2.0-for-spark-2.2.0-jar-with-dependencies.jar
* spark-rdma-2.0-for-spark-2.3.0-jar-with-dependencies.jar
* libdisni.so

libdisni.so **must** be installed on every Spark Master and Worker (usually in /usr/lib)
libdisni.so **must** be in `java.library.path` on every Spark Master and Worker (usually in /usr/lib)

### Configuration

Provide Spark the location of the SparkRDMA plugin jars by using the extraClassPath option. For standalone mode this can
be added to either spark-defaults.conf or any runtime configuration file. For client mode this **must** be added to spark-defaults.conf. For Spark 2.0.0 (Replace with 2.1.0 or 2.2.0 according to your Spark version):
be added to either spark-defaults.conf or any runtime configuration file. For client mode this **must** be added to spark-defaults.conf. For Spark 2.0.0 (Replace with 2.1.0, 2.2.0 or 2.3.0 according to your Spark version):
```
spark.driver.extraClassPath /path/to/SparkRDMA/target/spark-rdma-1.0-for-spark-2.0.0-jar-with-dependencies.jar
spark.executor.extraClassPath /path/to/SparkRDMA/target/spark-rdma-1.0-for-spark-2.0.0-jar-with-dependencies.jar
spark.driver.extraClassPath /path/to/SparkRDMA/target/spark-rdma-2.0-for-spark-2.0.0-jar-with-dependencies.jar
spark.executor.extraClassPath /path/to/SparkRDMA/target/spark-rdma-2.0-for-spark-2.0.0-jar-with-dependencies.jar
```

### Running
Expand All @@ -63,7 +64,7 @@ Building the SparkRDMA plugin requires [Apache Maven](http://maven.apache.org/)

1. Obtain a clone of [SparkRDMA](https://github.com/Mellanox/SparkRDMA)

2. Build the plugin for your Spark version (either 2.0.0, 2.1.0 or 2.2.0), e.g. for Spark 2.0.0:
2. Build the plugin for your Spark version (either 2.0.0, 2.1.0, 2.2.0 or 2.3.0), e.g. for Spark 2.0.0:
```
mvn -DskipTests clean package -Pspark-2.0.0
```
Expand All @@ -73,7 +74,7 @@ mvn -DskipTests clean package -Pspark-2.0.0
```
git clone https://github.com/zrlio/disni.git
cd disni
git checkout tags/v1.3 -b v1.3
git checkout tags/v1.4 -b v1.4
```

4. Compile and install only libdisni (the jars are already included in the SparkRDMA plugin):
Expand Down
163 changes: 163 additions & 0 deletions checkstyle.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->

<!DOCTYPE module PUBLIC
"-//Puppy Crawl//DTD Check Configuration 1.3//EN"
"http://www.puppycrawl.com/dtds/configuration_1_3.dtd">

<!--
Checkstyle configuration based on the Google coding conventions from:
- Google Java Style
https://google-styleguide.googlecode.com/svn-history/r130/trunk/javaguide.html
with Spark-specific changes from:
http://spark.apache.org/contributing.html#code-style-guide
Checkstyle is very configurable. Be sure to read the documentation at
http://checkstyle.sf.net (or in your downloaded distribution).
Most Checks are configurable, be sure to consult the documentation.
To completely disable a check, just comment it out or delete it from the file.
Authors: Max Vetrenko, Ruslan Diachenko, Roman Ivanov.
-->

<module name = "Checker">
<property name="charset" value="UTF-8"/>

<property name="severity" value="error"/>

<property name="fileExtensions" value="java, properties, xml"/>

<!-- Checks for whitespace -->
<!-- See http://checkstyle.sf.net/config_whitespace.html -->
<module name="FileTabCharacter">
<property name="eachLine" value="true"/>
</module>

<module name="RegexpSingleline">
<!-- \s matches whitespace character, $ matches end of line. -->
<property name="format" value="\s+$"/>
<property name="message" value="No trailing whitespace allowed."/>
</module>

<module name="NewlineAtEndOfFile">
<property name="lineSeparator" value="lf"/>
</module>
<module name="SuppressWarningsFilter" />

<module name="TreeWalker">
<module name="SuppressWarningsHolder" />
<module name="OuterTypeFilename"/>
<module name="IllegalTokenText">
<property name="tokens" value="STRING_LITERAL, CHAR_LITERAL"/>
<property name="format" value="\\u00(08|09|0(a|A)|0(c|C)|0(d|D)|22|27|5(C|c))|\\(0(10|11|12|14|15|42|47)|134)"/>
<property name="message" value="Avoid using corresponding octal or Unicode escape."/>
</module>
<module name="AvoidEscapedUnicodeCharacters">
<property name="allowEscapesForControlCharacters" value="true"/>
<property name="allowByTailComment" value="true"/>
<property name="allowNonPrintableEscapes" value="true"/>
</module>
<module name="LineLength">
<property name="max" value="100"/>
<property name="ignorePattern" value="^package.*|^import.*|a href|href|http://|https://|ftp://"/>
</module>
<module name="NoLineWrap"/>
<module name="EmptyBlock">
<property name="option" value="TEXT"/>
<property name="tokens" value="LITERAL_TRY, LITERAL_FINALLY, LITERAL_IF, LITERAL_ELSE, LITERAL_SWITCH"/>
</module>
<module name="NeedBraces">
<property name="allowSingleLineStatement" value="true"/>
</module>
<module name="OneStatementPerLine"/>
<module name="ArrayTypeStyle"/>
<module name="FallThrough"/>
<module name="UpperEll"/>
<module name="ModifierOrder"/>
<module name="SeparatorWrap">
<property name="tokens" value="DOT"/>
<property name="option" value="nl"/>
</module>
<module name="SeparatorWrap">
<property name="tokens" value="COMMA"/>
<property name="option" value="EOL"/>
</module>
<module name="PackageName">
<property name="format" value="^[a-z]+(\.[a-z][a-z0-9]*)*$"/>
<message key="name.invalidPattern"
value="Package name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="ClassTypeParameterName">
<property name="format" value="([A-Z][a-zA-Z0-9]*$)"/>
<message key="name.invalidPattern"
value="Class type name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="MethodTypeParameterName">
<property name="format" value="([A-Z][a-zA-Z0-9]*)"/>
<message key="name.invalidPattern"
value="Method type name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="GenericWhitespace">
<message key="ws.followed"
value="GenericWhitespace ''{0}'' is followed by whitespace."/>
<message key="ws.preceded"
value="GenericWhitespace ''{0}'' is preceded with whitespace."/>
<message key="ws.illegalFollow"
value="GenericWhitespace ''{0}'' should followed by whitespace."/>
<message key="ws.notPreceded"
value="GenericWhitespace ''{0}'' is not preceded with whitespace."/>
</module>

<module name="Indentation">
<property name="basicOffset" value="2"/>
<property name="braceAdjustment" value="0"/>
<property name="caseIndent" value="2"/>
<property name="throwsIndent" value="2"/>
<property name="lineWrappingIndentation" value="2"/>
<property name="arrayInitIndent" value="2"/>
</module>


<module name="MethodParamPad"/>
<module name="AnnotationLocation">
<property name="tokens" value="CLASS_DEF, INTERFACE_DEF, ENUM_DEF, METHOD_DEF, CTOR_DEF"/>
</module>
<module name="AnnotationLocation">
<property name="tokens" value="VARIABLE_DEF"/>
<property name="allowSamelineMultipleAnnotations" value="true"/>
</module>
<module name="MethodName">
<property name="format" value="^[a-z][a-z0-9][a-zA-Z0-9_]*$"/>
<message key="name.invalidPattern"
value="Method name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="EmptyCatchBlock">
<property name="exceptionVariableName" value="expected"/>
</module>
<module name="CommentsIndentation"/>
<module name="UnusedImports"/>
<module name="RedundantImport"/>
<module name="RedundantModifier"/>
</module>
</module>
61 changes: 58 additions & 3 deletions pom.xml
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

<groupId>com.github.mellanox</groupId>
<artifactId>spark-rdma</artifactId>
<version>1.0</version>
<version>2.0</version>
<name>${project.artifactId}</name>
<description>SparkRDMA Shuffle Manager Plugin</description>
<inceptionYear>2017</inceptionYear>
Expand Down Expand Up @@ -40,7 +40,7 @@
<profile>
<id>spark-2.0.0</id>
<properties>
<spark.version>2.0.0</spark.version>
<spark.version>2.0.1</spark.version>
</properties>
</profile>
<profile>
Expand All @@ -55,6 +55,12 @@
<spark.version>2.2.0</spark.version>
</properties>
</profile>
<profile>
<id>spark-2.3.0</id>
<properties>
<spark.version>2.3.0</spark.version>
</properties>
</profile>
</profiles>

<dependencies>
Expand All @@ -67,7 +73,7 @@
<dependency>
<groupId>com.ibm.disni</groupId>
<artifactId>disni</artifactId>
<version>1.3</version>
<version>1.4</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
Expand All @@ -84,6 +90,55 @@
<build>
<finalName>${project.artifactId}-${project.version}-for-spark-${spark.version}</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<id>validate</id>
<phase>validate</phase>
<configuration>
<configLocation>checkstyle.xml</configLocation>
<encoding>UTF-8</encoding>
<consoleOutput>true</consoleOutput>
<failsOnError>true</failsOnError>
<linkXRef>false</linkXRef>
<outputFile>${basedir}/target/checkstyle-output.xml</outputFile>
<inputEncoding>${project.build.sourceEncoding}</inputEncoding>
<outputEncoding>${project.reporting.outputEncoding}</outputEncoding>
</configuration>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.scalastyle</groupId>
<artifactId>scalastyle-maven-plugin</artifactId>
<version>1.0.0</version>
<configuration>
<verbose>false</verbose>
<failOnViolation>true</failOnViolation>
<includeTestSourceDirectory>true</includeTestSourceDirectory>
<failOnWarning>false</failOnWarning>
<sourceDirectory>${project.basedir}/src/main/scala</sourceDirectory>
<testSourceDirectory>${project.basedir}/src/test/scala</testSourceDirectory>
<configLocation>scalastyle_config.xml</configLocation>
<outputFile>${basedir}/target/scalastyle-output.xml</outputFile>
<outputEncoding>UTF-8</outputEncoding>
</configuration>
<executions>
<execution>
<id>validate</id>
<phase>validate</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
Expand Down
Loading

0 comments on commit 5519dd9

Please sign in to comment.