Remote Shuffle Implementation for Spark 3.3.1

.github/workflows/dev_cron.yml

@@ -25,19 +25,21 @@ on:
       - edited
       - synchronize
 
+permissions: read-all
+
 jobs:
   process:
     name: Process
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@ee0669bd1cc54295c223e0bb666b733df41de1c5 # v2.7.0
 
       - name: Comment Issues link
         if: |
           github.event_name == 'pull_request_target' &&
             (github.event.action == 'opened' ||
              github.event.action == 'edited')
-        uses: actions/github-script@v3
+        uses: actions/github-script@ffc2c79a5b2490bd33e0a41c1de74b877714d736 # v3.2.0
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           script: |
@@ -49,7 +51,7 @@ jobs:
           github.event_name == 'pull_request_target' &&
             (github.event.action == 'opened' ||
              github.event.action == 'edited')
-        uses: actions/github-script@v3
+        uses: actions/github-script@ffc2c79a5b2490bd33e0a41c1de74b877714d736 # v3.2.0
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           script: |

README.md

@@ -43,8 +43,8 @@ following configurations in spark-defaults.conf or Spark submit command line arg
 Note: For DAOS users, DAOS Hadoop/Java API jars should also be included in the classpath as we leverage DAOS Hadoop filesystem.
 
 ```
-    spark.executor.extraClassPath          $HOME/miniconda2/envs/oapenv/oap_jars/remote-shuffle-<version>.jar
-    spark.driver.extraClassPath            $HOME/miniconda2/envs/oapenv/oap_jars/remote-shuffle-<version>.jar
+    spark.executor.extraClassPath          $HOME/miniconda2/envs/oapenv/oap_jars/shuffle-hadoop-<version>.jar
+    spark.driver.extraClassPath            $HOME/miniconda2/envs/oapenv/oap_jars/shuffle-hadoop-<version>.jar
 ```
 
 Enable the remote shuffle manager and specify the Hadoop storage system URI holding shuffle data.

SECURITY.md

@@ -0,0 +1,12 @@
+# Security Policy
+
+## Report a Vulnerability
+
+Please report security issues or vulnerabilities to the [Intel® Security Center].
+
+For more information on how Intel® works to resolve security issues, see
+[Vulnerability Handling Guidelines].
+
+[Intel® Security Center]:https://www.intel.com/security
+
+[Vulnerability Handling Guidelines]:https://www.intel.com/content/www/us/en/security-center/vulnerability-handling-guidelines.html

docs/OAP-Developer-Guide.md

@@ -3,13 +3,13 @@
 This document contains the instructions & scripts on installing necessary dependencies and building OAP modules. 
 You can get more detailed information from OAP each module below.
 
-* [SQL Index and Data Source Cache](https://github.com/oap-project/sql-ds-cache/blob/v1.1.0-spark-3.0.0/docs/Developer-Guide.md)
-* [PMem Common](https://github.com/oap-project/pmem-common/tree/v1.1.0-spark-3.0.0)
-* [PMem Spill](https://github.com/oap-project/pmem-spill/tree/v1.1.0-spark-3.0.0)
-* [PMem Shuffle](https://github.com/oap-project/pmem-shuffle/tree/v1.1.0-spark-3.0.0#5-install-dependencies-for-pmem-shuffle)
-* [Remote Shuffle](https://github.com/oap-project/remote-shuffle/tree/v1.1.0-spark-3.0.0)
-* [OAP MLlib](https://github.com/oap-project/oap-mllib/tree/v1.1.0-spark-3.0.0)
-* [Native SQL Engine](https://github.com/oap-project/native-sql-engine/tree/v1.1.0-spark-3.0.0)
+* [SQL Index and Data Source Cache](https://github.com/oap-project/sql-ds-cache/blob/v1.2.0/docs/Developer-Guide.md)
+* [PMem Common](https://github.com/oap-project/pmem-common/tree/v1.2.0)
+* [PMem Spill](https://github.com/oap-project/pmem-spill/tree/v1.2.0)
+* [PMem Shuffle](https://github.com/oap-project/pmem-shuffle/tree/v1.2.0#5-install-dependencies-for-pmem-shuffle)
+* [Remote Shuffle](https://github.com/oap-project/remote-shuffle/tree/v1.2.0)
+* [OAP MLlib](https://github.com/oap-project/oap-mllib/tree/v1.2.0)
+* [Gazelle Plugin](https://github.com/oap-project/gazelle_plugin/tree/v1.2.0)
 
 ## Building OAP
 
@@ -22,45 +22,42 @@ We provide scripts to help automatically install dependencies required, please c
 # cd oap-tools
 # sh dev/install-compile-time-dependencies.sh
 ```
-*Note*: oap-tools tag version `v1.1.0-spark-3.0.0` corresponds to  all OAP modules' tag version `v1.1.0-spark-3.0.0`.
+*Note*: oap-tools tag version `v1.2.0` corresponds to  all OAP modules' tag version `v1.2.0`.
 
 Then the dependencies below will be installed:
 
-* [Cmake](https://help.directadmin.com/item.php?id=494)
+* [Cmake](https://cmake.org/install/)
 * [GCC > 7](https://gcc.gnu.org/wiki/InstallingGCC)
 * [Memkind](https://github.com/memkind/memkind/tree/v1.10.1)
 * [Vmemcache](https://github.com/pmem/vmemcache)
 * [HPNL](https://github.com/Intel-bigdata/HPNL)
 * [PMDK](https://github.com/pmem/pmdk)  
 * [OneAPI](https://software.intel.com/content/www/us/en/develop/tools/oneapi.html)
-* [Arrow](https://github.com/oap-project/arrow/tree/arrow-3.0.0-oap-1.1)
+* [Arrow](https://github.com/oap-project/arrow/tree/v4.0.0-oap-1.2.0)
 * [LLVM](https://llvm.org/) 
 
-Run the following command to learn more.
-
-```
-# sh dev/scripts/prepare_oap_env.sh --help
-```
-
-Run the following command to automatically install specific dependency such as Maven.
-
-```
-# sh dev/scripts/prepare_oap_env.sh --prepare_maven
-```
-
 - **Requirements for Shuffle Remote PMem Extension**  
 If enable Shuffle Remote PMem extension with RDMA, you can refer to [PMem Shuffle](https://github.com/oap-project/pmem-shuffle) to configure and validate RDMA in advance.
 
 ### Building
 
+#### Building OAP 
+
 OAP is built with [Apache Maven](http://maven.apache.org/) and Oracle Java 8.
 
-To build OAP package, run command below then you can find a tarball named `oap-$VERSION-bin-spark-$VERSION.tar.gz` under directory `$OAP_TOOLS_HOME/dev/release-package `.
+To build OAP package, run command below then you can find a tarball named `oap-$VERSION-*.tar.gz` under directory `$OAP_TOOLS_HOME/dev/release-package `, which contains all OAP module jars.
+Change to `root` user, run
+
 ```
-$ sh $OAP_TOOLS_HOME/dev/compile-oap.sh
+# cd oap-tools
+# sh dev/compile-oap.sh
 ```
 
-Building specified OAP Module, such as `sql-ds-cache`, run:
+#### Building OAP specific module 
+
+If you just want to build a specific OAP Module, such as `sql-ds-cache`, change to `root` user, then run:
+
 ```
-$ sh $OAP_TOOLS_HOME/dev/compile-oap.sh --sql-ds-cache
+# cd oap-tools
+# sh dev/compile-oap.sh --component=sql-ds-cache
 ```

docs/OAP-Installation-Guide.md

@@ -26,20 +26,19 @@ To test your installation,  run the command `conda list` in your terminal window
 ### Installing OAP
 
 Create a Conda environment and install OAP Conda package.
+
 ```bash
-$ conda create -n oapenv -y python=3.7
-$ conda activate oapenv
-$ conda install -c conda-forge -c intel -y oap=1.1.0
+$ conda create -n oapenv -c conda-forge -c intel -y oap=1.2.0
 ```
 
 Once finished steps above, you have completed OAP dependencies installation and OAP building, and will find built OAP jars under `$HOME/miniconda2/envs/oapenv/oap_jars`
 
 Dependencies below are required by OAP and all of them are included in OAP Conda package, they will be automatically installed in your cluster when you Conda install OAP. Ensure you have activated environment which you created in the previous steps.
 
-- [Arrow](https://github.com/Intel-bigdata/arrow)
+- [Arrow](https://github.com/oap-project/arrow/tree/v4.0.0-oap-1.2.0)
 - [Plasma](http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-object-store/)
-- [Memkind](https://anaconda.org/intel/memkind)
-- [Vmemcache](https://anaconda.org/intel/vmemcache)
+- [Memkind](https://github.com/memkind/memkind/tree/v1.10.1)
+- [Vmemcache](https://github.com/pmem/vmemcache.git)
 - [HPNL](https://anaconda.org/intel/hpnl)
 - [PMDK](https://github.com/pmem/pmdk)  
 - [OneAPI](https://software.intel.com/content/www/us/en/develop/tools/oneapi.html)

pom.xml

@@ -6,17 +6,17 @@
 
     <groupId>com.intel.oap</groupId>
     <artifactId>remote-shuffle-parent</artifactId>
-    <version>1.2.0</version>
+    <version>2.2.0-SNAPSHOT</version>
     <name>OAP Remote Shuffle Parent POM</name>
     <packaging>pom</packaging>
 
     <properties>
-	    <scala.version>2.12.10</scala.version>
-	    <scala.binary.version>2.12</scala.binary.version>
+        <scala.version>2.12.10</scala.version>
+        <scala.binary.version>2.12</scala.binary.version>
         <java.version>1.8</java.version>
         <maven.compiler.source>${java.version}</maven.compiler.source>
         <maven.compiler.target>${java.version}</maven.compiler.target>
-        <spark.version>3.0.0</spark.version>
+        <spark.version>3.3.3</spark.version>
     </properties>
 
     <build>
@@ -214,8 +214,9 @@
             </dependency>
             <dependency>
                 <groupId>org.apache.spark</groupId>
-                <artifactId>spark-core_2.12</artifactId>
+                <artifactId>spark-core_${scala.binary.version}</artifactId>
                 <version>${spark.version}</version>
+                <scope>provided</scope>
             </dependency>
             <dependency>
                 <groupId>junit</groupId>
@@ -225,13 +226,13 @@
             </dependency>
             <dependency>
                 <groupId>org.scalatest</groupId>
-                <artifactId>scalatest_2.12</artifactId>
-                <version>3.0.8</version>
+                <artifactId>scalatest_${scala.binary.version}</artifactId>
+                <version>3.2.3</version>
                 <scope>test</scope>
             </dependency>
             <dependency>
                 <groupId>org.apache.spark</groupId>
-                <artifactId>spark-core_2.12</artifactId>
+                <artifactId>spark-core_${scala.binary.version}</artifactId>
                 <version>${spark.version}</version>
                 <classifier>tests</classifier>
                 <type>test-jar</type>

shuffle-daos/README.md

@@ -1,6 +1,6 @@
 # Remote Shuffle Based on DAOS Object API
 
 A remote shuffle plugin based on DAOS Object API. You can find DAOS and DAOS Java Wrapper in https://github.com/daos-stack/daos and https://github.com/daos-stack/daos/tree/master/src/client/java.
-Thanks to DAOS, the plugin is espacially good for small shuffle block, such as around 200KB.
+Thanks to DAOS, the plugin is especially good for small shuffle block, such as around 200KB.
 
 See Shuffle DAOS related documentation in [Readme under project root](../README.md).

shuffle-daos/pom.xml

@@ -7,13 +7,14 @@
     <parent>
         <groupId>com.intel.oap</groupId>
         <artifactId>remote-shuffle-parent</artifactId>
-        <version>1.2.0</version>
+        <version>2.2.0-SNAPSHOT</version>
     </parent>
-    <artifactId>shuffle-daos</artifactId>
+    <artifactId>remote-shuffle-daos</artifactId>
     <name>OAP Remote Shuffle Based on DAOS Object API</name>
     <packaging>jar</packaging>
 
     <build>
+        <finalName>${project.artifactId}-${project.version}-with-spark-${spark.version}</finalName>
         <plugins>
             <plugin>
                 <groupId>org.codehaus.mojo</groupId>
@@ -114,6 +115,42 @@
                     <target>${java.version}</target>
                 </configuration>
             </plugin>
+            <plugin>
+                <artifactId>maven-shade-plugin</artifactId>
+                <executions>
+                    <execution>
+                        <id>shade-netty4</id>
+                        <phase>package</phase>
+                        <goals>
+                            <goal>shade</goal>
+                        </goals>
+                        <configuration>
+                            <shadedArtifactAttached>true</shadedArtifactAttached>
+                            <transformers>
+                                <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
+                                    <resource>META-INF/org.apache.hadoop.fs.FileSystem</resource>
+                                </transformer>
+                            </transformers>
+                            <relocations>
+                                <relocation>
+                                    <pattern>io.netty.buffer</pattern>
+                                    <shadedPattern>io.netty.buffershade4</shadedPattern>
+                                </relocation>
+                                <relocation>
+                                    <pattern>io.netty.util</pattern>
+                                    <shadedPattern>io.netty.utilshade4</shadedPattern>
+                                </relocation>
+                            </relocations>
+                            <artifactSet>
+                                <includes>
+                                    <include></include>
+                                </includes>
+                            </artifactSet>
+                            <finalName>${artifactId}-${version}-with-spark-${spark.version}-netty4-txf</finalName>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
             <plugin>
                 <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-surefire-plugin</artifactId>
@@ -200,12 +237,12 @@
         </dependency>
         <dependency>
             <groupId>org.apache.spark</groupId>
-            <artifactId>spark-core_2.12</artifactId>
+            <artifactId>spark-core_${scala.binary.version}</artifactId>
         </dependency>
         <dependency>
             <groupId>io.daos</groupId>
             <artifactId>daos-java</artifactId>
-            <version>1.2.2</version>
+            <version>2.4.1</version>
         </dependency>
         <dependency>
             <groupId>junit</groupId>
@@ -214,12 +251,12 @@
         </dependency>
         <dependency>
             <groupId>org.scalatest</groupId>
-            <artifactId>scalatest_2.12</artifactId>
+            <artifactId>scalatest_${scala.binary.version}</artifactId>
             <scope>test</scope>
         </dependency>
         <dependency>
             <groupId>org.apache.spark</groupId>
-            <artifactId>spark-core_2.12</artifactId>
+            <artifactId>spark-core_${scala.binary.version}</artifactId>
             <classifier>tests</classifier>
             <type>test-jar</type>
             <scope>test</scope>
@@ -241,4 +278,10 @@
         </dependency>
     </dependencies>
 
+    <repositories>
+        <repository>
+            <id>maven-snapshots</id>
+	    <url>http://oss.sonatype.org/content/repositories/snapshots</url>
+        </repository>
+    </repositories>
 </project>

shuffle-daos/src/main/java/org/apache/spark/shuffle/daos/BoundThreadExecutors.java

@@ -26,7 +26,10 @@
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import java.util.concurrent.*;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.Executor;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadFactory;
 import java.util.concurrent.atomic.AtomicInteger;
 
 /**