Getting lower RDMA perf that TCP/IP perf #21
Comments
|
Hi, sure. Would be glad to assist. Can you please tell your setup (number of nodes, hardware type, what workload are you running, spark/hibench config that you are using, etc). |
|
Hi Peter, I am using Hibench for running Tersort and Wordcont benchmarks. The setup is as follows: 10 nodes with 36 cores each @ 3Ghz and 384GB ram 100G ROCE over Mellanox CX4 cards I will upload the configs in a few. Thanks for the help! |
SPARK_MASTER_HOST=hpc-cluster-cn1-rdma
SPARK_MASTER_IP=192.18.18.1
SPARK_LOCAL_DIRS=/localvolume/hadoop/spark/tmp
SPARK_WORKER_DIR=/localvolume/hadoop/spark/worker
SPARK_LOCAL_DIRS=/localvolume/hadoop/spark/local
RDMA_INTERFACE=mlx5_0
--
HiBench spark.conf
# Spark home
hibench.spark.home $SPARK_HOME
# Spark master
# standalone mode: spark://xxx:7077
# YARN mode: yarn-client
hibench.spark.master spark://hpc-cluster-cn1-rdma:7077
#hibench.spark.master yarn
spark.shuffle.compress false
spark.shuffle.spill.compress false
spark.driver.extraClassPath /localvolume/nfs-share/hadoop-rdma/applications/mellanox/spark-2.4.0-bin-hadoop2.7/rdma/spark-rdma-3.1-for-spark-2.4.0-jar-with-dependencies.jar
spark.executor.extraClassPath /localvolume/nfs-share/hadoop-rdma/applications/mellanox/spark-2.4.0-bin-hadoop2.7/rdma/spark-rdma-3.1-for-spark-2.4.0-jar-with-dependencies.jar
spark.shuffle.manager org.apache.spark.shuffle.rdma.RdmaShuffleManager
spark.roce.enabled false
spark.ib.enabled false
spark.shuffle.rdma.device.name mlx5_0
spark.shuffle.rdma.pconn.enabled true
spark.broadcast.compress false spark.broadcast.checksum false
spark.locality.wait 0
#spark.shuffle.rdma.chunk.size 2097152 #spark.shuffle.rdma.memory.slab.size 134217728 # executor number and cores when running on Yarn hibench.yarn.executor.num 2
hibench.yarn.executor.cores 2
# executor and driver memory in standalone & YARN mode
spark.executor.memory 4g
spark.driver.memory 4g
--
[opc@hpc-cluster-cn1-rdma conf]$ cat hadoop.conf
# Hadoop home
hibench.hadoop.home $HADOOP_HOME # The path of hadoop executable
hibench.hadoop.executable ${hibench.hadoop.home}/bin/hadoop
# Hadoop configraution directory
hibench.hadoop.configure.dir ${hibench.hadoop.home}/etc/hadoop # The root HDFS path to store HiBench data hibench.hdfs.master hdfs://hpc-cluster-cn1-rdma:9000
# Hadoop release provider. Supported value: apache, cdh5, hdp
hibench.hadoop.release apache
-- |
|
Top of the hibench.conf |
|
@gilbertc777 thanks, what's the result you get for TCP and RDMA? |
|
ScalaSparkTerasort 2019-02-12 23:58:52 32000000000 112.184 285245667 28524566 ScalaSparkTerasort 2019-02-13 00:02:02 32000000000 122.759 260673351 26067335 The first test is running TCP, the secons is running with RDMA enabled. Note RDMA runtime is higher. |
|
I see.
|
|
Hi,I have the same problem.The configrure is almost same. The following is part of benchlog. Can you help me 19/07/11 14:55:11 INFO cluster.YarnClientSchedulerBackend: Stopped |
|
some of the yarn log： |
|
So these errors are during the shutdown phase. Does the job itself complete successfully? |
|
Does this mean complete successfully? |
|
Yes. so the job is finished. You can see job history by setting: |
|
In spark-defaults.conf, setting the below, there weren't logs in the directory. I run in yarn client mode. So do you have another way to check this issue? |
|
https://spark.apache.org/docs/latest/monitoring.html#viewing-after-the-fact |
|
Hi, Petro, I still have the problem. Can you help me analysis it？ Thanks. I use 2 arm machine node to run terasort with "huge" profile (about 29G data ). cpu info: Net card: Infiniband device 'mlx5_1' port 1 status: driver: mlx5_core OS: ubuntu 18.04 top of hibench.conf Mapper number in hadoop, partition number in Sparkhibench.default.map.parallelism 200 Reducer nubmer in hadoop, shuffle partition number in Sparkhibench.default.shuffle.parallelism 120 hadoop.conf Hadoop homehibench.hadoop.home /home/linux/xjm/spark_rdma/hadoop-2.7.4 The path of hadoop executablehibench.hadoop.executable ${hibench.hadoop.home}/bin/hadoop Hadoop configraution directoryhibench.hadoop.configure.dir ${hibench.hadoop.home}/etc/hadoop The root HDFS path to store HiBench datahibench.hdfs.master hdfs://wls-arm-cavium05:9000 Hadoop release provider. Supported value: apache, cdh5, hdphibench.hadoop.release apache The test result (the following is the duration time): The following are configuration files and executor log. Did my configuration have problems? stderr-excutor1-no-rdma.txt |
|
@jiameixie, so 2 is to few to get real advantage of rdma vs tcp. So try to increase number of executors + number of cores + workload size (at least gigantic or bigdata). |
Hi all,
I am running Hibench suite terasort against RDMA Spark. My current results show that TCP is performaing better than RDMA. Any thoughts on configs to look into? Happy to post config settings currently in place as well to help diagnose.
The text was updated successfully, but these errors were encountered: