-
Notifications
You must be signed in to change notification settings - Fork 70
ClassNotFoundException: org.apache.spark.shuffle.rdma.RdmaShuffleManager #36
Comments
Hi, thanks for using SparkRDMA.
Thanks, |
So, BTW we're in progress of releasing Spark over UCX. It'll have all the functionality of SparkRDMA + better performance + support other transports (cuda, shared memory) and protocols. Keep in touch: https://github.com/openucx/sparkucx/ |
Oh,thank you very much. |
the spark-defaults.conf of both workers and master are
and I have placed
libdisni.so
in/usr/lib
.When I run TeraGen build from spark-terasort,I can run it with --master spark://master:7077 --deploy-mode client,which whole command is
but failed with ClassNotFoundException while using --master spark://master:7077 --deploy-mode cluster,which whole command is
the error info is
How should I fix it?
What's more, in client deployment , when I generate 50GB data using teragen and use raw spark with no spark-defaults.conf, the transfer speed between master and slave is about 270MB/s. However,when I change my spark-defaults.conf and replace spark.shuffle.manager to org.apache.spark.shuffle.rdma.RdmaShuffleManager, the speed is also 270MB/s.
Is this because I used hdfs storage but it has nothing to do with spark shuffle?
Can you recommed a workload for me to significantly improve completion time when using spark rdma?
Thanks a lot !
The text was updated successfully, but these errors were encountered: