-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-3619] Part 2. Upgrade to Mesos 0.21 to work around MESOS-1688 #4361
Conversation
jongyoul
commented
Feb 4, 2015
- MESOS_NATIVE_LIBRARY become deprecated
- Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY
- MESOS_NATIVE_LIBRARY become deprecated - Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY
Test build #26739 has started for PR 4361 at commit
|
Test build #26739 has finished for PR 4361 at commit
|
Test PASSed. |
You also need to update the docs with this change. |
@tnachen I've already changed docs/running_on_mesos.md. Did you talk about "Known issues"? I think that doesn't need anymore. What do you think? |
- MESOS_NATIVE_LIBRARY become deprecated - Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY
- Removed 'Known issues' section
… SPARK-3619-1 * 'SPARK-3619-1' of https://github.com/jongyoul/spark: [SPARK-3619] Upgrade to Mesos 0.21 to work around MESOS-1688 - MESOS_NATIVE_LIBRARY become deprecated - Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY
Test build #27833 has started for PR 4361 at commit
|
Test build #27833 has finished for PR 4361 at commit
|
Test PASSed. |
I believe spark-env.sh.template in conf folder has a comment about Mesos native library. Can you change that as well? |
@@ -15,7 +15,7 @@ | |||
# - SPARK_PUBLIC_DNS, to set the public DNS name of the driver program | |||
# - SPARK_CLASSPATH, default classpath entries to append | |||
# - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data | |||
# - MESOS_NATIVE_LIBRARY, to point to your libmesos.so if you use Mesos |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tnachen I've found only this setting for mesos. Do you think there is another comment?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is it, thanks !
this PR LGTM.
@pwendell @srowen Review this PR, please. This is about SPARK-3619 which is already completed but has some missing documentation about ENV name. From mesos-0.21.0, MESOS_NATIVE_LIBRARY is deprecated and MESOS_NATIVE_JAVA_LIBRARY is introduced newly. we should fix a configuration on this. @tnachen completed reviews on this PR. |
OK, so, this change is appropriate for Mesos 0.21.0, but not perhaps earlier versions. So we would only merge into master for 1.4. I don't know Mesos but trust your reviews. I assume that removing the note about MESOS-1688 was intentional because it's no longer a problem with Mesos 0.21.0. |
@srowen Master and Branch-1.3 have a dependency of Mesos 0.21.0. Is it ok to merge it into branch-1.3? And MESOS-1688 is resolved by Mesos 0.21.0. So Spark isn't influenced by that issue anymore. So I removed that part. |
You're right, SPARK-3619 was merged into 1.3 too. This should follow as well. Will do. |
- MESOS_NATIVE_LIBRARY become deprecated - Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY Author: Jongyoul Lee <jongyoul@gmail.com> Closes #4361 from jongyoul/SPARK-3619-1 and squashes the following commits: f1ea91f [Jongyoul Lee] Merge branch 'SPARK-3619-1' of https://github.com/jongyoul/spark into SPARK-3619-1 a6a00c2 [Jongyoul Lee] [SPARK-3619] Upgrade to Mesos 0.21 to work around MESOS-1688 - Removed 'Known issues' section 2e15a21 [Jongyoul Lee] [SPARK-3619] Upgrade to Mesos 0.21 to work around MESOS-1688 - MESOS_NATIVE_LIBRARY become deprecated - Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY 0dace7b [Jongyoul Lee] [SPARK-3619] Upgrade to Mesos 0.21 to work around MESOS-1688 - MESOS_NATIVE_LIBRARY become deprecated - Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY