Skip to content

Commit

Permalink
Give Databricks-specific messages when new SparkContext or SQLContext…
Browse files Browse the repository at this point in the history
… is created

## What changes were proposed in this pull request?
This commit hard-codes allowMultipleContexts to false and change the error message to a DBC-specific one. This is originally authored by yhuai.

Author: Yin Huai <yhuai@databricks.com>

Closes apache#43 from rxin/rxin-dbc-sparkcontext-msg.
  • Loading branch information
yhuai authored and rxin committed Jul 30, 2016
1 parent fecf8d0 commit 67d5114
Showing 1 changed file with 17 additions and 13 deletions.
30 changes: 17 additions & 13 deletions core/src/main/scala/org/apache/spark/SparkContext.scala
Original file line number Diff line number Diff line change
Expand Up @@ -2215,9 +2215,11 @@ object SparkContext extends Logging {
allowMultipleContexts: Boolean): Unit = {
SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
Option(activeContext.get()).filter(_ ne sc).foreach { ctx =>
val errMsg = "Only one SparkContext may be running in this JVM (see SPARK-2243)." +
" To ignore this error, set spark.driver.allowMultipleContexts = true. " +
s"The currently running SparkContext was created at:\n${ctx.creationSite.longForm}"
val ctx = activeContext.get()
val errMsg = "In Databricks, developers should utilize the shared SparkContext " +
"instead of creating one using the constructor. In Scala and Python notebooks, " +
"the shared context can be accessed as sc. When running a job, " +
"you can access the shared context by calling SparkContext.getOrCreate()."
val exception = new SparkException(errMsg)
if (allowMultipleContexts) {
logWarning("Multiple running SparkContexts detected in the same JVM!", exception)
Expand All @@ -2226,16 +2228,18 @@ object SparkContext extends Logging {
}
}

contextBeingConstructed.filter(_ ne sc).foreach { otherContext =>
// Since otherContext might point to a partially-constructed context, guard against
// its creationSite field being null:
val otherContextCreationSite =
Option(otherContext.creationSite).map(_.longForm).getOrElse("unknown location")
val warnMsg = "Another SparkContext is being constructed (or threw an exception in its" +
" constructor). This may indicate an error, since only one SparkContext may be" +
" running in this JVM (see SPARK-2243)." +
s" The other SparkContext was created at:\n$otherContextCreationSite"
logWarning(warnMsg)
contextBeingConstructed.foreach { otherContext =>
if (otherContext ne sc) { // checks for reference equality
// Since otherContext might point to a partially-constructed context, guard against
// its creationSite field being null:
val otherContextCreationSite =
Option(otherContext.creationSite).map(_.longForm).getOrElse("unknown location")
val warnMsg = "Another SparkContext is being constructed (or threw an exception in its" +
" constructor). This may indicate an error, since only one SparkContext may be" +
" running in this JVM (see SPARK-2243)." +
s" The other SparkContext was created at:\n$otherContextCreationSite"
logWarning(warnMsg)
}
}
}
}
Expand Down

0 comments on commit 67d5114

Please sign in to comment.