Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SPARK-1230: [WIP] Enable SparkContext.addJars() to load classes not in CLASSPATH #119

Closed
wants to merge 15 commits into from
Closed

SPARK-1230: [WIP] Enable SparkContext.addJars() to load classes not in CLASSPATH #119

wants to merge 15 commits into from

Commits on Dec 13, 2013

  1. Print out the class that was not found

    Evan Chan committed Dec 13, 2013
    Configuration menu
    Copy the full SHA
    8854dab View commit details
    Browse the repository at this point in the history

  2. Fix bug where classes in jars added via addJar() are not available to… 

    … the driver

Adds a driver-specific class loader that knows about the jars added via addJar(), and
initialize it first thing in SparkEnv, so that all other ThreadPool/ActorSystem
initialization will take advantage of the classloader, and user-added jars are
made available to all Spark subsystems.
    Evan Chan committed Dec 13, 2013
    Configuration menu
    Copy the full SHA
    ae1c199 View commit details
    Browse the repository at this point in the history

Commits on Dec 27, 2013

  1. Merge branch 'master' into debug-class-not-found 

    Conflicts:
	core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala
    Evan Chan committed Dec 27, 2013
    Configuration menu
    Copy the full SHA
    5701bf4 View commit details
    Browse the repository at this point in the history

Commits on Dec 29, 2013

  1. Move classLoader initialization to SparkContext 

    SparkEnv is used by Executor as well; we want this change to affect driver only.
    Evan Chan committed Dec 29, 2013
    Configuration menu
    Copy the full SHA
    d3df241 View commit details
    Browse the repository at this point in the history

  2. Pass the current class loader when creating Akka threadpool

    Evan Chan committed Dec 29, 2013
    Configuration menu
    Copy the full SHA
    5c24b0b View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2014

  1. CR from pwendell

    Evan Chan committed Jan 12, 2014
    Configuration menu
    Copy the full SHA
    cbabe80 View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2014

  1. Merge branch 'master' into debug-class-not-found 

    Conflicts:
	core/src/main/scala/org/apache/spark/SparkContext.scala
	core/src/main/scala/org/apache/spark/SparkEnv.scala
	core/src/main/scala/org/apache/spark/util/AkkaUtils.scala
    Evan Chan committed Jan 17, 2014
    Configuration menu
    Copy the full SHA
    b0cc61d View commit details
    Browse the repository at this point in the history

  2. CR: Rename ExecutorURLClassLoader -> SparkURLClassLoader

    Evan Chan committed Jan 17, 2014
    Configuration menu
    Copy the full SHA
    97a19b5 View commit details
    Browse the repository at this point in the history

  3. Use SparkURLClassLoader for driver only if spark.driver.add-dynamic-j… 

    …ars is set
    Evan Chan committed Jan 17, 2014
    Configuration menu
    Copy the full SHA
    9757c6f View commit details
    Browse the repository at this point in the history

  4. Document new config option and effect on addJar

    Evan Chan committed Jan 17, 2014
    Configuration menu
    Copy the full SHA
    3e52b0c View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2014

  1. Merge remote-tracking branch 'apache/master' into addJars 

    Conflicts:
	core/src/main/scala/org/apache/spark/SparkContext.scala
	core/src/main/scala/org/apache/spark/SparkEnv.scala
	core/src/main/scala/org/apache/spark/util/AkkaUtils.scala
    @pwendell
    pwendell committed Mar 11, 2014
    Configuration menu
    Copy the full SHA
    178083d View commit details
    Browse the repository at this point in the history

  2. Minor changes

    @pwendell
    pwendell committed Mar 11, 2014
    Configuration menu
    Copy the full SHA
    b132d7b View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2014

  1. Adding unit tests.

    @pwendell
    pwendell committed Mar 12, 2014
    Configuration menu
    Copy the full SHA
    95b24f2 View commit details
    Browse the repository at this point in the history

  2. Review feedback

    @pwendell
    pwendell committed Mar 12, 2014
    Configuration menu
    Copy the full SHA
    fbcb4a0 View commit details
    Browse the repository at this point in the history

  3. Style fixes

    @pwendell
    pwendell committed Mar 12, 2014
    Configuration menu
    Copy the full SHA
    9637d21 View commit details
    Browse the repository at this point in the history