Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPY-317] Backports of SPARK-1582, SPARK-1601 and SPARK-1602 #6

Merged
merged 3 commits into from
Apr 24, 2014
Merged

[SPY-317] Backports of SPARK-1582, SPARK-1601 and SPARK-1602 #6

merged 3 commits into from
Apr 24, 2014

Commits on Apr 24, 2014

  1. SPARK-1601 & SPARK-1602: two bug fixes related to cancellation 

    This should go into 1.0 since it would return wrong data when the bug happens (which is pretty likely if cancellation is used). Test case attached.

1. Do not put partially executed partitions into cache (in task killing).

2. Iterator returned by CacheManager#getOrCompute was not an InterruptibleIterator, and was thus leading to uninterruptible jobs.

Thanks @aarondav and @ahirreddy for reporting and helping debug.

Author: Reynold Xin <rxin@apache.org>

Closes apache#521 from rxin/kill and squashes the following commits:

401033f [Reynold Xin] Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/spark into kill
7a7bdd2 [Reynold Xin] Add a new line in the end of JobCancellationSuite.scala.
35cd9f7 [Reynold Xin] Fixed a bug that partially executed partitions can be put into cache (in task killing).

Conflicts:
	core/src/main/scala/org/apache/spark/CacheManager.scala
	core/src/main/scala/org/apache/spark/executor/Executor.scala
	core/src/test/scala/org/apache/spark/JobCancellationSuite.scala
    @rxin @markhamstra
    rxin authored and markhamstra committed Apr 24, 2014
    Configuration menu
    Copy the full SHA
    e794d53 View commit details
    Browse the repository at this point in the history

  2. SPARK-1582 Invoke Thread.interrupt() when cancelling jobs 

    Sometimes executor threads are blocked waiting for IO or monitors, and the current implementation of job cancellation may never recover these threads. By simply invoking Thread.interrupt() during cancellation, we can often safely unblock the threads and use them for subsequent work.

Note that this feature must remain optional for now because of a bug in HDFS where Thread.interrupt() may cause nodes to be marked as permanently dead (as the InterruptedException is reinterpreted as an IOException during communication with some node).

Author: Aaron Davidson <aaron@databricks.com>

Closes apache#498 from aarondav/cancel and squashes the following commits:

e52b829 [Aaron Davidson] Don't use job.properties when null
82f78bb [Aaron Davidson] Update DAGSchedulerSuite
b67f472 [Aaron Davidson] Add comment on why interruptOnCancel is in setJobGroup
4cb9fd6 [Aaron Davidson] SPARK-1582 Invoke Thread.interrupt() when cancelling jobs

Conflicts:
	core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
	core/src/main/scala/org/apache/spark/executor/Executor.scala
	core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
	core/src/main/scala/org/apache/spark/scheduler/SchedulerBackend.scala
	core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
	core/src/main/scala/org/apache/spark/scheduler/local/LocalBackend.scala
	core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
    @aarondav @markhamstra
    aarondav authored and markhamstra committed Apr 24, 2014
    Configuration menu
    Copy the full SHA
    f96d67d View commit details
    Browse the repository at this point in the history

  3. Changes to accommodate backporting of SPARK-1582, SPARK-1601 and SPAR… 

    …K-1602
    @markhamstra
    markhamstra committed Apr 24, 2014
    Configuration menu
    Copy the full SHA
    f8a68d7 View commit details
    Browse the repository at this point in the history