Code

Patch for SPARK-942#50

Code

Closed

kellrott wants to merge 29 commits intoapache:masterapache/spark:masterfrom kellrott:iterator-to-diskkellrott/spark:iterator-to-diskCopy head branch name to clipboard

+226-53

Commits

Commits on Nov 13, 2013

• Changing CacheManager and BlockManager to pass iterators directly to the serializer when a 'DISK_ONLY' persist is called.

  Show description for efe1102
  kellrott

  committed
  efe1102

  Copy full SHA for efe1102

• Fixing MemoryStore, so that it converts incoming iterators to ArrayBuffer objects. This was previously done higher up the stack.

  kellrott

  committed
  cac1fad

  Copy full SHA for cac1fad

• Merge remote-tracking branch 'origin/master' into iterator-to-disk

  kellrott

  committed
  d32992f

  Copy full SHA for d32992f

Commits on Nov 14, 2013

• Adding unit test for straight to disk iterator methods.

  kellrott

  committed
  81d670c

  Copy full SHA for 81d670c

Commits on Nov 15, 2013

• Merge branch 'master' into iterator-to-disk

  kellrott

  committed
  f403826

  Copy full SHA for f403826

Commits on Nov 17, 2013

• Changing the JavaSerializer reset to occur every 1000 objects.

  kellrott

  committed
  5eb2b7e

  Copy full SHA for 5eb2b7e

• Adding some comments.

  kellrott

  committed
  44ec35a

  Copy full SHA for 44ec35a

Commits on Feb 4, 2014

• Merge branch 'master' into iterator-to-disk

  Show description for 56f71cd
  kellrott

  committed
  56f71cd

  Copy full SHA for 56f71cd

Commits on Feb 24, 2014

• Simplifying StorageLevel checks

  kellrott

  committed
  95c7f67

  Copy full SHA for 95c7f67

• Deleting temp output directory when done

  kellrott

  committed
  0e6f808

  Copy full SHA for 0e6f808

Commits on Feb 25, 2014

• Fixing dumb mistake ("||" instead of "&&")

  kellrott

  committed
  2eeda75

  Copy full SHA for 2eeda75

• Wrapping long line

  kellrott

  committed
  a6424ba

  Copy full SHA for a6424ba

• Added check to make sure that streamed-to-dist RDD actually returns good data in the LargeIteratorSuite

  kellrott

  committed
  9df0276

  Copy full SHA for 9df0276

• Removing un-needed semi-colons

  kellrott

  committed
  31fe08e

  Copy full SHA for 31fe08e

• Removing rouge space

  kellrott

  committed
  40fe1d7

  Copy full SHA for 40fe1d7

• Making the Java ObjectStreamSerializer reset rate configurable by the system variable 'spark.serializer.objectStreamReset', default is not 10000.

  kellrott

  committed
  00c98e0

  Copy full SHA for 00c98e0

• Merge branch 'master' into iterator-to-disk

  kellrott

  committed
  8644ee8

  Copy full SHA for 8644ee8

• Fixing the JavaSerializer to read from the SparkConf rather then the System property.

  kellrott

  committed
  656c33e

  Copy full SHA for 656c33e

• Adding second putValues to BlockStore interface that accepts an ArrayBuffer (rather then an Iterator).

  Show description for 0f28ec7
  kellrott

  committed
  0f28ec7

  Copy full SHA for 0f28ec7

• Wrapping a few long lines

  kellrott

  committed
  627a8b7

  Copy full SHA for 627a8b7

Commits on Feb 26, 2014

• Removing more un-needed array-buffer to iterator conversions

  kellrott

  committed
  c2fb430

  Copy full SHA for c2fb430

• Streamlined the LargeIteratorSuite unit test. It should now run in ~25 seconds. Confirmed that it still crashes an unpatched copy of Spark.

  kellrott

  committed
  16a4cea

  Copy full SHA for 16a4cea

Commits on Feb 27, 2014

• Moving the 'LargeIteratorSuite' to simply test persistance of iterators. It doesn't try to invoke a OOM error any more

  kellrott

  committed
  7ccc74b

  Copy full SHA for 7ccc74b

• Adding docs for spark.serializer.objectStreamReset configuration

  kellrott

  committed
  f70d069

  Copy full SHA for f70d069

• Refactoring the BlockManager to replace the Either[Either[A,B]] usage. Now using trait 'Values'. Also modified BlockStore.putBytes call to return PutResult, so that it behaves like putValues.

  kellrott

  committed
  2f684ea

  Copy full SHA for 2f684ea

• Merge branch 'iterator-to-disk' of github.com:kellrott/incubator-spark into iterator-to-disk

  Show description for 33ac390
  kellrott

  committed
  33ac390

  Copy full SHA for 33ac390

Commits on Feb 28, 2014

• Merge ../incubator-spark into iterator-to-disk

  kellrott

  committed
  8aa31cd

  Copy full SHA for 8aa31cd

Commits on Mar 4, 2014

• Fixing issues (formatting, variable names, etc.) from review comments

  kellrott

  committed
  60e0c57

  Copy full SHA for 60e0c57

• Fixing formatting issues.

  kellrott

  committed
  9ef7cb8

  Copy full SHA for 9ef7cb8