From 06899953d5d542ef5aa39f7dcc7330b74c7e41ae Mon Sep 17 00:00:00 2001 From: Evan Chan Date: Thu, 3 Apr 2014 15:42:36 -0700 Subject: [PATCH] CR from @aarondav - move config, clarify for standalone mode --- docs/configuration.md | 50 ++++++++++++++++++++++--------------------- 1 file changed, 26 insertions(+), 24 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index b907cd312d782..c0bee954bf144 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -333,6 +333,32 @@ Apart from these, the following properties are also available, and may be useful receives no heartbeats. + + spark.worker.cleanup.enabled + true + + Enable periodic cleanup of worker / application directories. Note that this only affects standalone + mode, as YARN works differently. + + + + spark.worker.cleanup.interval + 1800 (30 minutes) + + Controls the interval, in seconds, at which the worker cleans up old application work dirs + on the local machine. + + + + spark.worker.cleanup.appDataTtl + 7 * 24 * 3600 (7 days) + + The number of seconds to retain application work directories on each worker. This is a Time To Live + and should depend on the amount of available disk space you have. Application logs and jars are + downloaded to each application work dir. Over time, the work dirs can quickly fill up disk space, + especially if you run jobs very frequently. + + spark.akka.frameSize 10 @@ -586,30 +612,6 @@ Apart from these, the following properties are also available, and may be useful Number of cores to allocate for each task. - - spark.worker.cleanup.enabled - true - - Enable periodic cleanup of worker / application directories - - - - spark.worker.cleanup.interval - 1800 (30 minutes) - - Controls the interval, in seconds, at which the worker cleans up old application work dirs - on the local machine. - - - - spark.worker.cleanup.appDataTtl - 7 * 24 * 3600 (7 days) - - The number of seconds to retain application work directories on each worker. This is a Time To Live - and should depend on the amount of available disk space you have. Application logs and jars are - downloaded to each application work dir. Over time, the work dirs can quickly fill up disk space, - especially if you run jobs very frequently. - ## Viewing Spark Properties