-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SPARK-1154: Clean up app folders in worker nodes #288
Changes from all commits
b92752b
e3c408e
dc1a311
ad99955
72f7d2d
cb52f2b
8dc9cb5
553d8c2
f2f6027
9f10d96
0689995
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -536,6 +536,21 @@ private[spark] object Utils extends Logging { | |
} | ||
} | ||
|
||
/** | ||
* Finds all the files in a directory whose last modified time is older than cutoff seconds. | ||
* @param dir must be the path to a directory, or IllegalArgumentException is thrown | ||
* @param cutoff measured in seconds. Files older than this are returned. | ||
*/ | ||
def findOldFiles(dir: File, cutoff: Long): Seq[File] = { | ||
val currentTimeMillis = System.currentTimeMillis | ||
if (dir.isDirectory) { | ||
val files = listFilesSafely(dir) | ||
files.filter { file => file.lastModified < (currentTimeMillis - cutoff * 1000) } | ||
} else { | ||
throw new IllegalArgumentException(dir + " is not a directory!") | ||
} | ||
} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there a reason for not making this There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Easier to test for sure. I just prefer smaller more functional pieces of code in general for easier testing and to limit side effects. |
||
|
||
/** | ||
* Convert a Java memory parameter passed to -Xmx (such as 300m or 1g) to a number of megabytes. | ||
*/ | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -19,7 +19,7 @@ package org.apache.spark.util | |
|
||
import scala.util.Random | ||
|
||
import java.io.{ByteArrayOutputStream, ByteArrayInputStream, FileOutputStream} | ||
import java.io.{File, ByteArrayOutputStream, ByteArrayInputStream, FileOutputStream} | ||
import java.nio.{ByteBuffer, ByteOrder} | ||
|
||
import com.google.common.base.Charsets | ||
|
@@ -154,5 +154,18 @@ class UtilsSuite extends FunSuite { | |
val iterator = Iterator.range(0, 5) | ||
assert(Utils.getIteratorSize(iterator) === 5L) | ||
} | ||
|
||
test("findOldFiles") { | ||
// create some temporary directories and files | ||
val parent: File = Utils.createTempDir() | ||
val child1: File = Utils.createTempDir(parent.getCanonicalPath) // The parent directory has two child directories | ||
val child2: File = Utils.createTempDir(parent.getCanonicalPath) | ||
// set the last modified time of child1 to 10 secs old | ||
child1.setLastModified(System.currentTimeMillis() - (1000 * 10)) | ||
|
||
val result = Utils.findOldFiles(parent, 5) // find files older than 5 secs | ||
assert(result.size.equals(1)) | ||
assert(result(0).getCanonicalPath.equals(child1.getCanonicalPath)) | ||
} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It would be good to also test the actual "delete recursively" part. Right now if the deleting part fails for some reason this test doesn't catch it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That's true. We could add a unit test for deleteRecursively. Actually, we noticed there isn't a test for Worker in general. |
||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't need to rev this again just for this - but is there supposed to be one space or two spaces after the parameter name? It's inconsistent here.