Interface for per-file-parallelism with Hadoop FS

[rnowling]

For many of the use cases we've seen, parallelism per file versus per line is sufficient performance wise and makes parsing files easier. (e.g. JSON, CSV, etc. file formats which may require keeping track of state from the start of the file is problematic for chunking up files.)

We'd like to provide an API for per-file-parallelism. We'd like to reuse Hadoop's Filesystem interfaces which provides a unified interface to readers for a variety of sources including S3, FTP, HDFS, and POSIX. If using lazy evaluation is possible, we wouldn't have to keep large files in memory. We may also want a way to list and filter on the list of files. (E.g., more than just glob-type expressions.)

[rnowling]

Follow up idea courtesy of @willb: implement the API as a custom RDD and provide a sc.partitionPerFileRDD("~/input") method.

[rnowling]

The SparkContext provides a wholeTextFile() method which returns pairs of (filename, contents). For large files, this may kill performance so the idea of 1 partition per file may offer better granularity.