[SPARK-6144] [core] Fix addFile when source files are on "hdfs:" #4894
Conversation
|
@trystanleftwich I'm not trying to steal your thunder :-). It's just that this is pretty urgent given the release schedule. You'll be given full credit for the change. I tested this patch with both files and directories being distributed with |
|
Test build #28269 has started for PR 4894 at commit
|
| @@ -624,7 +624,8 @@ private[spark] object Utils extends Logging { | |||
| case _ => | |||
| val fs = getHadoopFileSystem(uri, hadoopConf) | |||
| val path = new Path(uri) | |||
| fetchHcfsFile(path, new File(targetDir, path.getName), fs, conf, hadoopConf, fileOverwrite) | |||
| fetchHcfsFile(path, targetDir, fs, conf, hadoopConf, fileOverwrite, | |||
| Some(filename)) | |||
There was a problem hiding this comment.
I think you can bump this to the previous line
There was a problem hiding this comment.
Oh actually this doesn't follow the convention for named params (filename = Some(filename)). I'll fix it, and in that case it doesn't fit in the previous line...
|
LGTM provided that tests pass |
|
Test build #28270 has started for PR 4894 at commit
|
|
Test build #28269 has finished for PR 4894 at commit
|
|
Test PASSed. |
|
I'm merging this into master and 1.3. Thanks @vanzin and @trystanleftwich for your fixes. |
The code failed in two modes: it complained when it tried to re-create a directory that already existed, and it was placing some files in the wrong parent directory. The patch fixes both issues. Author: Marcelo Vanzin <vanzin@cloudera.com> Author: trystanleftwich <trystan@atscale.com> Closes #4894 from vanzin/SPARK-6144 and squashes the following commits: 100b3a1 [Marcelo Vanzin] Style fix. 58266aa [Marcelo Vanzin] Fix fetchHcfs file for directories. 91733b7 [trystanleftwich] [SPARK-6144]When in cluster mode using ADD JAR with a hdfs:// sourced jar will fail (cherry picked from commit 3a35a0d) Signed-off-by: Andrew Or <andrew@databricks.com>
|
Test build #28270 has finished for PR 4894 at commit
|
|
Test PASSed. |
The code failed in two modes: it complained when it tried to re-create a directory that already existed, and it was placing some files in the wrong parent directory. The patch fixes both issues.