[SPARK-5068] [SQL] Fix bug query data when path doesn't exist for HiveContext #4356
Conversation
|
Test build #26729 has started for PR 4356 at commit
|
|
Test build #26729 has finished for PR 4356 at commit
|
|
Test PASSed. |
| filteredFiles.mkString(",") | ||
| case None => path.toString | ||
| private def applyFilterIfNeeded(path: Path, filterOpt: Option[PathFilter]): Option[String] = { | ||
| if (fs.exists(path)) { |
There was a problem hiding this comment.
I think we'd better get fs from the path,because in the hadoop namenode federation we may get some problems like Wrong FS exception if we use the FileSystem.get(sc.hiveconf) to get fs.
|
Test build #26747 has started for PR 4356 at commit
|
|
Test build #26747 has finished for PR 4356 at commit
|
|
Test PASSed. |
| case None => path.toString | ||
| private def applyFilterIfNeeded(path: Path, filterOpt: Option[PathFilter]): Option[String] = { | ||
| val fs = path.getFileSystem(sc.hiveconf) | ||
| if (fs.exists(path)) { |
There was a problem hiding this comment.
My concern is similar to what @marmbrus mentioned in #3981. It's pretty expensive to check each path in serial for tables with lots of partitions. Especially when the data reside on S3. Can we use listStatus or globStatus to retrieve all FileStatus objects under some path(s), and then do the filtering locally?
|
Can you reconcile this with #5059 and if that looks good close this issue? If we decide to go with the other one it would be good to include your test cases if you think they are valuable. |
|
Sorry for the delay, I am closing it. |
…ontext This PR follow up PR #3907 & #3891 & #4356. According to marmbrus liancheng 's comments, I try to use fs.globStatus to retrieve all FileStatus objects under path(s), and then do the filtering locally. [1]. get pathPattern by path, and put it into pathPatternSet. (hdfs://cluster/user/demo/2016/08/12 -> hdfs://cluster/user/demo/*/*/*) [2]. retrieve all FileStatus objects ,and cache them by undating existPathSet. [3]. do the filtering locally [4]. if we have new pathPattern,do 1,2 step again. (external table maybe have more than one partition pathPattern) chenghao-intel jeanlyn Author: lazymam500 <lazyman500@gmail.com> Author: lazyman <lazyman500@gmail.com> Closes #5059 from lazyman500/SPARK-5068 and squashes the following commits: 5bfcbfd [lazyman] move spark.sql.hive.verifyPartitionPath to SQLConf,fix scala style e1d6386 [lazymam500] fix scala style f23133f [lazymam500] bug fix 47e0023 [lazymam500] fix scala style,add config flag,break the chaining 04c443c [lazyman] SPARK-5068: fix bug when partition path doesn't exists #2 41f60ce [lazymam500] Merge pull request #1 from apache/master
This is a follow up for #3907 & #3891 .
Hive actually support the not existed path(either table or partition path) by yielding an empty row, but Spark SQL will throws exception.
Ideally, we need to check the path existence during the partition processing, however, the
InputFormatalways computes the file splits before that, hence exception will raised if the specified path doesn't exists.This PR backs to the solution of #3891, and check the partition/table paths existence in spark plan generation. And of course we can move that logic into
HadoopRDDif it support the non exist path in the future.@jeanlyn, @marmbrus, @srowen