Unify the logic for column pruning, projection, and filtering of table scans. #213
Conversation
|
@liancheng and @AndreSchumacher, please take a look. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
All automated tests passed. |
|
Two unused imports are left in |
| object ParquetScans extends Strategy { | ||
| def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match { | ||
| // TODO: need to support writing to other types of files. | ||
| case logical.WriteToFile(path, child) => |
There was a problem hiding this comment.
Seems a little confusing to have a write operation within a scan strategy here... Maybe we should move this into a separate strategy object similar to HiveStrategies.DataSinks?
|
LGTM, much cleaner :) |
…e scans for both Hive and Parquet relations. Fix tests now that we are doing a better job of column pruning.
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
All automated tests passed. |
|
Merged, thanks |
* use beta candidate * all backports added to build * move/add supervize with checkpointing test to hdfs * add kerberos args to supervise test * fix job watchers * add native blas * remove old supervise test
Using cascading delete
This removes duplicated logic, dead code and casting when planning parquet table scans and hive table scans.
Other changes:
WHERE falsefrom getting pushed into table scans, sinceHiveTableScan(reasonably) refuses to apply partition pruning predicates to non-partitioned tables.