Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-2393][SQL] Cost estimation optimization framework for Catalyst logical plans & sample usage. #1238

Closed
wants to merge 27 commits into from
Closed

[SPARK-2393][SQL] Cost estimation optimization framework for Catalyst logical plans & sample usage. #1238

wants to merge 27 commits into from

Commits on Jul 29, 2014

  1. Prototype impl of estimations for Catalyst logical plans. 

    - Also add simple size-getters for ParquetRelation and
  MetastoreRelation.
- Add a rule to auto-convert equi-joins to BroadcastHashJoin, if a table
  has smaller size, based on the above getter (for MetastoreRelation).
    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    56a8e6e View commit details
    Browse the repository at this point in the history

  2. Typo.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    5bf5586 View commit details
    Browse the repository at this point in the history

  3. Refactors. 

    - Remove BaseRelation from Catalyst and clean up related code (e.g.
  unmake SparkLogicalPlan a BaseRelation).
- Remove broadcastTables from SQLConf and clean up related code.
- Add EstimatesSuite.
- Address some review comments.
    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    84301a4 View commit details
    Browse the repository at this point in the history

  4. Cleanups.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    dcff9bd View commit details
    Browse the repository at this point in the history

  5. Add parquetAfter() properly in test.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    de3ae13 View commit details
    Browse the repository at this point in the history

  6. s/Estimates/Statistics, s/cardinality/numTuples.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    7a60ab7 View commit details
    Browse the repository at this point in the history

  7. Move SQLConf to Catalyst & add default val for sizeInBytes. 

    Conflicts:
	sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/SQLConf.scala
    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    73412be View commit details
    Browse the repository at this point in the history

  8. Move SQLConf back. Assign default sizeInBytes to SparkLogicalPlan.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    73cde01 View commit details
    Browse the repository at this point in the history

  9. Apply estimation to planning ShuffleHashJoin & BroadcastNestedLoopJoin.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    7d9216a View commit details
    Browse the repository at this point in the history

  10. Fix optimization conditions & update scala docs to explain.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    e5bcf5b View commit details
    Browse the repository at this point in the history

  11. Add comment.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    3ba8f3e View commit details
    Browse the repository at this point in the history

  12. Make Statistics a case class.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    4ef0d26 View commit details
    Browse the repository at this point in the history

  13. Use multiplication instead of sum for default estimates.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    0ef9e5b View commit details
    Browse the repository at this point in the history

  14. Revert optimization for BroadcastNestedLoopJoin (this fixes tests).

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    43d38a6 View commit details
    Browse the repository at this point in the history

  15. Inject SQLContext into SparkLogicalPlan, removing SQLConf mixin from it.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    ca5b825 View commit details
    Browse the repository at this point in the history

  16. {Cleanup, use synchronized in, enrich} StatisticsSuite.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    2d99eb5 View commit details
    Browse the repository at this point in the history

  17. Remove singleton SQLConf and move back settings to the trait.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    573e644 View commit details
    Browse the repository at this point in the history

  18. Update docs to be more explicit.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    729a8e2 View commit details
    Browse the repository at this point in the history

  19. Remove numTuples in Statistics for now.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    549061c View commit details
    Browse the repository at this point in the history

  20. Update scaladoc for a field and move it to @param section.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    01b7a3e View commit details
    Browse the repository at this point in the history

  21. Get size info from metastore for MetastoreRelation. 

    Additionally, remove size estimate from ParquetRelation since the Hadoop
FileSystem API calls can be expensive (e.g. S3FileSystem has a lot of
RPCs).
    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    6e594b8 View commit details
    Browse the repository at this point in the history

  22. Add a note on performance of statistics.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    8bd2816 View commit details
    Browse the repository at this point in the history

  23. Avoid calling statistics on plans if auto join conversion is disabled.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    16fc60a View commit details
    Browse the repository at this point in the history

  24. Remove childrenStats.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    9951305 View commit details
    Browse the repository at this point in the history

  25. Fix statistics for SparkLogicalPlan.

    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    2f2fb89 View commit details
    Browse the repository at this point in the history

  26. Use BigInt for stat; for logical leaves, by default throw an exception. 

    Also cleanups & scaladoc fixes per review comments.
    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    8663e84 View commit details
    Browse the repository at this point in the history

  27. Address review comments; turn config name from string to field in SQL… 

    …Conf.
    @concretevitamin
    concretevitamin committed Jul 29, 2014
    Configuration menu
    Copy the full SHA
    329071d View commit details
    Browse the repository at this point in the history