Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SQL] Decrease partitions when testing #2164

Closed
wants to merge 6 commits into from
Closed

[SQL] Decrease partitions when testing #2164

wants to merge 6 commits into from

Conversation

marmbrus
Copy link
Contributor

No description provided.

@marmbrus marmbrus mentioned this pull request Aug 27, 2014
Closed
rxin
rxin reviewed Aug 27, 2014
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/test/TestSQLContext.scala

/** Fewer partitions to speed up testing. */
override private[spark] def numShufflePartitions: Int =
getConf(SQLConf.SHUFFLE_PARTITIONS, "5").toInt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know... I was thinking a little more parallelism might be more likely to find bugs without incurring too much overhead. I could be convinced otherwise...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there isn't any parallelism really here since we run with "local", which is single threaded. increasing this from 2 to 5 simply breaks each dataset into more chunks, to be processed sequentially.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess parallelism isn't really what I meant, I'm thinking more about bugs that could be related to expecting data to be copartitioned when it actually isn't.

That said, perhaps the test should also run in local[2] or higher. We have found a couple of bugs after deploying that are the result of concurrency issues (scala reflection... i'm looking at you :P)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to run this using local[2]

@SparkQA
Copy link

SparkQA commented Aug 27, 2014

QA tests have started for PR 2164 at commit 50aca12.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 27, 2014

QA tests have finished for PR 2164 at commit 50aca12.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor Author

ok to test

@SparkQA
Copy link

SparkQA commented Aug 27, 2014

QA tests have started for PR 2164 at commit b035325.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 27, 2014

QA tests have finished for PR 2164 at commit b035325.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented Aug 28, 2014

Oops some parquet tests failed

@SparkQA
Copy link

SparkQA commented Sep 5, 2014

QA tests have started for PR 2164 at commit b035325.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Sep 6, 2014

QA tests have finished for PR 2164 at commit b035325.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented Sep 6, 2014

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Sep 6, 2014

QA tests have started for PR 2164 at commit b035325.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Sep 6, 2014

QA tests have finished for PR 2164 at commit b035325.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor Author

Jenkins, test this please

@SparkQA
Copy link

SparkQA commented Sep 10, 2014

QA tests have started for PR 2164 at commit ee687cd.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Sep 10, 2014

QA tests have finished for PR 2164 at commit ee687cd.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class Last(child: Expression) extends PartialAggregate with trees.UnaryNode[Expression]
    • case class LastFunction(expr: Expression, base: AggregateExpression) extends AggregateFunction
    • case class Abs(child: Expression) extends UnaryExpression

@SparkQA
Copy link

SparkQA commented Sep 10, 2014

QA tests have started for PR 2164 at commit ee687cd.

  • This patch does not merge cleanly!

@SparkQA
Copy link

SparkQA commented Sep 10, 2014

QA tests have started for PR 2164 at commit dc7cb6e.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Sep 10, 2014

QA tests have finished for PR 2164 at commit ee687cd.

  • This patch fails unit tests.
  • This patch does not merge cleanly!

@SparkQA
Copy link

SparkQA commented Sep 10, 2014

QA tests have finished for PR 2164 at commit dc7cb6e.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus marmbrus mentioned this pull request Sep 11, 2014
Closed
3 tasks
@SparkQA
Copy link

SparkQA commented Sep 13, 2014

QA tests have started for PR 2164 at commit 2dabae3.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Sep 13, 2014

QA tests have finished for PR 2164 at commit 2dabae3.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 13, 2014

QA tests have started for PR 2164 at commit 2dabae3.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Sep 13, 2014

QA tests have started for PR 2164 at commit 0da1e8c.

  • This patch merges cleanly.

@marmbrus marmbrus mentioned this pull request Sep 13, 2014
Closed
@SparkQA
Copy link

SparkQA commented Sep 13, 2014

QA tests have finished for PR 2164 at commit 2dabae3.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 13, 2014

QA tests have finished for PR 2164 at commit 0da1e8c.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor Author

I'm going to merge this to avoid more test timeouts.

@asfgit asfgit closed this in 0f8c4ed Sep 13, 2014
@marmbrus marmbrus deleted the shufflePartitions branch September 22, 2014 19:54
@scwf scwf mentioned this pull request Nov 8, 2014
Closed
liancheng pushed a commit to liancheng/spark that referenced this pull request Nov 9, 2014
@marmbrus @liancheng
[SQL] Decrease partitions when testing
3202a36 
Author: Michael Armbrust <michael@databricks.com>

Closes apache#2164 from marmbrus/shufflePartitions and squashes the following commits:

0da1e8c [Michael Armbrust] test hax
ef2d985 [Michael Armbrust] more test hacks.
2dabae3 [Michael Armbrust] more test fixes
0bdbf21 [Michael Armbrust] Make parquet tests less order dependent
b42eeab [Michael Armbrust] increase test parallelism
80453d5 [Michael Armbrust] Decrease partitions when testing
marmbrus pushed a commit to marmbrus/spark that referenced this pull request Nov 11, 2014
@liancheng @marmbrus
[SPARK-3971][SQL] Backport apache#2843 to branch-1.1
64945f8 
This PR backports apache#2843 to branch-1.1. The key difference is that this one doesn't support Hive 0.13.1 and thus always returns `0.12.0` when `spark.sql.hive.version` is queried.

6 other commits on which apache#2843 depends were also backported, they are:

- apache#2887 for `SessionState` lifecycle control
- apache#2675, apache#2823 & apache#3060 for major test suite refactoring and bug fixes
- apache#2164, for Parquet test suites updates
- apache#2493, for reading `spark.sql.*` configurations

Author: Cheng Lian <lian@databricks.com>
Author: Cheng Lian <lian.cs.zju@gmail.com>
Author: Michael Armbrust <michael@databricks.com>

Closes apache#3113 from liancheng/get-info-for-1.1 and squashes the following commits:

d354161 [Cheng Lian] Provides Spark and Hive version in HiveThriftServer2 for branch-1.1
0c2a244 [Michael Armbrust] [SPARK-3646][SQL] Copy SQL configuration from SparkConf when a SQLContext is created.
3202a36 [Michael Armbrust] [SQL] Decrease partitions when testing
7f395b7 [Cheng Lian] [SQL] Fixes race condition in CliSuite
0dd28ec [Cheng Lian] [SQL] Fixes the race condition that may cause test failure
5928b39 [Cheng Lian] [SPARK-3809][SQL] Fixes test suites in hive-thriftserver
faeca62 [Cheng Lian] [SPARK-4037][SQL] Removes the SessionState instance created in HiveThriftServer2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@marmbrus @SparkQA @rxin