Skip to content

Commit

Permalink
[SPARK-13972][SQ] hive tests should fail if SQL generation failed
Browse files Browse the repository at this point in the history
## What changes were proposed in this pull request?

Now we should be able to convert all logical plans to SQL string, if they are parsed from hive query. This PR changes the error handling to throw exceptions instead of just log.

We will send new PRs for spotted bugs, and merge this one after all bugs are fixed.

## How was this patch tested?

existing tests.

Author: Wenchen Fan <wenchen@databricks.com>

Closes apache#11782 from cloud-fan/test.
  • Loading branch information
cloud-fan authored and roygao94 committed Mar 22, 2016
1 parent 65de1c4 commit 8a1cf25
Showing 1 changed file with 28 additions and 43 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -132,25 +132,7 @@ abstract class HiveComparisonTest
new java.math.BigInteger(1, digest.digest).toString(16)
}

/** Used for testing [[SQLBuilder]] */
private var numConvertibleQueries: Int = 0
private var numTotalQueries: Int = 0

override protected def afterAll(): Unit = {
logInfo({
val percentage = if (numTotalQueries > 0) {
numConvertibleQueries.toDouble / numTotalQueries * 100
} else {
0D
}

s"""SQLBuilder statistics:
|- Total query number: $numTotalQueries
|- Number of convertible queries: $numConvertibleQueries
|- Percentage of convertible queries: $percentage%
""".stripMargin
})

try {
TestHive.reset()
} finally {
Expand Down Expand Up @@ -412,32 +394,35 @@ abstract class HiveComparisonTest
if (containsCommands) {
originalQuery
} else {
numTotalQueries += 1
val convertedSQL = try {
new SQLBuilder(originalQuery.analyzed, TestHive).toSQL
} catch {
case NonFatal(e) => fail(
s"""Cannot convert the following HiveQL query plan back to SQL query string:
|
|# Original HiveQL query string:
|$queryString
|
|# Resolved query plan:
|${originalQuery.analyzed.treeString}
""".stripMargin, e)
}

try {
val sql = new SQLBuilder(originalQuery.analyzed, TestHive).toSQL
numConvertibleQueries += 1
logInfo(
s"""
|### Running SQL generation round-trip test {{{
|${originalQuery.analyzed.treeString}
|Original SQL:
|$queryString
|
|Generated SQL:
|$sql
|}}}
""".stripMargin.trim)
new TestHive.QueryExecution(sql)
} catch { case NonFatal(e) =>
logInfo(
s"""
|### Cannot convert the following logical plan back to SQL {{{
|${originalQuery.analyzed.treeString}
|Original SQL:
|$queryString
|}}}
""".stripMargin.trim)
originalQuery
new TestHive.QueryExecution(convertedSQL)
} catch {
case NonFatal(e) => fail(
s"""Failed to analyze the converted SQL string:
|
|# Original HiveQL query string:
|$queryString
|
|# Resolved query plan:
|${originalQuery.analyzed.treeString}
|
|# Converted SQL query string:
|$convertedSQL
""".stripMargin, e)
}
}
}
Expand Down

0 comments on commit 8a1cf25

Please sign in to comment.