Skip to content

Commit

Permalink
[SPARK-2561][SQL] Fix apply schema
Browse files Browse the repository at this point in the history
We need to use the analyzed attributes otherwise we end up with a tree that will never resolve.

Author: Michael Armbrust <michael@databricks.com>

Closes apache#1470 from marmbrus/fixApplySchema and squashes the following commits:

f968195 [Michael Armbrust] Use analyzed attributes when applying the schema.
4969015 [Michael Armbrust] Add test case.
  • Loading branch information
marmbrus committed Jul 22, 2014
1 parent a4d6020 commit 511a731
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -430,7 +430,7 @@ class SchemaRDD(
* @group schema
*/
private def applySchema(rdd: RDD[Row]): SchemaRDD = {
new SchemaRDD(sqlContext, SparkLogicalPlan(ExistingRdd(logicalPlan.output, rdd)))
new SchemaRDD(sqlContext, SparkLogicalPlan(ExistingRdd(queryExecution.analyzed.output, rdd)))
}

// =======================================================================
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,12 @@ class DslQuerySuite extends QueryTest {
testData.collect().toSeq)
}

test("repartition") {
checkAnswer(
testData.select('key).repartition(10).select('key),
testData.select('key).collect().toSeq)
}

test("agg") {
checkAnswer(
testData2.groupBy('a)('a, Sum('b)),
Expand Down

0 comments on commit 511a731

Please sign in to comment.