Skip to content

Commit

Permalink
[SPARK-44402][SQL][TESTS] Add tests for schema pruning in delta-based…
Browse files Browse the repository at this point in the history
… DELETEs

### What changes were proposed in this pull request?

This PR adds tests for schema pruning for delta-based DELETEs, similar to the ones we have for MERGE and UPDATE.

### Why are the changes needed?

These changes are needed to verify schema pruning works as expected.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

This PR comes with tests.

Closes apache#41975 from aokolnychyi/spark-44402.

Authored-by: aokolnychyi <aokolnychyi@apple.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
aokolnychyi authored and dongjoon-hyun committed Jul 13, 2023
1 parent efed395 commit af27400
Showing 1 changed file with 20 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

package org.apache.spark.sql.connector

import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.{AnalysisException, Row}

class DeltaBasedDeleteFromTableSuite extends DeleteFromTableSuiteBase {

Expand Down Expand Up @@ -59,4 +59,23 @@ class DeltaBasedDeleteFromTableSuite extends DeleteFromTableSuiteBase {
}
assert(exception.message.contains("Row ID attributes cannot be nullable"))
}

test("delete with schema pruning") {
createAndInitTable("pk INT NOT NULL, id INT, country STRING, dep STRING",
"""{ "pk": 1, "id": 1, "country": "uk", "dep": "hr" }
|{ "pk": 2, "id": 2, "country": "us", "dep": "software" }
|{ "pk": 3, "id": 3, "country": "canada", "dep": "hr" }
|""".stripMargin)

executeAndCheckScan(
s"DELETE FROM $tableNameAsString WHERE id <= 1",
// `pk` is used to encode deletes
// `id` is used in the condition
// `_partition` is used in the requested write distribution
expectedScanSchema = "pk INT, id INT, _partition STRING")

checkAnswer(
sql(s"SELECT * FROM $tableNameAsString"),
Row(2, 2, "us", "software") :: Row(3, 3, "canada", "hr") :: Nil)
}
}

0 comments on commit af27400

Please sign in to comment.