From 91f1882b05c6d791a7ef9f6ba764e7055507eeb9 Mon Sep 17 00:00:00 2001 From: Amanda Liu Date: Sun, 30 Jun 2024 07:34:18 -0700 Subject: [PATCH 1/2] add create table behavior change to SQL migration docs --- docs/sql-migration-guide.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md index 8f6a415569863..76b6ca58985ca 100644 --- a/docs/sql-migration-guide.md +++ b/docs/sql-migration-guide.md @@ -97,6 +97,7 @@ license: | - Since Spark 3.4, `BinaryType` is not supported in CSV datasource. In Spark 3.3 or earlier, users can write binary columns in CSV datasource, but the output content in CSV files is `Object.toString()` which is meaningless; meanwhile, if users read CSV tables with binary columns, Spark will throw an `Unsupported type: binary` exception. - Since Spark 3.4, bloom filter joins are enabled by default. To restore the legacy behavior, set `spark.sql.optimizer.runtime.bloomFilter.enabled` to `false`. - Since Spark 3.4, when schema inference on external Parquet files, INT64 timestamps with annotation `isAdjustedToUTC=false` will be inferred as TimestampNTZ type instead of Timestamp type. To restore the legacy behavior, set `spark.sql.parquet.inferTimestampNTZ.enabled` to `false`. + - Since Spark 3.4, the behaviour for `CREATE TABLE AS SELECT ...` is changed from OVERWRITE to APPEND when `spark.sql.legacy.allowNonEmptyLocationInCTAS` is set to `true`. To restore the legacy behavior, set `spark.sql.legacy.allowNonEmptyLocationInCTAS` to `false`. ## Upgrading from Spark SQL 3.2 to 3.3 From f023b1e72bf32553b17f69fc34a1427410fe763e Mon Sep 17 00:00:00 2001 From: Amanda Liu Date: Mon, 1 Jul 2024 08:08:48 -0700 Subject: [PATCH 2/2] ask users to avoid CTAS with nonempty table location --- docs/sql-migration-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md index 76b6ca58985ca..4707e491fa674 100644 --- a/docs/sql-migration-guide.md +++ b/docs/sql-migration-guide.md @@ -97,7 +97,7 @@ license: | - Since Spark 3.4, `BinaryType` is not supported in CSV datasource. In Spark 3.3 or earlier, users can write binary columns in CSV datasource, but the output content in CSV files is `Object.toString()` which is meaningless; meanwhile, if users read CSV tables with binary columns, Spark will throw an `Unsupported type: binary` exception. - Since Spark 3.4, bloom filter joins are enabled by default. To restore the legacy behavior, set `spark.sql.optimizer.runtime.bloomFilter.enabled` to `false`. - Since Spark 3.4, when schema inference on external Parquet files, INT64 timestamps with annotation `isAdjustedToUTC=false` will be inferred as TimestampNTZ type instead of Timestamp type. To restore the legacy behavior, set `spark.sql.parquet.inferTimestampNTZ.enabled` to `false`. - - Since Spark 3.4, the behaviour for `CREATE TABLE AS SELECT ...` is changed from OVERWRITE to APPEND when `spark.sql.legacy.allowNonEmptyLocationInCTAS` is set to `true`. To restore the legacy behavior, set `spark.sql.legacy.allowNonEmptyLocationInCTAS` to `false`. + - Since Spark 3.4, the behavior for `CREATE TABLE AS SELECT ...` is changed from OVERWRITE to APPEND when `spark.sql.legacy.allowNonEmptyLocationInCTAS` is set to `true`. Users are recommended to avoid CTAS with a non-empty table location. ## Upgrading from Spark SQL 3.2 to 3.3