Skip to content

Commit

Permalink
[SPARK-47307][SQL][FOLLOWUP] Promote spark.sql.legacy.chunkBase64Stri…
Browse files Browse the repository at this point in the history
…ng.enabled from a legacy/internal config to a regular/public one

### What changes were proposed in this pull request?

+ Promote spark.sql.legacy.chunkBase64String.enabled from a legacy/internal config to a regular/public one.
+ Add test cases for unbase64

### Why are the changes needed?

Keep the same behavior as before. More details: apache#47303 (comment)

### Does this PR introduce _any_ user-facing change?

yes, revert behavior change introduced in apache#47303

### How was this patch tested?

existing unit test

### Was this patch authored or co-authored using generative AI tooling?

No

Closes apache#47410 from wForget/SPARK-47307_followup.

Lead-authored-by: wforget <643348094@qq.com>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
  • Loading branch information
wForget and yaooqinn committed Jul 19, 2024
1 parent 3f6e2d6 commit af5eb08
Show file tree
Hide file tree
Showing 3 changed files with 7 additions and 4 deletions.
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
Project [static_invoke(Base64.encode(cast(g#0 as binary), false)) AS base64(CAST(g AS BINARY))#0]
Project [static_invoke(Base64.encode(cast(g#0 as binary), true)) AS base64(CAST(g AS BINARY))#0]
+- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
Original file line number Diff line number Diff line change
Expand Up @@ -3525,14 +3525,13 @@ object SQLConf {
.booleanConf
.createWithDefault(false)

val CHUNK_BASE64_STRING_ENABLED = buildConf("spark.sql.legacy.chunkBase64String.enabled")
.internal()
val CHUNK_BASE64_STRING_ENABLED = buildConf("spark.sql.chunkBase64String.enabled")
.doc("Whether to truncate string generated by the `Base64` function. When true, base64" +
" strings generated by the base64 function are chunked into lines of at most 76" +
" characters. When false, the base64 strings are not chunked.")
.version("3.5.2")
.booleanConf
.createWithDefault(false)
.createWithDefault(true)

val ENABLE_DEFAULT_COLUMNS =
buildConf("spark.sql.defaultColumn.enabled")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -520,6 +520,10 @@ class StringExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
withSQLConf(SQLConf.CHUNK_BASE64_STRING_ENABLED.key -> "true") {
checkEvaluation(Base64(Literal(longString.getBytes)), chunkEncoded)
}

// check if unbase64 works well for chunked and non-chunked encoded strings
checkEvaluation(StringDecode(UnBase64(Literal(encoded)), Literal("utf-8")), longString)
checkEvaluation(StringDecode(UnBase64(Literal(chunkEncoded)), Literal("utf-8")), longString)
}

test("initcap unit test") {
Expand Down

0 comments on commit af5eb08

Please sign in to comment.