Skip to content

Commit

Permalink
[SPARK-26409][SQL][TESTS] SQLConf should be serializable in test sess…
Browse files Browse the repository at this point in the history
…ions

## What changes were proposed in this pull request?

`SQLConf` is supposed to be serializable. However, currently it is not  serializable in `WithTestConf`. `WithTestConf` uses the method `overrideConfs` in closure, while the classes which implements it (`TestHiveSessionStateBuilder` and `TestSQLSessionStateBuilder`) are not serializable.

This PR is to use a local variable to fix it.

## How was this patch tested?

Add unit test.

Closes apache#23352 from gengliangwang/serializableSQLConf.

Authored-by: Gengliang Wang <gengliang.wang@databricks.com>
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
  • Loading branch information
gengliangwang authored and holdenk committed Jan 5, 2019
1 parent b011de3 commit f4e4c1a
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -309,13 +309,14 @@ private[sql] trait WithTestConf { self: BaseSessionStateBuilder =>
def overrideConfs: Map[String, String]

override protected lazy val conf: SQLConf = {
val overrideConfigurations = overrideConfs
val conf = parentState.map(_.conf.clone()).getOrElse {
new SQLConf {
clear()
override def clear(): Unit = {
super.clear()
// Make sure we start with the default test configs even after clear
overrideConfs.foreach { case (key, value) => setConfString(key, value) }
overrideConfigurations.foreach { case (key, value) => setConfString(key, value) }
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,9 @@ class SerializationSuite extends SparkFunSuite with SharedSQLContext {
val spark = SparkSession.builder.getOrCreate()
new JavaSerializer(new SparkConf()).newInstance().serialize(spark.sqlContext)
}

test("[SPARK-26409] SQLConf should be serializable") {
val spark = SparkSession.builder.getOrCreate()
new JavaSerializer(new SparkConf()).newInstance().serialize(spark.sessionState.conf)
}
}

0 comments on commit f4e4c1a

Please sign in to comment.