[SPARK-26818][ML] Make MLEvents JSON ser/de safe

[HyukjinKwon]

What changes were proposed in this pull request?

Currently, it looks it's not going to cause any virtually effective problem apparently (if I didn't misread the codes).

I see one place that JSON formatted events are being used.

spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala

Line 148 in ec506bd

val eventJson = JsonProtocol.sparkEventToJson(event)

It's okay because it just logs when the exception is ignorable

spark/core/src/main/scala/org/apache/spark/util/ListenerBus.scala

Line 111 in 9690eba

logError(s"Listener ${Utils.getFormattedClassName(listener)} threw an exception", e)

I guess it should be best to stay safe - I don't want this unstable experimental feature breaks anything in any case. It also disables logEvent in SparkListenerEvent for the same reason.

This is also to match SQL execution events side:

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLListener.scala

Lines 41 to 57 in ca545f7

	case class SparkListenerSQLExecutionEnd(executionId: Long, time: Long)
	extends SparkListenerEvent {
	// The name of the execution, e.g. `df.collect` will trigger a SQL execution with name "collect".
	@JsonIgnore private[sql] var executionName: Option[String] = None
	// The following 3 fields are only accessed when `executionName` is defined.
	// The duration of the SQL execution, in nanoseconds.
	@JsonIgnore private[sql] var duration: Long = 0L
	// The `QueryExecution` instance that represents the SQL execution
	@JsonIgnore private[sql] var qe: QueryExecution = null
	// The exception object that caused this execution to fail. None if the execution doesn't fail.
	@JsonIgnore private[sql] var executionFailure: Option[Exception] = None
	}

to make ML events JSON ser/de safe.

How was this patch tested?

Manually tested, and unit tests were added.

[HyukjinKwon]

cc @vanzin, @srowen, and @cloud-fan, would you mind if I ask to take a look please?

[HyukjinKwon]

Let me turn off this to be not logged by EventLoggingListener to be even more conservative.

[SparkQA]

Test build #102016 has finished for PR 23728 at commit 1687844.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class TransformStart() extends MLEvent
• case class TransformEnd() extends MLEvent
• case class FitStart[M <: Model[M]]() extends MLEvent
• case class FitEnd[M <: Model[M]]() extends MLEvent
• case class LoadInstanceStart[T](path: String) extends MLEvent
• case class LoadInstanceEnd[T]() extends MLEvent
• case class SaveInstanceStart(path: String) extends MLEvent
• case class SaveInstanceEnd(path: String) extends MLEvent

[SparkQA]

Test build #102018 has finished for PR 23728 at commit bd80031.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• sealed trait MLEvent extends SparkListenerEvent

[HyukjinKwon]

Thanks, @srowen and @viirya. This PR makes it more conservative and safer.

Merged to master.