feat: Use CometPlugin as main entrypoint

[andygrove]

Which issue does this PR close?

Closes #.

Rationale for this change

What changes are included in this PR?

How are these changes tested?

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 37.50000% with 5 lines in your changes missing coverage. Please review.

Project coverage is 34.25%. Comparing base (c9af3f4) to head (7befdd0).
Report is 3 commits behind head on main.

Files	Patch %	Lines
...park/src/main/scala/org/apache/spark/Plugins.scala	37.50%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #853      +/-   ##
============================================
+ Coverage     34.05%   34.25%   +0.20%     
- Complexity      879      888       +9     
============================================
  Files           112      112              
  Lines         42959    43016      +57     
  Branches       9488     9492       +4     
============================================
+ Hits          14629    14736     +107     
+ Misses        25330    25301      -29     
+ Partials       3000     2979      -21     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[comphead]

its probably would be good to log the overriden value for spark.sql.extensions ?

[comphead]

Great idea of having CometPlugin as main entry point

[edmondop]

I don't have an IDE opened now, but I think this code can be simplified using this overloaded method of get https://spark.apache.org/docs/3.2.1/api/java/org/apache/spark/SparkConf.html#get-java.lang.String-java.lang.String-

Suggested change

	if (sc.conf.contains(extensionKey)) {
	val extensions = sc.conf.get(extensionKey)
	if (!extensions.split(",").map(_.trim).contains(extensionClass)) {
	sc.conf.set(extensionKey, s"$extensions,$extensionClass")
	} else {
	sc.conf.set(extensionKey, extensionClass)
	}
	} else {
	sc.conf.set(extensionKey, extensionClass)
	}
	val extensions = sc.conf.get(extensionKey, "")
	if (!extensions.split(",").map(_.trim).contains(extensionClass)) {
	sc.conf.set(extensionKey, s"$extensions,$extensionClass")
	} else {
	sc.conf.set(extensionKey, extensionClass)
	}

[edmondop]

The else clause it's also suspicious, are we sure if there are multiple extensions with the key we want to override everything? I suspect the code doesn't need an else branch in reality

[andygrove]

Thanks for the review @edmondop. You are correct that the original code was not correct. I have added a unit test and fixed the issue. It is close to your suggestion.

[comphead]

lgtm thanks @andygrove
the formatting needs to be fixed