Test spark 3.4.0 protobuf support, see details

run notebook(scala kernel) locally

we are using scala jupyter kernel Almond.sh to run spark locally. Assume we already have couriser installed locally (we should have if we correctly setup our dev environment when company onboarding), in case you don't have it, please follow the steps in details.

install Almond

$ cs launch --fork almond --scala 2.12 -- --install

build shaded uber jar

since there would be a protobuf-java conflict with spark itself, we need to shade com.google.protobuf.**, see guideline.

ThisBuild / assemblyShadeRules := Seq(
  ShadeRule.rename("com.google.protobuf.**" -> "org.sparkproject.spark_protobuf.protobuf.@1").inAll
)

publish to local repo

$ sbt publishLocal

play within notebook test.ipynb