Downgrade Spark job runtime to use protobuf v2 to make maven jar work.

[blrchen]

Description

Currently there is a conflict on protobuf-java versions between job runtime (v3.19.4) and spark runtime (v2.6.1). This will This PR does following to downgrade protobuf usage scala from v3 to v2 to resolve version conflict.

• Regenerate java protobuf file FeatureValueOuterClass.java with v2 syntax
• Download runtime job to take protobuf v2 as dependency
• No changes on python client side, as v3 protobuf can decode data created with v2. This also ensures backward compatibility of python client.

On a side note, sbt assembly has a feature to shade/rename dependency, for example following code in build.sbt will rename com.google.protobuf to shade.protobuf,

  ShadeRule.rename("com.google.protobuf.**" -> "shade.protobuf.@1").inAll,

this can resolve the version conflict in fat jar, aka the blob jar published to Azure Storage. However, this can not resolve the issue for maven jar which does not contains any additional dependencies. Downgrade protobuf from v3 to v2 is still required to fix maven jar.

Resolves #616

How was this PR tested?

Following test has been done to verify latest python client can read feature data from online store that is wrotten by protobuf v2

• Test fat jar
  • Publish https://azurefeathrstorage.blob.core.windows.net/public/feathr-protobuf-v2.jar
  • Run materilization job with this jar and verify python client can get online data from redis successfully.
• Test thin jar locally
  • Build a thin jar (2.2mb) locally
  • Submit a local pyspark job and the java.lang.IllegalAccessError does not occur

spark-submit \
    --master local \
    --conf "spark.driver.extraClassPath=...,com.google.protobuf:protobuf-java:2.6.1
    --class com.linkedin.feathr.offline.job.FeatureGenJob \
    <feathr_root>/target/scala-2.12/feathr_2.12-0.7.2.jar \
    ...

• Test maven jar in Databricks, published v0.7.2 to maven and verified in this job https://adb-1996253548709298.18.azuredatabricks.net/?o=1996253548709298#job/1005975536002963/run/150911
• Test with CI, CI test of this PR and main branch both pass, this can prove latest python sdk can read data produced by either protobuf v2 and v3.

Does this PR introduce any user-facing changes?

• No. You can skip the rest of this section.
• Yes. Make sure to clarify your proposed changes.

[xiaoyongzhu]

Thanks Blair!