[BUG] CI fails at with error NoSuchMethodError in many sbt test cases

[blrchen]

Willingness to contribute

No. I cannot contribute a bug fix at this time.

Feathr version

latest main branch

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 20.0):
• Python version:
• Spark version, if reporting runtime issue:

Describe the problem

After PR #632 is merged, CI starts failing with error NoSuchMethodError in many sbt test cases

Following is most recent failure link: https://github.com/feathr-ai/feathr/actions/runs/3080323962/jobs/4977528123

cc @windoze @YihuiGuo @xiaoyongzhu

Tracking information

There are many test cases failed, picking up a random failed test log as an example here:
[info] TestPostGenPruner:
[info] - testStandardizedKeyColumnsAndNonNullableFeatureColumns *** FAILED ***
[info] java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonParser.getReadCapabilities()Lcom/fasterxml/jackson/core/util/JacksonFeatureSet;
[info] at com.fasterxml.jackson.databind.DeserializationContext.(DeserializationContext.java:211)
[info] at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.(DefaultDeserializationContext.java:50)
[info] at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext$Impl.(DefaultDeserializationContext.java:391)
[info] at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext$Impl.createInstance(DefaultDeserializationContext.java:413)
[info] at com.fasterxml.jackson.databind.ObjectMapper.createDeserializationContext(ObjectMapper.java:4656)
[info] at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4585)
[info] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3548)
[info] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3516)
[info] at org.apache.spark.rdd.RDDOperationScope$.fromJson(RDDOperationScope.scala:86)
[info] at org.apache.spark.rdd.RDDOperationScope$.$anonfun$withScope$6(RDDOperationScope.scala:137)

Code to reproduce bug

No response

What component(s) does this bug affect?

• Python Client: This is the client users use to interact with most of our API. Mostly written in Python.
• Computation Engine: The computation engine that execute the actual feature join and generation work. Mostly in Scala and Spark.
• Feature Registry API: The frontend API layer supports SQL, Purview(Atlas) as storage. The API layer is in Python(FAST API)
• Feature Registry Web UI: The Web UI for feature registry. Written in React

[blrchen]

More details can be viewed at https://feathrai.slack.com/archives/C036HEUQM08/p1663206131660039?thread_ts=1662643787.795709&cid=C036HEUQM08

[Yuqing-cat]

Related PR merged.