Encountered Hadoop version (3.2.1) conflict issue on AWS EMR-6.3.0

[haojinIntel]

For OAP-1.2, we target to support all features on cloud. We choose emr-6.3.0 and the versions of software are showed below:

We try to start spark-shell after adding extra configurations for native-sql-engine and we encounter the issue like:

Is the issued caused by incompatible hadoop version?

[haojinIntel]

@zhouyuan @weiting-chen @zhixingheyi-tian Please help to track the issue. Thanks!

[zhouyuan]

please try to re-build your package by -Phadoop-3.2

[weiting-chen]

Yes, I can confirm this issue is from hadoop version support.
I can pass this issue in AWS EMR6.3.0 by using hadoop.version 3.2.1 in pom.xml
Please notice "-Phadoop-3.2" is using hadoop 3.2.0, I am not sure it can pass in AWS EMR.

[weiting-chen]

After the compile has been passed, another issue is NoSuchMethodError when creating external tables.
Add screenshot in below:

[zhouyuan]

the CaseInsensitiveMap is a util class under spark catalyst
can you try to add spark-catalyst_2.12-3.1.1.jar to your classpath?
https://mvnrepository.com/artifact/org.apache.spark/spark-catalyst_2.12/3.1.1

[haojinIntel]

@zhouyuan @weiting-chen After I rebuild native-sql-engine by "mvn clean package -am -DskipTests -Dcpp_tests=OFF -Dbuild_arrow=OFF -Dstatic_arrow=OFF -Dbuild_protobuf=ON -Phadoop-3.2", we meet new issue like:

[weiting-chen]

Let's use Hadoop 3.2.1 and ignore the error in 3.2.0 since AWS EMR 6.3.0 supports 3.2.1.
I will try to build one version that can run on AWS EMR 6.3.0 first.
We can use this issue to track the status.

[weiting-chen]

Done to test TPC-DS on AWS EMR 6.3.0.

1. Pass all the 99 TPC-DS queries in both Datasource v1 and v2 API(turn of adaptive). There are some configurations that must be set for AWS EMR.
2. Fail to run TPC-DS when adaptive is enabled, the error message "NoSuchMethodError" shows in below screenshot: