[NSE-732] Support Map complex type in Shuffle

[zhixingheyi-tian]

What changes were proposed in this pull request?

• Implement Map type support in JNI buffer building
• Support infinite recursion in same complex types
• Support infinite recursion between different complex types

How was this patch tested?

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

[github-actions]

#732

[zhixingheyi-tian]

@zhouyuan
Have passed Jenkins SparkSQL uts and other workloads.

[zhouyuan]

@zhixingheyi-tian these configurations are not set in the new tests, we may lose some coverage, is this intended?

        .set("spark.oap.sql.columnar.rowtocolumnar", "false")
        .set("spark.oap.sql.columnar.columnartorow", "false")

[zhixingheyi-tian]

@zhixingheyi-tian these configurations are not set in the new tests, we may lose some coverage, is this intended?

        .set("spark.oap.sql.columnar.rowtocolumnar", "false")
        .set("spark.oap.sql.columnar.columnartorow", "false")

Because Native row=>column, column => row have not supported complex types enough. So used the non-native conversion.

[zhouyuan]

@zhixingheyi-tian these configurations are not set in the new tests, we may lose some coverage, is this intended?

        .set("spark.oap.sql.columnar.rowtocolumnar", "false")
        .set("spark.oap.sql.columnar.columnartorow", "false")

Because Native row=>column, column => row have not supported complex types enough. So used the non-native conversion.

can you also improve the missing build_check() in c2r & r2c?

[zhixingheyi-tian]

[zhixingheyi-tian]

@zhixingheyi-tian these configurations are not set in the new tests, we may lose some coverage, is this intended?

        .set("spark.oap.sql.columnar.rowtocolumnar", "false")
        .set("spark.oap.sql.columnar.columnartorow", "false")

Because Native row=>column, column => row have not supported complex types enough. So used the non-native conversion.

can you also improve the missing build_check() in c2r & r2c?

Yes. Done

[zhixingheyi-tian]

@zhouyuan
Also added Operator check in SQL executedPlan.

[zhouyuan]

note:

• columnar shuffle works fine with array/struct/map as payload
• columnar shuffle failed with array/struct/map as shuffle keys due to gandiva expr doesnot support these types (rare case)

[zhixingheyi-tian]

@zhouyuan
Added Partitioning keys check.
Also added Fall back UTs about complex type in Partitioning keys