Adding RESTCatalog based Spark Smoke Test

[haizhou-zhao]

Feature Request / Improvement

Current Spark smoke test runs on Hive Catalog, it would be nice if we run the same test cases on top of REST Catalog as well. Example: https://github.com/apache/iceberg/blob/main/spark/v3.5/spark-runtime/src/integration/java/org/apache/iceberg/spark/SmokeTest.java

Query engine

None

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

[haizhou-zhao]

cc: @danielcweeks

As discussed offline, I think the we can build on top of RCK classes from #10908 for REST Catalog based spark integration tests.

[haizhou-zhao]

Found several issues while attempting to enable integration test based on REST Catalog for Spark. Listing them below as I troubleshoot them one by one.

[haizhou-zhao]

#11103

[haizhou-zhao]

#11109

[haizhou-zhao]

#11154