Add flags for Iceberg and Lake Formation and Security Lake as a data source type.

[asuresh8]

Description

Previously, Iceberg catalog was set as the default catalog. This poses problems as the behavior to fall back to default Spark catalog is only correct in some versions of Iceberg. Rather than always opt into Iceberg, Iceberg should be an option.

Additionally, the Lake Formation flag enabled Lake Formation for the EMR Serverless job, but this did not work as expected. Setting Lake Formation enabled for the entire EMR Serverless job result in EMR Serverless system space being used. This system space does not work with Flint. The full limitations are documented in https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/emr-serverless-lf-enable-considerations.html. Instead Lake Formation can be enabled using the Iceberg catalog implementation.

Testing

Built and deployed to cluster and then queried Iceberg tables to verify functionality still works as well as testing the new functionality.

Setup

Created 2 data sources:

curl \
--request POST \
--url http://localhost:9200/_plugins/_query/_datasources \
--header 'content-type: application/x-ndjson' \
--data '{"name": "gdc2","description": "","connector": "S3GLUE","allowedRoles": [],
"properties": {"glue.auth.type": "iam_role","glue.auth.role_arn": "arn:aws:iam::476834799096:role/DirectQueryWithPublicLakeFormation","glue.indexstore.opensearch.uri": "http://ip-10-1-41-51.us-west-2.compute.internal:9200","glue.indexstore.
opensearch.auth": "noauth", "glue.iceberg.enabled": "true"}}'

curl \
--request POST \
--url http://localhost:9200/_plugins/_query/_datasources \
--header 'content-type: application/x-ndjson' \
--data '{"name": "gdc3","description": "","connector": "S3GLUE","allowedRoles": [],"properties": {"glue.auth.type": "iam_role","glue.auth.role_arn": "arn:aws:iam::476834799096:role/DirectQueryWithPublicLakeFormation","glue.indexstore.opensearch.uri": "http://ip-10-1-41-51.us-west-2.compute.internal:9200","glue.indexstore.opensearch.auth": "noauth", "glue.iceberg.enabled": "true", "glue.lakeformation.enabled": "true", "glue.lakeformation.session_tag": "directquery"}}'

Scenario 1 (S3 IAM permissions, Yes Iceberg and No Lake Formation flags set) Iceberg table should work

curl --request  POST --url http://localhost:9200/_plugins/_async_query --header 'content-type: application/x-ndjson' --data '{"datasource": "gdc2","lang": "sql","query": "SELECT * FROM gdc2.amazon_security_lake_glue_db_us_west_2.amazon_security_lake_table_us_west_2_vpc_flow_2_0 LIMIT 1;"}'
{
  "queryId": "aWt3aDQyM0JxaGdkYzI=",
  ...
}

curl --request GET --url http://localhost:9200/_plugins/_async_query/aWt3aDQyM0JxaGdkYzI=

Scenario 2 (LF permissions, Yes Iceberg and Yes Lake Formation flags set) Iceberg table should work

curl --request  POST --url http://localhost:9200/_plugins/_async_query --header 'content-type: application/x-ndjson' --data '{"datasource": "gdc3","lang": "sql","query": "SELECT * FROM gdc3.amazon_security_lake_glue_db_us_west_2.amazon_security_lake_table_us_west_2_vpc_flow_2_0 LIMIT 1;"}'
{
  "queryId": "T3A3RWlTSGVRSWdkYzM=",
  ...
}

curl --request GET --url http://localhost:9200/_plugins/_async_query/T3A3RWlTSGVRSWdkYzM=
{
  "status": "SUCCESS",
...
}

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[vamsi-amazon]

Have we introduced validations on these conditions while creating Glue datasource?

[asuresh8]

Thanks for catching this! I've added them now.

[penghuo]

@ykmr1224 could u take a look.
@asuresh8 could u check Java CI failure.

[asuresh8]

@asuresh8 could u check Java CI failure.

Failure does not look related to this change.

[ykmr1224]

Same as above. The value should be region name.

[ykmr1224]

Is it safe to specify the specific version? When do we notice the issue if we have version inconsistency?

[asuresh8]

This is hard to say. Some things I observed here is that using the Iceberg version in EMR was causing issues in EMR versions prior to 7.2 (Spark 3.5.1). So specifying iceberg from Maven central is more stable than that. On the AWS version, I'm not sure. The AWS sdk v2 is only used with Iceberg in the EMR 6.x versions, and this version doesn't conflict. That's not to say it couldn't be an issue in EMR 7.x.

[penghuo]

BWC Test failed, it is not related to this PR.
Track it seperatelly.