HDDS-10572. Implement multiDelete using OMKeysDeleteRequest

[Tejaskriya]

What changes were proposed in this pull request?

BucketEndpoint#multiDelete deletes keys one by one. This causes 2 problems:

1. Bad performance: To delete each key, a round trip to the OM is made.
2. DIRECTORY_NOT_EMPTY errors, due to deletion order (in FSO buckets): For FSO buckets, if the directory is mentioned before the keys(inside the directory) in the multidelete request, it causes the request to fail even though it should be successful.

By reimplementing multidelete using OMKeysDeleteRequest to delete all items in a batch, these problems can be resolved.
With the existing API deleteKeys, we would not be able to get a list of errors for the keys that could not be deleted. Moreover, a OMException was being thrown if any errors were encountered.
In this PR, a new API is introduced deleteKeysQuiet to tackle these changes. It returns the list of errors without throwing an exception in case of any errors. This API is used by multidelete in the S3 gateway's BucketEndpoint#multiDelete

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10572

How was this patch tested?

Existing tests and locally in docker setup cluster.

$ aws s3 ls --endpoint http://localhost:9878 --recursive s3://bfso  
2024-05-31 12:38:53          0 dir1/
2024-05-31 12:38:53          0 dir1/dir2/
2024-05-31 12:38:54          3 dir1/dir2/k1
2024-05-17 13:13:54          3 k2

$ aws s3api delete-objects --endpoint http://localhost:9878 --delete file://delete.json --bucket bfso 
{
    "Deleted": [
        {
            "Key": "dir1"
        },
        {
            "Key": "dir1/dir2"
        },
        {
            "Key": "dir1/dir2/k1"
        }
    ]
}

$ aws s3 ls --endpoint http://localhost:9878 --recursive s3://bfso   
2024-05-17 13:13:54          3 k2


$ aws s3 ls --endpoint http://localhost:9878 --recursive s3://bos    
2024-05-31 12:38:45         3 dir1/dir2/k1
2024-05-17 13:14:13          3 k2

$ aws s3api delete-objects --endpoint http://localhost:9878 --delete file://delete.json --bucket bos   
{
    "Deleted": [
        {
            "Key": "dir1"
        },
        {
            "Key": "dir1/dir2"
        },
        {
            "Key": "dir1/dir2/k1"
        }
    ]
}

$ aws s3 ls --endpoint http://localhost:9878 --recursive s3://bos    
2024-05-17 13:14:13          3 k2

[adoroszlai]

Thanks @Tejaskriya for the patch. The implementation works fine. I have some suggestion for the new API, and some nits.

Also, please enable ITestS3AContractMkdir, which was disabled due to the inferior existing multiDelete implementation:

diff --git hadoop-ozone/dist/src/main/compose/common/s3a-test.sh hadoop-ozone/dist/src/main/compose/common/s3a-test.sh
index 1a784b1161..554b22b5a3 100644
--- hadoop-ozone/dist/src/main/compose/common/s3a-test.sh
+++ hadoop-ozone/dist/src/main/compose/common/s3a-test.sh
@@ -95,10 +95,9 @@ EOF
   # Some tests are skipped due to known issues.
   # - ITestS3AContractDistCp: HDDS-10616
   # - ITestS3AContractGetFileStatusV1List: HDDS-10617
-  # - ITestS3AContractMkdir: HDDS-10572
   # - ITestS3AContractRename: HDDS-10665
   mvn -B -V --fail-never --no-transfer-progress \
-    -Dtest='ITestS3AContract*, ITestS3ACommitterMRJob, !ITestS3AContractDistCp, !ITestS3AContractGetFileStatusV1List, !ITestS3AContractMkdir, !ITestS3AContractRename' \
+    -Dtest='ITestS3AContract*, ITestS3ACommitterMRJob, !ITestS3AContractDistCp, !ITestS3AContractGetFileStatusV1List, !ITestS3AContractRename' \
     clean test
 
   local target="${RESULT_DIR}/junit/${bucket}/target"

[Tejaskriya]

@adoroszlai Thank you for the review! I have enabled the S3A contract tests and addressed all your comments. Could you please take another look at the updated patch?

[ashishkumar50]

@Tejaskriya Thanks for the patch.

[Tejaskriya]

@ashishkumar50 Thanks for the review! I have addressed the review comments. Could you please review the new patch?

[ashishkumar50]

@Tejaskriya Thanks for updating patch, LGTM.

[adoroszlai]

Thanks @Tejaskriya for updating the patch.

[Tejaskriya]

@adoroszlai thanks for the review and approval! I took a look at the CI failures and they don't seem to be related to the PR changes. They execute successfully locally. Could you please re-trigger them?

[adoroszlai]

Thanks @Tejaskriya for the patch, @ashishkumar50 for the review.