HDDS-10630. Add missing parent directories deleted between initiate and complete MPU #6496

SaketaChalamchala · 2024-04-08T15:42:19Z

What changes were proposed in this pull request?

S3a provides multiple mapreduce committers. When using the directory staging committer fs.s3a.committer.name=directory with replace conflict mode fs.s3a.committer.staging.conflict-mode=replace and writing to FSO buckets, the job fails with the following errors.

This is because the logic to add back missing parent directories and missing output prefix directories similar to Initiate MPU request (S3InitiateMultipartUploadRequestWithFSO.java, S3InitiateMultipartUploadResponseWithFSO.java) is missing in the Complete MPU request.

The proposed solution adds the missing parent and output prefix directories back to DB before completing the MPU.

Errors:

##When creating getDBOzoneKey
StateMachine ApplyTransaction Thread - 0]-org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequest: MultipartUpload Complete request failed for Key: st-data-con-jqmmwt/qetest/terasort/output-1710494153/part-r-00000 in Volume/Bucket s3v/qe-dataconn-bucket
DIRECTORY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to find parent directory of st-data-con-jqmmwt/qetest/terasort/output-1710494153/part-r-00000
	at org.apache.hadoop.ozone.om.request.file.OMFileRequest.getParentID(OMFileRequest.java:1008)
	at org.apache.hadoop.ozone.om.request.file.OMFileRequest.getParentID(OMFileRequest.java:958)
	at org.apache.hadoop.ozone.om.request.file.OMFileRequest.getParentId(OMFileRequest.java:1038)
	at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequestWithFSO.getDBOzoneKey(S3MultipartUploadCompleteRequestWithFSO.java:114)
	at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequest.validateAndUpdateCache(S3MultipartUploadCompleteRequest.java:157)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:378)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:568)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:363)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

## When getting omKeyInfo from DB
StateMachine ApplyTransaction Thread - 0]-org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine: Terminating with exit status 1: Request cmdType: CompleteMultiPartUpload
traceID: ""
clientId: "client-5168AA460706"
userInfo {
  userName: "xxx"
  remoteAddress: "..."
  hostName: "..."
}
version: 3
completeMultiPartUploadRequest {
  keyArgs {
    volumeName: "s3v"
    bucketName: "qe-dataconn-bucket"
    keyName: "st-data-con-jqmmwt/qetest/terasort/output-1710494153/part-r-00000"
    multipartUploadID: "e1c08d2c-5798-4c81-ab2c-7a0bd0fea4c9-112101950169613319"
    acls {
      type: USER
      name: "..."
      rights: "\200"
      aclScope: ACCESS
    }
    acls {
      type: GROUP
      name: "hrt_qa"
      rights: "\000\001"
      aclScope: ACCESS
    }
    acls {
      type: GROUP
      name: "users"
      rights: "\000\001"
      aclScope: ACCESS
    }
    acls {
      type: GROUP
      name: "hivetest"
      rights: "\000\001"
      aclScope: ACCESS
    }
    modificationTime: 1710540186541
  }
  partsList {
    partNumber: 1
    partName: "/s3v/qe-dataconn-bucket/st-data-con-jqmmwt/qetest/terasort/output-1710494153/part-r-00000-e1c08d2c-5798-4c81-ab2c-7a0bd0fea4c9-112101950169613319-1"
  }
}
s3Authentication {
  stringToSign: "..."
  signature: "..."
  accessId: "..."
}
 failed with exception
java.lang.NullPointerException
	at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequest.getOmKeyInfo(S3MultipartUploadCompleteRequest.java:378)
	at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequest.validateAndUpdateCache(S3MultipartUploadCompleteRequest.java:202)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:378)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:568)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:363)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10630

How was this patch tested?

Ran the s3a hadoop contract test ITestS3ACommitterMRJob to verify. There is another PR out to add s3a contract tests to acceptance testing.

 ## Startup unsecure ozone cluster using docker-compose and create an FSO bucket
cd hadoop-ozone/dist/target/ozone-*-SNAPSHOT/compose/ozone
docker-compose up -d --scale datanode=3
ozone sh bucket create /s3v/fso-bucket -l FILE_SYSTEM_OPTIMIZED

## Download the hadoop-aws source
curl -LSs -o "hadoop-src.tar.gz" https://archive.apache.org/dist/hadoop/common/hadoop-3.3.6/hadoop-3.3.6-src.tar.gz
tar -x -z -C "hadoop-src" --strip-components=3 -f "hadoop-src.tar.gz" 'hadoop-*-src/hadoop-tools/hadoop-aws'

## Create auth-keys.xml
vi hadoop-src/src/test/resources/auth-keys.xml 
<configuration>

  <property>
    <name>fs.s3a.endpoint</name>
    <value>http://localhost:9878</value>
  </property>

  <property>
    <name>fs.s3a.access.key</name>
    <value>s3a-contract</value>
  </property>

  <property>
    <name>fs.s3a.secret.key</name>
    <value>unsecure</value>
  </property>

 <property>
    <name>fs.s3a.committer.staging.conflict-mode</name>
    <value>replace</value>
  </property>

  <property>
    <name>fs.contract.test.fs.s3a</name>
    <value>s3a://fso-bucket/</value>
  </property>

  <property>
    <name>test.fs.s3a.name</name>
    <value>s3a://fso-bucket/</value>
  </property>

  <property>
    <name>test.fs.s3a.sts.enabled</name>
    <value>false</value>
  </property>

  <property>
    <name>fs.s3a.path.style.access</name>
    <value>true</value>
  </property>

  <property>
    <name>fs.s3a.directory.marker.retention</name>
    <value>keep</value>
  </property>

</configuration>

## Run the ITestS3ACommitterMRJob contract test
mvn clean test -B -V --no-transfer-progress -Dtest='ITestS3ACommitterMRJob'

…leted between commit and complete mpu uploads.

adoroszlai

Thanks @SaketaChalamchala for the patch. Functionality looks OK to me. Please take a look at failing unit test. Would be nice to reuse some code from S3InitiateMultipartUploadRequestWithFSO.

… are missing.

SaketaChalamchala · 2024-04-09T04:32:31Z

Thanks for the review @adoroszlai. The unit tests have been fixed.
The CI is running on my fork: https://github.com/SaketaChalamchala/ozone/actions/runs/8609322147. There's just a recon test that's timing out. Retrying it.

adoroszlai · 2024-04-10T04:47:09Z

...org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCompleteRequestWithFSO.java

+    // Create missing parent directory entries.
+    try (BatchOperation batchOperation = omMetadataManager.getStore()
+        .initBatchOperation()) {
+      for (OmDirectoryInfo parentDirInfo : missingParentInfos) {
+        final String parentKey = omMetadataManager.getOzonePathKey(
+            volumeId, bucketId, parentDirInfo.getParentObjectID(),
+            parentDirInfo.getName());
+        omMetadataManager.getDirectoryTable().putWithBatch(batchOperation,
+            parentKey, parentDirInfo);
+      }
+
+      // namespace quota changes for parent directory
+      String bucketKey = omMetadataManager.getBucketKey(
+          omBucketInfo.getVolumeName(),
+          omBucketInfo.getBucketName());
+      omMetadataManager.getBucketTable().putWithBatch(batchOperation,
+          bucketKey, omBucketInfo);
+
+      omMetadataManager.getStore().commitBatchOperation(batchOperation);
+    }


I think adding to DB batch should be handled in the response object by design. It uses an existing batch and commits only if the response is OK.

ozone/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/s3/multipart/S3InitiateMultipartUploadResponseWithFSO.java

Lines 79 to 102 in 06c0d81

@Override

public void addToDBBatch(OMMetadataManager omMetadataManager,

BatchOperation batchOperation) throws IOException {

/**

* Create parent directory entries during MultiPartFileKey Create - do not

* wait for File Commit request.

*/

if (parentDirInfos != null) {

for (OmDirectoryInfo parentDirInfo : parentDirInfos) {

final String parentKey = omMetadataManager.getOzonePathKey(

volumeId, bucketId, parentDirInfo.getParentObjectID(),

parentDirInfo.getName());

omMetadataManager.getDirectoryTable().putWithBatch(batchOperation,

parentKey, parentDirInfo);

}

// namespace quota changes for parent directory

String bucketKey = omMetadataManager.getBucketKey(

omBucketInfo.getVolumeName(),

omBucketInfo.getBucketName());

omMetadataManager.getBucketTable().putWithBatch(batchOperation,

bucketKey, omBucketInfo);

}

ashishkumar50

@SaketaChalamchala Thanks for the patch, Please find the comment below.
Only cache entry should be added in request and db insert should be done in response.

ashishkumar50 · 2024-04-10T04:49:05Z

...org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCompleteRequestWithFSO.java

+        final String parentKey = omMetadataManager.getOzonePathKey(
+            volumeId, bucketId, parentDirInfo.getParentObjectID(),
+            parentDirInfo.getName());
+        omMetadataManager.getDirectoryTable().putWithBatch(batchOperation,


I think this should be done in S3MultipartUploadCompleteResponseWithFSO instead here or else db entry will be added only on the leader OM.

ashishkumar50 · 2024-04-10T04:49:47Z

...org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCompleteRequestWithFSO.java

+    try (BatchOperation batchOperation = omMetadataManager.getStore()
+        .initBatchOperation()) {
+
+      OMFileRequest.addToOpenFileTableForMultipart(omMetadataManager,


Even this should be done in S3MultipartUploadCompleteResponseWithFSO instead here.

…onseWithFSO.java

SaketaChalamchala · 2024-04-11T03:11:30Z

@adoroszlai and @ashishkumar50 updated the patch based on your comments. Please let me know if you have any more comments.

adoroszlai

Thanks @SaketaChalamchala for updating the patch, LGTM. Also tested with ITestS3ACommitterMRJob.

ashishkumar50

@SaketaChalamchala Thanks for updating patch, Change LGTM.

adoroszlai · 2024-04-12T06:40:48Z

Thanks @SaketaChalamchala for the patch, @ashishkumar50 for the review.

…nd complete MPU (apache#6496)

…nd complete MPU (apache#6496) (cherry picked from commit c1b27a8)

…s deleted between initiate and complete MPU (apache#6496) (cherry picked from commit c1b27a8) Change-Id: I365ea43497bf47d5c00570c665f2c293e73f4c09

saketa and others added 3 commits April 4, 2024 22:47

HDDS-10630. Adding missing parent and prefix directories that were de…

322035f

…leted between commit and complete mpu uploads.

HDDS-10630. Fixed checkstyle errors.

7c102bb

Merge branch 'apache:master' into HDDS-10630

a1d7755

adoroszlai added the s3 S3 Gateway label Apr 8, 2024

kerneltime requested review from kerneltime, duongkame and adoroszlai April 8, 2024 16:22

adoroszlai reviewed Apr 8, 2024

View reviewed changes

SaketaChalamchala and others added 2 commits April 8, 2024 19:07

Merge branch 'apache:master' into HDDS-10630

4a0bbb6

HDDS-10630. Add multi part to open key table only when it and parents…

bff4553

… are missing.

adoroszlai requested review from ashishkumar50 and removed request for ashishkumar50 April 9, 2024 08:42

adoroszlai reviewed Apr 10, 2024

View reviewed changes

ashishkumar50 reviewed Apr 10, 2024

View reviewed changes

saketa added 2 commits April 10, 2024 19:49

HDDS-10630. Moved DB batch operation to S3MultipartUploadCompleteResp…

c200cf1

…onseWithFSO.java

HDDS-10630. Fixed checkstyle errors.

a1c5c53

adoroszlai reviewed Apr 11, 2024

View reviewed changes

adoroszlai changed the title ~~HDDS-10630. Add missing parent and prefix directories that were deleted between commit and complete mpu uploads.~~ HDDS-10630. Add missing parent directories deleted between initiate and complete MPU Apr 11, 2024

ashishkumar50 approved these changes Apr 12, 2024

View reviewed changes

adoroszlai merged commit c1b27a8 into apache:master Apr 12, 2024
40 checks passed

Tejaskriya pushed a commit to Tejaskriya/ozone that referenced this pull request Apr 17, 2024

HDDS-10630. Add missing parent directories deleted between initiate a…

96c7cec

…nd complete MPU (apache#6496)

xichen01 pushed a commit to xichen01/ozone that referenced this pull request Apr 17, 2024

HDDS-10630. Add missing parent directories deleted between initiate a…

35d19a7

…nd complete MPU (apache#6496) (cherry picked from commit c1b27a8)

xichen01 pushed a commit to xichen01/ozone that referenced this pull request Apr 18, 2024

HDDS-10630. Add missing parent directories deleted between initiate a…

a0f9887

…nd complete MPU (apache#6496) (cherry picked from commit c1b27a8)

xichen01 pushed a commit to xichen01/ozone that referenced this pull request Apr 18, 2024

HDDS-10630. Add missing parent directories deleted between initiate a…

a38906f

…nd complete MPU (apache#6496) (cherry picked from commit c1b27a8)

xichen01 mentioned this pull request Apr 18, 2024

[DO NOT MERGE] Backport some fixes from master to ozone-1.4 #6553

Merged

xichen01 pushed a commit to xichen01/ozone that referenced this pull request Apr 23, 2024

HDDS-10630. Add missing parent directories deleted between initiate a…

727ef2c

…nd complete MPU (apache#6496) (cherry picked from commit c1b27a8)

jojochuang pushed a commit to jojochuang/ozone that referenced this pull request May 29, 2024

HDDS-10630. Add missing parent directories deleted between initiate a…

8383ac3

…nd complete MPU (apache#6496) (cherry picked from commit c1b27a8)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-10630. Add missing parent directories deleted between initiate and complete MPU #6496

HDDS-10630. Add missing parent directories deleted between initiate and complete MPU #6496

SaketaChalamchala commented Apr 8, 2024 •

edited by adoroszlai

Loading

adoroszlai left a comment

SaketaChalamchala commented Apr 9, 2024

adoroszlai Apr 10, 2024

ashishkumar50 left a comment

ashishkumar50 Apr 10, 2024

ashishkumar50 Apr 10, 2024

SaketaChalamchala commented Apr 11, 2024

adoroszlai left a comment

ashishkumar50 left a comment

adoroszlai commented Apr 12, 2024

	@Override
	public void addToDBBatch(OMMetadataManager omMetadataManager,
	BatchOperation batchOperation) throws IOException {

	/**
	* Create parent directory entries during MultiPartFileKey Create - do not
	* wait for File Commit request.
	*/
	if (parentDirInfos != null) {
	for (OmDirectoryInfo parentDirInfo : parentDirInfos) {
	final String parentKey = omMetadataManager.getOzonePathKey(
	volumeId, bucketId, parentDirInfo.getParentObjectID(),
	parentDirInfo.getName());
	omMetadataManager.getDirectoryTable().putWithBatch(batchOperation,
	parentKey, parentDirInfo);
	}

	// namespace quota changes for parent directory
	String bucketKey = omMetadataManager.getBucketKey(
	omBucketInfo.getVolumeName(),
	omBucketInfo.getBucketName());
	omMetadataManager.getBucketTable().putWithBatch(batchOperation,
	bucketKey, omBucketInfo);
	}

HDDS-10630. Add missing parent directories deleted between initiate and complete MPU #6496

HDDS-10630. Add missing parent directories deleted between initiate and complete MPU #6496

Conversation

SaketaChalamchala commented Apr 8, 2024 • edited by adoroszlai Loading

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

adoroszlai left a comment

Choose a reason for hiding this comment

SaketaChalamchala commented Apr 9, 2024

adoroszlai Apr 10, 2024

Choose a reason for hiding this comment

ashishkumar50 left a comment

Choose a reason for hiding this comment

ashishkumar50 Apr 10, 2024

Choose a reason for hiding this comment

ashishkumar50 Apr 10, 2024

Choose a reason for hiding this comment

SaketaChalamchala commented Apr 11, 2024

adoroszlai left a comment

Choose a reason for hiding this comment

ashishkumar50 left a comment

Choose a reason for hiding this comment

adoroszlai commented Apr 12, 2024

SaketaChalamchala commented Apr 8, 2024 •

edited by adoroszlai

Loading