Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HUDI-6401] should not throw exception when create marker file for lo… #9003

Merged
merged 1 commit into from
Jun 19, 2023
Merged

[HUDI-6401] should not throw exception when create marker file for lo… #9003

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,9 +294,8 @@ protected class AppendLogWriteCallback implements HoodieLogFileWriteCallback {

@Override
public boolean preLogFileOpen(HoodieLogFile logFileToAppend) {
// we use create rather than createIfNotExists because create method can trigger marker-based early conflict detection.
WriteMarkers writeMarkers = WriteMarkersFactory.get(config.getMarkersType(), hoodieTable, instantTime);
return writeMarkers.create(partitionPath, logFileToAppend.getFileName(), IOType.APPEND,
return writeMarkers.createIfNotExists(partitionPath, logFileToAppend.getFileName(), IOType.APPEND,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also needs to fix the comment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed incorrect comment.

config, fileId, hoodieTable.getMetaClient().getActiveTimeline()).isPresent();
}

Expand Down
28 changes: 28 additions & 0 deletions hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/marker/WriteMarkers.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,34 @@ public Option<Path> createIfNotExists(String partitionPath, String fileName, IOT
return create(partitionPath, fileName, type, true);
}

/**
* Creates a marker if the marker does not exist.
* This can invoke marker-based early conflict detection when enabled for multi-writers.
*
* @param partitionPath partition path in the table
* @param fileName file name
* @param type write IO type
* @param writeConfig Hudi write configs.
* @param fileId File ID.
* @param activeTimeline Active timeline for the write operation.
* @return the marker path.
*/
public Option<Path> createIfNotExists(String partitionPath, String fileName, IOType type, HoodieWriteConfig writeConfig,
String fileId, HoodieActiveTimeline activeTimeline) {
if (writeConfig.isEarlyConflictDetectionEnable()
&& writeConfig.getWriteConcurrencyMode().supportsOptimisticConcurrencyControl()) {
HoodieTimeline pendingCompactionTimeline = activeTimeline.filterPendingCompactionTimeline();
HoodieTimeline pendingReplaceTimeline = activeTimeline.filterPendingReplaceTimeline();
// TODO If current is compact or clustering then create marker directly without early conflict detection.
// Need to support early conflict detection between table service and common writers.
if (pendingCompactionTimeline.containsInstant(instantTime) || pendingReplaceTimeline.containsInstant(instantTime)) {
return create(partitionPath, fileName, type, true);
}
return createWithEarlyConflictDetection(partitionPath, fileName, type, false, writeConfig, fileId, activeTimeline);
}
return create(partitionPath, fileName, type, true);
}

/**
* Quietly deletes the marker directory.
*
Expand Down