Skip to content

Commit

Permalink
[SPARK-44703][CORE] Log eventLog rewrite duration when compact old ev…
Browse files Browse the repository at this point in the history
…ent log files

### What changes were proposed in this pull request?
Log eventLog rewrite duration when compact old event log files.

### Why are the changes needed?
When enable `spark.eventLog.rolling.enabled` and the number of eventLog files exceeds the value of `spark.history.fs.eventLog.rolling.maxFilesToRetain`, HistoryServer will compact the old event log files into one compact file.

Currently there is no log the rewrite duration in rewrite method, this metric is useful for understand the compact duration, so we need add logs in the method.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Manual test.

Closes apache#42378 from shuyouZZ/SPARK-44703.

Authored-by: Shuyou Dong <shuyou.dong@shopee.com>
Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
  • Loading branch information
shuyouZZ authored and HeartSaVioR committed Aug 8, 2023
1 parent 590b77f commit b4b9121
Showing 1 changed file with 3 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ class EventLogFileCompactor(
val logWriter = new CompactedEventLogFileWriter(lastIndexEventLogPath, "dummy", None,
lastIndexEventLogPath.getParent.toUri, sparkConf, hadoopConf)

val startTime = System.currentTimeMillis()
logWriter.start()
eventLogFiles.foreach { file =>
EventFilter.applyFilterToFile(fs, filters, file.getPath,
Expand All @@ -158,6 +159,8 @@ class EventLogFileCompactor(
)
}
logWriter.stop()
val duration = System.currentTimeMillis() - startTime
logInfo(s"Finished rewriting eventLog files to ${logWriter.logPath} took $duration ms.")

logWriter.logPath
}
Expand Down

0 comments on commit b4b9121

Please sign in to comment.