This fixes #1866 (#4525)

Make this more consistent with implementation of FileFormatWriter change implemented in SPARK-30227 by adding finallyBlock with close() call on dataWriter Signed-off-by: Navin Kumar <navink@nvidia.com>
NVIDIA · Jan 14, 2022 · 957da55 · 957da55
1 parent b64e316
commit 957da55
Showing 1 changed file with 2 additions and 0 deletions.
diff --git a/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuFileFormatWriter.scala b/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuFileFormatWriter.scala
@@ -310,6 +310,8 @@ object GpuFileFormatWriter extends Logging {
         // If there is an error, abort the task
         dataWriter.abort()
         logError(s"Job $jobId aborted.")
+      }, finallyBlock = {
+        dataWriter.close()
       })
     } catch {
       case e: FetchFailedException =>