Skip to content

Latest commit

 

History

History
144 lines (124 loc) · 3.83 KB

release-1.0.2.md

File metadata and controls

144 lines (124 loc) · 3.83 KB
Raw

.NET for Apache Spark 1.0.2 Release Notes

This is a maintenance release which contains backported PRs to support capturing Spark .NET exception stacktrace.

New Features/Improvements

  • Capture dotnet application error stack trace (#1054)

Bug Fixes

  • None

Infrastructure / Documentation / Etc.

  • Fix build pipeline due to bintray no longer available as spark-package repository (#1052)
  • Switch to 1ES hosted pools (#1056)

Breaking Changes

  • None

Known Issues

  • Broadcast variables do not work with dotnet-interactive (#561)
  • UDFs defined using class objects with closures does not work with dotnet-interactive (#619)
  • In dotnet-interactive blocking Spark methods that require external threads to unblock them does not work. ie StreamingQuery.AwaitTermination requires StreamingQuery.Stop to unblock (#736)
  • GroupedMap does not work on Spark 3.0.0 (#654)

Supported Spark Versions

The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with:

Spark Version microsoft-spark JAR
2.3.* microsoft-spark-2-3_2.11-1.0.2.jar
2.4.0 microsoft-spark-2-4_2.11-1.0.2.jar
2.4.1
2.4.3
2.4.4
2.4.5
2.4.6
2.4.7
2.4.8
2.4.2 Not supported
3.0.0 microsoft-spark-3-0_2.12-1.0.2.jar
3.0.1

Supported Delta Versions

The following table outlines the supported Delta versions along with the Microsoft.Spark.Extensions version to use with:

Delta Version Microsoft.Spark.Extensions.Delta
0.1.0 1.0.2
0.2.0
0.3.0
0.4.0
0.5.0
0.6.0
0.6.1
0.7.0

Supported Hyperspace Versions

The following table outlines the supported Hyperspace versions along with the Microsoft.Spark.Extensions version to use with:

Hyperspace Version Microsoft.Spark.Extensions.Hyperspace
0.1.0 1.0.2
0.2.0