Skip to content

Latest commit

 

History

History
203 lines (177 loc) · 5 KB

release-2.1.0.md

File metadata and controls

203 lines (177 loc) · 5 KB
Raw

.NET for Apache Spark 2.1.0 Release Notes

New Features/Improvements

  • Support for Spark 3.2.0 and 3.2.1 (#1010)
  • Add Spark 3.2.0 Functions APIs (#1013)
  • Update Microsoft.Data.Analysis version to 0.18.0 (#977)

Bug Fixes

  • None

Infrastructure / Documentation / Etc.

  • Remove Ubuntu 1604 agent pool image as it is no longer supported (#1007)

Breaking Changes

  • None

Known Issues

  • Broadcast variables do not work with dotnet-interactive (#561)
  • UDFs defined using class objects with closures does not work with dotnet-interactive (#619)
  • In dotnet-interactive blocking Spark methods that require external threads to unblock them does not work. ie StreamingQuery.AwaitTermination requires StreamingQuery.Stop to unblock (#736)

Compatibility

Backward compatibility

The following table describes the oldest version of the worker that the current version is compatible with, along with new features that are incompatible with the worker.

Oldest compatible Microsoft.Spark.Worker version
v2.0.0

Forward compatibility

The following table describes the oldest version of .NET for Apache Spark release that the current worker is compatible with.

Oldest compatible .NET for Apache Spark release version
v2.0.0

Supported Spark Versions

The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with:

Spark Version microsoft-spark JAR
2.4.0 microsoft-spark-2-4_2.11-2.1.0.jar
2.4.1
2.4.3
2.4.4
2.4.5
2.4.6
2.4.7
2.4.2 Not supported
3.0.0 microsoft-spark-3-0_2.12-2.1.0.jar
3.0.1
3.0.2
3.1.1 microsoft-spark-3-1_2.12-2.1.0.jar
3.1.2
3.2.0 microsoft-spark-3-2_2.12-2.1.0.jar
3.2.1

Supported Delta Versions

The following table outlines the supported Delta versions along with the Microsoft.Spark.Extensions version to use with:

Delta Version Microsoft.Spark.Extensions.Delta
0.1.0 2.1.0
0.2.0
0.3.0
0.4.0
0.5.0
0.6.0
0.6.1
0.7.0
0.8.0
1.0.0
1.1.0

Supported Hyperspace Versions

The following table outlines the supported Hyperspace versions along with the Microsoft.Spark.Extensions version to use with:

Hyperspace Version Microsoft.Spark.Extensions.Hyperspace
0.1.0 2.1.0
0.2.0
0.3.0
0.4.0