- Add base classes for ML and refine code base (#1031, #1049)
- Capture dotnet application error stack trace (#1047)
- JvmBridge semaphore fix (#1061)
- None
- Switch to 1ES hosted pools on main (#1021)
- None
- Broadcast variables do not work with dotnet-interactive (#561)
- UDFs defined using class objects with closures does not work with dotnet-interactive (#619)
- In dotnet-interactive blocking Spark methods that require external threads to unblock them does not work. ie
StreamingQuery.AwaitTermination
requiresStreamingQuery.Stop
to unblock (#736)
The following table describes the oldest version of the worker that the current version is compatible with, along with new features that are incompatible with the worker.
Oldest compatible Microsoft.Spark.Worker version |
---|
v2.0.0 |
The following table describes the oldest version of .NET for Apache Spark release that the current worker is compatible with.
Oldest compatible .NET for Apache Spark release version |
---|
v2.0.0 |
The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with:
Spark Version | microsoft-spark JAR |
---|---|
2.4.0 | microsoft-spark-2-4_2.11-2.1.1.jar |
2.4.1 | |
2.4.3 | |
2.4.4 | |
2.4.5 | |
2.4.6 | |
2.4.7 | |
2.4.2 | Not supported |
3.0.0 | microsoft-spark-3-0_2.12-2.1.1.jar |
3.0.1 | |
3.0.2 | |
3.1.1 | microsoft-spark-3-1_2.12-2.1.1.jar |
3.1.2 | |
3.2.0 | microsoft-spark-3-2_2.12-2.1.1.jar |
3.2.1 |
The following table outlines the supported Delta versions along with the Microsoft.Spark.Extensions version to use with:
Delta Version | Microsoft.Spark.Extensions.Delta |
---|---|
0.1.0 | 2.1.1 |
0.2.0 | |
0.3.0 | |
0.4.0 | |
0.5.0 | |
0.6.0 | |
0.6.1 | |
0.7.0 | |
0.8.0 | |
1.0.0 | |
1.1.0 |
The following table outlines the supported Hyperspace versions along with the Microsoft.Spark.Extensions version to use with:
Hyperspace Version | Microsoft.Spark.Extensions.Hyperspace |
---|---|
0.1.0 | 2.1.1 |
0.2.0 | |
0.3.0 | |
0.4.0 |