- Note that there was a breaking forward compatibility change (#871) that went into the 1.1.0 release. This is the re-release of 1.1.0 after reverting the change. If you are using the 1.1.0 release, it is strongly encouraged to upgrade to 1.1.1.
We are planning to drop the support for Spark 2.3 in the 2.0 release, which will be the next release. The last Spark 2.3 release (2.3.4) was back in September 2019, and no new release is planned for Spark 2.3. Since there have been no new features introduced for Spark 2.3 in the last few releases of .NET for Apache Spark, if you are relying on Spark 2.3, you should be able to continue using .NET for Apache Spark 1.x.
- Support for Arrow 2.0 and GroupedMapUdf in Spark 3.0.0 (#711)
- Use pattern matching in arrow test utils to improve readability (#725)
- Check whether file is found before trying to dereference it (#759)
- Ml/feature hasher has only internal contructors (#761)
- Support for stop words removers (#726)
- Support for adding NGram functionality (#734)
- Add support for SQLTransformer ML feature (#781)
- Add skeletal support for FileSystem extension (#787)
- Using (processId, threadId) as key to mantain threadpool executor instead of only threadId (#793)
- Support for Hyperspace 0.4.0 (#815)
- Support for Delta Lake 0.8.0 (#823)
- Add support for Spark 3.0.2 (#833)
- Add DOTNET_WORKER__DIR environment variable (#861)
- Add spark.dotnet.ignoreSparkPatchVersionCheck conf to ignore patch version in DotnetRunner (#862)
- Fix signer information mismatch issue (#752)
- Fix package-worker.ps1 to handle output path with ":" (#742)
- Fix for using Broadcast variables in Databricks (#766)
- Fix macOS Catalina Permissions (#784)
- Fix for memory leak in JVMObjectTracker (#801)
- Bug Fix for Spark 3.x - Avoid converting converted Row values (#868)
- Add 'Z' to the string format in Timestamp.ToString() to indicate UTC time (#871)
- Fix flaky CallbackTests.TestCallbackHandlers Test (#745)
- Run E2E tests on Linux in build pipeline and add Backward/Forward E2E tests (#737)
- Update dotnet-interactive deprecated feed (#807, #808)
- Remove unnecessary RestoreSources (#812)
- Migrating master to main branch (#847, #849)
- None
- Broadcast variables do not work with dotnet-interactive (#561)
- UDFs defined using class objects with closures does not work with dotnet-interactive (#619)
- In dotnet-interactive blocking Spark methods that require external threads to unblock them does not work. ie
StreamingQuery.AwaitTermination
requiresStreamingQuery.Stop
to unblock (#736) - UDFs don't work in Linux with Spark 2.3.0 (#753)
The following table describes the oldest version of the worker that the current version is compatible with, along with new features that are incompatible with the worker.
Oldest compatible Microsoft.Spark.Worker version | Incompatible features |
---|---|
v1.0.0 | GroupedMap in Spark 3.0 is not compatible with Worker 1.0 (#654)* |
The following table describes the oldest version of .NET for Apache Spark release that the current worker is compatible with.
Oldest compatible .NET for Apache Spark release version |
---|
v1.0.0 |
The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with:
Spark Version | microsoft-spark JAR |
---|---|
2.3.* | microsoft-spark-2-3_2.11-1.1.1.jar |
2.4.0 | microsoft-spark-2-4_2.11-1.1.1.jar |
2.4.1 | |
2.4.3 | |
2.4.4 | |
2.4.5 | |
2.4.6 | |
2.4.7 | |
2.4.2 | Not supported |
3.0.0 | microsoft-spark-3-0_2.12-1.1.1.jar |
3.0.1 | |
3.0.2 |
The following table outlines the supported Delta versions along with the Microsoft.Spark.Extensions version to use with:
Delta Version | Microsoft.Spark.Extensions.Delta |
---|---|
0.1.0 | 1.1.1 |
0.2.0 | |
0.3.0 | |
0.4.0 | |
0.5.0 | |
0.6.0 | |
0.6.1 | |
0.7.0 | |
0.8.0 |
The following table outlines the supported Hyperspace versions along with the Microsoft.Spark.Extensions version to use with:
Hyperspace Version | Microsoft.Spark.Extensions.Hyperspace |
---|---|
0.1.0 | 1.1.1 |
0.2.0 | |
0.3.0 | |
0.4.0 |