Skip to content

Latest commit

 

History

History
101 lines (86 loc) · 3.18 KB

release-0.11.md

File metadata and controls

101 lines (86 loc) · 3.18 KB
Raw

.NET for Apache Spark 0.11 Release Notes

New Features and Improvements

  • Ability to pass and return corefxlab DataFrames to UDF APIs (#277)
  • Support for ML TF-IDF (Term frequency-inverse document frequency) feature vectorization method (#394)
  • Support for TimestampType in DataFrame.Collect(), CreateDataFrame and UDFs (#428)
  • Support for Broadcast Variables (#414)
  • Support for ML feature Word2Vec (#491)
  • Streamline logging when there is a failure (#439)

Breaking Changes

  • SparkSession.Catalog is changed from a method to a property (#508)

Compatibility

Backward compatibility

The following table describes the oldest version of the worker that the current version is compatible with, along with new features that are incompatible with the worker.

Oldest compatible Microsoft.Spark.Worker version Incompatible features
v0.9.0 DataFrame with Grouped Map UDF (#277)
DataFrame with Vector UDF (#277)
Support for Broadcast Variables (#414)
Support for TimestampType (#428)

Forward compatibility

The following table describes the oldest version of .NET for Apache Spark release that the current worker is compatible with.

Oldest compatible .NET for Apache Spark release version
v0.9.0

Supported Spark Versions

The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with:

Spark Version microsoft-spark JAR
2.3.* microsoft-spark-2.3.x-0.11.0.jar
2.4.0 microsoft-spark-2.4.x-0.11.0.jar
2.4.1
2.4.3
2.4.4
2.4.5
2.4.2 Not supported