Releasing 2.9.0

G-Research · Aug 23, 2023 · 5834d59 · 5834d59
1 parent bdddb3c
commit 5834d59
Show file tree

Hide file tree

Showing 5 changed files with 31 additions and 17 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,6 +3,20 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## [2.9.0] - 2023-08-23
+
+### Added
+
+- Add key order sensitive map comparator (#187)
+
+### Changed
+
+- Use dataset encoder rather than implicit value encoder for implicit dataset extension class (#183)
+
+### Fixed
+
+- Fix key-sensitivity in map comparator (#186)
+
 ## [2.8.0] - 2023-05-24
 
 ### Added

diff --git a/README.md b/README.md
@@ -119,7 +119,7 @@ The package version has the following semantics: `spark-extension_{SCALA_COMPAT_
 Add this line to your `build.sbt` file:
 
 ```sbt
-libraryDependencies += "uk.co.gresearch.spark" %% "spark-extension" % "2.8.0-3.4"
+libraryDependencies += "uk.co.gresearch.spark" %% "spark-extension" % "2.9.0-3.4"
 ```
 
 ### Maven
@@ -130,7 +130,7 @@ Add this dependency to your `pom.xml` file:
 <dependency>
   <groupId>uk.co.gresearch.spark</groupId>
   <artifactId>spark-extension_2.12</artifactId>
-  <version>2.8.0-3.4</version>
+  <version>2.9.0-3.4</version>
 </dependency>
 ```
 
@@ -140,7 +140,7 @@ Add this dependency to your `build.gradle` file:
 
 ```groovy
 dependencies {
-    implementation "uk.co.gresearch.spark:spark-extension_2.12:2.8.0-3.4"
+    implementation "uk.co.gresearch.spark:spark-extension_2.12:2.9.0-3.4"
 }
 ```
 
@@ -149,7 +149,7 @@ dependencies {
 Submit your Spark app with the Spark Extension dependency (version ≥1.1.0) as follows:
 
 ```shell script
-spark-submit --packages uk.co.gresearch.spark:spark-extension_2.12:2.8.0-3.4 [jar]
+spark-submit --packages uk.co.gresearch.spark:spark-extension_2.12:2.9.0-3.4 [jar]
 ```
 
 Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depending on your Spark version.
@@ -159,7 +159,7 @@ Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depe
 Launch a Spark Shell with the Spark Extension dependency (version ≥1.1.0) as follows:
 
 ```shell script
-spark-shell --packages uk.co.gresearch.spark:spark-extension_2.12:2.8.0-3.4
+spark-shell --packages uk.co.gresearch.spark:spark-extension_2.12:2.9.0-3.4
 ```
 
 Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depending on your Spark Shell version.
@@ -175,7 +175,7 @@ from pyspark.sql import SparkSession
 
 spark = SparkSession \
     .builder \
-    .config("spark.jars.packages", "uk.co.gresearch.spark:spark-extension_2.12:2.8.0-3.4") \
+    .config("spark.jars.packages", "uk.co.gresearch.spark:spark-extension_2.12:2.9.0-3.4") \
     .getOrCreate()
 ```
 
@@ -186,7 +186,7 @@ Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depe
 Launch the Python Spark REPL with the Spark Extension dependency (version ≥1.1.0) as follows:
 
 ```shell script
-pyspark --packages uk.co.gresearch.spark:spark-extension_2.12:2.8.0-3.4
+pyspark --packages uk.co.gresearch.spark:spark-extension_2.12:2.9.0-3.4
 ```
 
 Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depending on your PySpark version.
@@ -196,7 +196,7 @@ Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depe
 Run your Python scripts that use PySpark via `spark-submit`:
 
 ```shell script
-spark-submit --packages uk.co.gresearch.spark:spark-extension_2.12:2.8.0-3.4 [script.py]
+spark-submit --packages uk.co.gresearch.spark:spark-extension_2.12:2.9.0-3.4 [script.py]
 ```
 
 Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depending on your Spark version.
@@ -210,7 +210,7 @@ Running your Python application on a Spark cluster will still require one of the
 to add the Scala package to the Spark environment.
 
 ```shell script
-pip install pyspark-extension==2.8.0.3.4
+pip install pyspark-extension==2.9.0.3.4
 ```
 
 Note: Pick the right Spark version (here 3.4) depending on your PySpark version.
@@ -220,7 +220,7 @@ Note: Pick the right Spark version (here 3.4) depending on your PySpark version.
 There are plenty of [Data Science notebooks](https://datasciencenotebook.org/) around. To use this library,
 add **a jar dependency** to your notebook using these **Maven coordinates**:
 
-    uk.co.gresearch.spark:spark-extension_2.12:2.8.0-3.4
+    uk.co.gresearch.spark:spark-extension_2.12:2.9.0-3.4
 
 Or [download the jar](https://mvnrepository.com/artifact/uk.co.gresearch.spark/spark-extension) and place it
 on a filesystem where it is accessible by the notebook, and reference that jar file directly.

diff --git a/pom.xml b/pom.xml
@@ -2,7 +2,7 @@
   <modelVersion>4.0.0</modelVersion>
   <groupId>uk.co.gresearch.spark</groupId>
   <artifactId>spark-extension_2.13</artifactId>
-  <version>2.9.0-3.4-SNAPSHOT</version>
+  <version>2.9.0-3.4</version>
   <name>Spark Extension</name>
   <description>A library that provides useful extensions to Apache Spark.</description>
   <inceptionYear>2020</inceptionYear>

diff --git a/python/README.md b/python/README.md
@@ -72,7 +72,7 @@ Running your Python application on a Spark cluster will still require one of the
 to add the Scala package to the Spark environment.
 
 ```shell script
-pip install pyspark-extension==2.8.0.3.4
+pip install pyspark-extension==2.9.0.3.4
 ```
 
 Note: Pick the right Spark version (here 3.4) depending on your PySpark version.
@@ -86,7 +86,7 @@ from pyspark.sql import SparkSession
 
 spark = SparkSession \
     .builder \
-    .config("spark.jars.packages", "uk.co.gresearch.spark:spark-extension_2.12:2.8.0-3.4") \
+    .config("spark.jars.packages", "uk.co.gresearch.spark:spark-extension_2.12:2.9.0-3.4") \
     .getOrCreate()
 ```
 
@@ -97,7 +97,7 @@ Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depe
 Launch the Python Spark REPL with the Spark Extension dependency (version ≥1.1.0) as follows:
 
 ```shell script
-pyspark --packages uk.co.gresearch.spark:spark-extension_2.12:2.8.0-3.4
+pyspark --packages uk.co.gresearch.spark:spark-extension_2.12:2.9.0-3.4
 ```
 
 Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depending on your PySpark version.
@@ -107,7 +107,7 @@ Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depe
 Run your Python scripts that use PySpark via `spark-submit`:
 
 ```shell script
-spark-submit --packages uk.co.gresearch.spark:spark-extension_2.12:2.8.0-3.4 [script.py]
+spark-submit --packages uk.co.gresearch.spark:spark-extension_2.12:2.9.0-3.4 [script.py]
 ```
 
 Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depending on your Spark version.
@@ -117,7 +117,7 @@ Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depe
 There are plenty of [Data Science notebooks](https://datasciencenotebook.org/) around. To use this library,
 add **a jar dependency** to your notebook using these **Maven coordinates**:
 
-    uk.co.gresearch.spark:spark-extension_2.12:2.8.0-3.4
+    uk.co.gresearch.spark:spark-extension_2.12:2.9.0-3.4
 
 Or [download the jar](https://mvnrepository.com/artifact/uk.co.gresearch.spark/spark-extension) and place it
 on a filesystem where it is accessible by the notebook, and reference that jar file directly.

diff --git a/python/setup.py b/python/setup.py
@@ -17,7 +17,7 @@
 from pathlib import Path
 from setuptools import setup
 
-jar_version = '2.9.0-3.4-SNAPSHOT'
+jar_version = '2.9.0-3.4'
 scala_version = '2.13.8'
 scala_compat_version = '.'.join(scala_version.split('.')[:2])
 spark_compat_version = jar_version.split('-')[1]