Delta Datasets: Loosen up delta-spark requirement

[rwpurvis]

Description

Currently the delta-base dataset requires delta-spark~=1.2.1 which requires spark 3.2. This means that any databricks users are limited to spark 3.2. Can we loosen this requirement?

Context

How has this bug affected you? What were you trying to accomplish?

• Unable to use databricks.ManagedTableDataset in current project

Steps to Reproduce

1. Create a project requiring pyspark>=3.3
2. attempt to install kedro-datasets[databricks.ManagedTableDataset]

Expected Result

Able to use latest version of spark or delta lake

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• Kedro version used (pip show kedro or kedro -V): 0.19.3
• Kedro plugin and kedro plugin version used (pip show kedro-airflow): any
• Python version used (python -V): 3.10
• Operating system and version: windows/ubuntu

[rwpurvis]

It appears this may be to the update of the dependency names quoted below. I have updated dependencies to the PEP 685 compliant names and am testing.

EDIT: I am able to get dependencies to resolve but this is still pinned to pyspark~=3.2, which should be loosened IMO.

From kedro-datasets version 3.0.0 onwards, the names of the optional dataset-level dependencies have been normalised to follow PEP 685. The '.' character has been replaced with a '-' character and the names are in lowercase. For example, if you had kedro-datasets[pandas.ExcelDataset] in your requirements file, it would have to be changed to kedro-datasets[pandas-exceldataset].

[DimedS]

Thank you for bringing this up, @rwpurvis! This issue was actually discussed earlier in issue #571, but for some reasons, it wasn't implemented at that time.

Would you be interested in making a pull request to address this?

[rwpurvis]

Yes, I can take it up.

[rwpurvis]

Looks like @felipemonroy beat me to it with #780! 🚀

[astrojuanlu]

Uh as discussed there, not quite. We'll need another bump.

[rwpurvis]

Opened #814

[astrojuanlu]

Fixed in #814