fsspec 2021.5.0 & compatibility with pytorch lightning >= 1.3.0

[martinsotir]

Description

Pytorch Lightning requires fsspec 2021.5.0 which clashes with Kedro 's requirement of fsspec < 0.9.

Context

Pytorch Lightning is a popular training loop wrapper for Pytorch. I am using it in a data pipeline to train neural nets.
The latest major release, 1.3, includes many bug fixes. I think they added a constraint on the fsspec versions because of some bugs in previous versions (for instance: fsspec/filesystem_spec#591). (However this makes it difficult to keep up with compatible versions. It's not the first time that Lightning change the constraint on fsspec)/

Possible Alternatives

Downgrade to Lightning 1.2.10 with fsspec 0.8.7

[ignacioparicio]

Hi @martinsotir and thanks for bringing up your issue! This is on our radar, we recently discussed internally fsspec's change from SemVer to CalVer. I created a ticket for it, but please understand that these are very recent (<2 months) changes for a single external dependency, so it might not make it to the next Kedro release. In any event, someone from the team will come back to this issue once we upgrade fsspec.

[ignacioparicio]

We have upgraded our fsspec requirement from fsspec>=0.5.1, <0.9 to fsspec>=2021.04, <2022.01 (the upper bound to be revised past that date). The change will come into effect with the next Kedro release, be it 0.17.5 or 0.18.0.

I'm closing this issue for now, but please shout if you want it reopen.