How can we improve dataset versioning?

[merelcht]

Introduction

The catalog and dataset APIs and code needs to be improved. One of the most complex things related to datasets and the catalog is versioning. As a team we need to get a better understanding of how versioning works, so we can refactor it.

Background

#1778

What's in scope

• Understand how versioning works: can be a group exercise or one person can to a deep dive and teach the team
• A proposal of how to improve versioning.

[noklam]

Linked one interesting PR opened by our community earlier Configurable Versioning. I think this is interesting because we don't have much thought about sub-classing Version, at the moment this class is basically a nametuple.

[astrojuanlu]

xref #2355 (the evolution of the PR above)

[astrojuanlu]

xref also #2703 #2691 # 1731 these are all related, one way or another, to versioning.

[astrojuanlu]

Also partially related #1654

[astrojuanlu]

See what I found: an issue from 2019 where @Galileo-Galilei said

A huge thanks for the framework, which is really useful. My team decided to use it for most of its projects, especially to ensure collaboration. Data abstraction is really an important feature. However, we have a major disagreement of how data versioning is implemented in kedro. We decided to move on and to develop our own layer of versioning above your framework.

#113

[astrojuanlu]

And a bit of interesting historical insight from @/lorenabalan:

The current versioning behaviour actually follows the Spark notation - it's modelling exactly what Spark does under the hood when writing a file to multiple partitions, which is why we prefer the current implementation.

#371