Create migration content for users updating their Kedro version

[noklam]

This issue serves as a summary doc, and a central thread to discuss how we can make migration easier.
For example, can we make the dependencies similar, how do you upgrade our dependency once you have a running project?

If we have enough stuff then we can turn this thread into some documentation pieces.

• Migration Guide in the RELEASE.md is the thing to look at - it mentions all the files you need to changes.
• existing project - upgrade strategy?
• do people know that you can do kedro new and use a specific version of starters?
• How can we handle the changes of files better - maybe some guideline here at least?

Workflow for upgrading from really old kedro version to latest

1. Make sure you commit all your changes before you start the migration (VERY IMPORTANT!)
2. Upgrade kedro in your conda or any virtual environment you are using
3. do kedro new to generate a new project, the project name should be the same as your original project (Do it in a separate directory)
4. Copy all the files from the old repository to the newly created project.
5. Copy all the files to the old repository (If there are new files, you should now see them with `git status), there maybe some files that you don't need anymore, check the Migration Guide in the RELEASE.md

Turns a kedro new project into a Spark project

• Update settings.py with SparkHook
• hooks.py with the SparkHook implementation
• spark.yml with the spark config
• requirementst.txt for spark-related library

[astrojuanlu]

I can try to do this in the upcoming sprint. If someone wants to share some lessons learned with me over a videoconf or similar I'd be happy to incorporate those.

[astrojuanlu]

Also, the whole "do kedro new in a new directory and copy things over" feels quite sloppy. I know this is a cookiecutter limitation, but depending on what's the output of #2388, I'm wondering if we could consider creating a kedro upgrade command of sorts. (This is out of scope for this issue.)

[astrojuanlu]

Remembered today about a @deepyaman's comment and went to see most downloaded Kedro versions. I think there's evidence that

• Users don't really upgrade,
• New versions are not immediately adopted.

https://pepy.tech/project/kedro?versions=0.*&display=daily&versions=0.17.0&versions=0.17.7&versions=0.16.3&versions=0.18.3&versions=0.18.8&versions=0.18.4&versions=0.18.7

[yetudada]

Actions:

• Let's document the current migration strategy
• Do some research by helping teams upgrade their projects (talk to the Verticals)

[astrojuanlu]

I'll open a separate ticket for the research.

[stichbury]

This would be a good ticket to consider as priority in coming sprints I think? Adding (tentatively) to next sprint.

[stichbury]

@astrojuanlu I don't know if you made a ticket for the research but i'm going to convert this ticket into the first of the action list to document migration strategy.

This will be three-fold:

• Draft a migration guide specifically for 0.18 -> 0.19 (this will be in the release notes of the develop branch)
• Draft a page about "How to update your Kedro version" for the docs
• Add some content (somewhere) for people that don't want to update (e.g. how to use an earlier version of a starter)

[stichbury]

I'm marking this done as we have

(a) release notes in progress about migration (ticket to revise/edit/improve those)
(b) FAQ about how to update Kedro version and how to roll back to use older versions.