%reload_kedro taking 2mins+ on Databricks

[sagrawal128]

Description

Hi, I am using Kedro 18.14 on Databricks 11.4 LTS. I am trying to run kedro as suggested in documentation and using %reload_kedro to refresh session. However, it takes more than 2 mins for this command to finish. Is this expected behaviour?

Context

Slow run time

Steps to Reproduce

1. Create Databricks compute with 11.3 machine
2. Install kedro 18.14 in the notebook
3. Add a kedro project to workspace using the git repository option on Databricks
4. Try to load kedro project using %reload_kedro magic

Expected Result

The reload magic should be fast, a few seconds probably

Actual Result

Your Environment

• Kedro version used (pip show kedro or kedro -V): 18.14
• Python version used (python -V): 3.9
• Operating system and version: Databricks (details in image below)

[ravi-kumar-pilla]

Hi @sagrawal128 , Thanks for raising the issue. We will try to replicate the issue on our end and see if there are areas to improve. Thank you

[noklam]

Sorry for the late response, @sagrawal128 How big is the project? Is the setup very heavy that it takes a lot of time? During %reload_kedro, Kedro basically try to create

• DataCatalog
• Pipeline
• ConfigLoader

These are all components that is necessary before a pipeline run, usually they should be fast. It would be also good to check what kind of dataset are you using, if possible upgrade kedro-datasets. There were issue with some older dataset that setup a database connection early which is causing performance issue.