[KED-2768] kedro install doesn't play well with hooks

[mzjp2]

Description

Pipline registry for hooks

All Kedro CLI commands go through kedro/framework/cli/cli.py::KedroCLI whose __init__ runs bootstrap_project. Kedro's bootstrap_project reads the users project metadata and imports the user settings.py file.

The users settings.py file (at least in modern Kedro versions) by default imports ProjectHooks from the users hooks.py file.

If users choose to register their pipelines via the register_pipelines hooks inside ProjectHooks class (or any other class inside hooks.py) and import their pipelines at the module-level of hooks.py to use them inside register_pipelines, this triggers a chain import from bootstrap_project that imports settings.py -> hooks.py -> imported_pipeline -> imported_pipeline/pipeline.py -> imported_pipeline/nodes.py -> third_party_library.

This is fine for pretty much all CLI commands (though can contribute to slowing CLI commands down for large projects that have many imports, for CLI commands that don't necessarily need the whole project imported to do something) since having the projects requirements installed is a sensible pre-requisite to running any of them (e.g kedro run).

However, the exception to that is kedro install and kedro build-reqs which are presumably written to help users install their requirements. That is, the starting position of using those commands is that the users don't have their project requirements installed.

Third party hooks

If users want to register a third-party hook that doesn't register itself automatically, their settings.py looks something like:

from third_party_hook import ThirdPartyHook

hooks = (ProjectHooks(), ThirdPartyHook())

which kedro install imports and runs and fails at if third_party_hook isn't already installed.

Steps to Reproduce

kedro new --starter=pandas-iris

Put a import tensorflow or import non_existent_library in data_engineering/nodes.py and then register the data engineering pipeline (or really just import it) at the top-level of your hooks.py and try running kedro install.

Expected Result

kedro install should execute and install the project requirements.

Actual Result

kedro install fails with an import error.

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• Kedro version used (pip show kedro or kedro -V): Kedro 0.17.4
• Python version used (python -V): Python 3.7.11
• Operating system and version: MacOS Mojave

[limdauto]

Related issue in starters repo: kedro-org/kedro-starters#38. So this is not necessarily just about hooks but rather kedro install needs to evaluate settings.py and any import there from third-party dependency will break the command.

[ignacioparicio]

I'm closing this issue as it is now solved in master as part of [KED-2768] Fix kedro install flow (#1203). Thank you @mzjp2 for the amazing write-up and please shout is you'd like to discuss this further!