Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for how plugin CLI commands can be made lazy loaded #4027

Merged
merged 3 commits into from
Jul 24, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 48 additions & 3 deletions docs/source/extend_kedro/plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
pass


@commands.command()
@commands.command(name="to_json")
@click.pass_obj
def to_json(metadata):
"""Display the pipeline in JSON format"""
Expand All @@ -48,7 +48,6 @@

The `click Group` will be merged into the main CLI Group. In the process, the options on the group are lost, as is any processing that was done as part of its callback function.


## Project context

When they run, plugins may request information about the current project by creating a session and loading its context:
Expand Down Expand Up @@ -78,7 +77,53 @@

## Suggested command convention

We use the following command convention: `kedro <plugin-name> <command>`, with `kedro <plugin-name>` acting as a top-level command group. This is our suggested way of structuring your plugin bit it is not necessary for your plugin to work.
We use the following command convention: `kedro <plugin-name> <command>`, with `kedro <plugin-name>` acting as a top-level command group. This is our suggested way of structuring your plugin but it is not necessary for your plugin to work.

## Advanced: Lazy loading of plugin commands

If you are developing a plugin with a large set of CLI commands or with certain large libraries that are slow to import but are used by a small subset of commands, consider using lazy loading of these commands. This can significantly improve the performance of the plugin as well as the overall performance of Kedro CLI. To do this, you can follow [the instructions on the

Check notice on line 84 in docs/source/extend_kedro/plugins.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/extend_kedro/plugins.md#L84

[Kedro.sentencelength] Try to keep your sentence length to 30 words or fewer.
Raw output
{"message": "[Kedro.sentencelength] Try to keep your sentence length to 30 words or fewer.", "location": {"path": "docs/source/extend_kedro/plugins.md", "range": {"start": {"line": 84, "column": 1}}}, "severity": "INFO"}

Check warning on line 84 in docs/source/extend_kedro/plugins.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/extend_kedro/plugins.md#L84

[Kedro.weaselwords] 'significantly' is a weasel word!
Raw output
{"message": "[Kedro.weaselwords] 'significantly' is a weasel word!", "location": {"path": "docs/source/extend_kedro/plugins.md", "range": {"start": {"line": 84, "column": 221}}}, "severity": "WARNING"}

Check warning on line 84 in docs/source/extend_kedro/plugins.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/extend_kedro/plugins.md#L84

[Kedro.toowordy] 'overall' is too wordy
Raw output
{"message": "[Kedro.toowordy] 'overall' is too wordy", "location": {"path": "docs/source/extend_kedro/plugins.md", "range": {"start": {"line": 84, "column": 288}}}, "severity": "WARNING"}
`click` documentation on lazy loading of commands](https://click.palletsprojects.com/en/8.1.x/complex/#lazily-loading-subcommands). From Kedro 0.19.7, the [Kedro commands are declared as lazy loaded command groups](https://github.com/kedro-org/kedro/blob/main/kedro/framework/cli/cli.py) that you can use as a reference for the implementation.

Consider the previous example of the `kedrojson` plugin. Suppose the plugin has two commands, `kedro to_json pipelines` and `kedro to_json nodes`. The `to_json pipelines` command is used more frequently than the `to_json nodes` command and the `to_json nodes` command requires a large library to be imported. In this case, you can define your commands to be lazily loaded with delayed imports as follows:

Check warning on line 87 in docs/source/extend_kedro/plugins.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/extend_kedro/plugins.md#L87

[Kedro.weaselwords] 'lazily' is a weasel word!
Raw output
{"message": "[Kedro.weaselwords] 'lazily' is a weasel word!", "location": {"path": "docs/source/extend_kedro/plugins.md", "range": {"start": {"line": 87, "column": 359}}}, "severity": "WARNING"}

In `kedrojson/plugin.py`:
```python
import click
from kedro.framework.project import pipelines
from kedro.framework.cli.utils import LazyGroup

@click.group()
def commands():
pass

@commands.group(
name="to_json",
cls=LazyGroup,
lazy_subcommands={
"nodes": "kedrojson.plugin.nodes",
"pipelines": "kedrojson.plugin.pipelines"
}
)
def to_json():
"""Convert Kedro nodes and pipelines to JSON"""
pass

@click.command(name="nodes")
def nodes():
"""Convert Kedro nodes to JSON"""
import some_large_library
print("Converting nodes to JSON")
...

@click.command("pipelines")
def pipelines():
"""Convert Kedro pipelines to JSON"""
print("Converting pipelines to JSON")
...
```

The loading of the individual `nodes` and `pipelines` commands and subsequently the imports of the large libraries will be delayed until the respective commands are called.


## Hooks

Expand Down