Performance analysis and smart re-running of pipelines

[merelcht]

Description

Kedro currently doesn't offer any options to analyse the performance of pipelines. Additionally, our users have flagged that they would like to be able to re-run only parts of their pipeline.

Implementation ideas

• Stats on performance of pipelines
• Record Kedro run metadata and use it for suggesting pipeline optimisations
• Make Kedro run faster
• Rerun Kedro Pipeline should be smarter. Only re-run necessary data "kedro run -xxx"

Questions

[noklam]

I think it's better to further split this into two issues, but I will leave my comments for both topics.

We have something like PipelineMonitoringHook in our docs. It requires some infrastructure and it's not easy to set up by regular users.

Pipeline Statistic

How would these pipeline stats be useful?

1. Just having a summary of the statistic could be very useful to understanding the pipeline performance - instead of parsing the log by yourself
2. In addition, we could add visualization on kedro-viz side to show where's the bottleneck and help user to optimize their pipeline
3. The pipeline statistic could be useful for ParallelRunner or something similar. Currently, the workload is distributed naively, but not every node is equal.

Smarter way to re-run the pipeline

Similar to the Pipeline's run_only_missing, but a more sophisticated one. During development, it's common that you are working on one particular node and you just need to refresh one node(or a few dependent nodes). We can so some back-tracking.

Alternative

Currently, it requires users to figure out which nodes are not necessary, and do kedro run --from-nodes to skip unnecessary computation

Summary

One key realization of this change is that Run need to have memory. To optimize runtime performance, it needs to know how it is run previously. To re-run the pipeline in a smart way, it needs to know the previous run(s) and figure out what's the minimal computation.

[noklam]

Related Issue:

• Draft Pull Request : Add incremental run method #2005

[astrojuanlu]

This is a very frequent question actually, will try to collect more evidence for it going forward.

There's different things when considering performance, namely (1) execution time, and (2) RAM usage. There are different tools for each of these purposes, so most likely we would need dedicated efforts.

I think execution time is probably the most urgent one. This is how I used pyinstrument #3033 (comment)