Skip to content

Commit

Permalink
Merge branch 'pipeline-performance-test' of github.com:kedro-org/kedr…
Browse files Browse the repository at this point in the history
…o into pipeline-performance-test
  • Loading branch information
lrcouto committed Sep 23, 2024
2 parents 60f06ad + e6ec50f commit 3399c37
Show file tree
Hide file tree
Showing 6 changed files with 25 additions and 3 deletions.
3 changes: 3 additions & 0 deletions RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@
## Bug fixes and other changes
* Fixed bug where using dataset factories breaks with `ThreadRunner`.
* Fixed template projects example tests.
* Made credentials loading consistent between `KedroContext._get_catalog()` and `resolve_patterns` so that both us
e `_get_config_credentials()`

## Breaking changes to the API
* Removed `ShelveStore` to address a security vulnerability.
Expand All @@ -21,6 +23,7 @@
* [Puneet](https://github.com/puneeter)
* [ethanknights](https://github.com/ethanknights)
* [Manezki](https://github.com/Manezki)
* [MigQ2](https://github.com/MigQ2)

# Release 0.19.8

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
9 changes: 8 additions & 1 deletion docs/source/nodes_and_pipelines/slice_a_pipeline.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
# Slice a pipeline

Sometimes it is desirable to run a subset, or a 'slice' of a pipeline's nodes. In this page, we illustrate the programmatic options that Kedro provides. You can also use the [Kedro CLI to pass parameters to `kedro run`](../development/commands_reference.md#run-the-project) command and slice a pipeline.
Sometimes it is desirable to run a subset, or a 'slice' of a pipeline's nodes. There are two primary ways to achieve this:


1. **Visually through Kedro-Viz:** This approach allows you to visually choose and slice pipeline nodes, which then generates a run command for executing the slice within your Kedro project. Detailed steps on how to achieve this are available in the Kedro-Viz documentation: [Slice a Pipeline](https://docs.kedro.org/projects/kedro-viz/en/stable/slice_a_pipeline.html).

![](../meta/images/slice_pipeline_kedro_viz.gif)

2. **Programmatically with the Kedro CLI.** You can also use the [Kedro CLI to pass parameters to `kedro run`](../development/commands_reference.md#run-the-project) command and slice a pipeline. In this page, we illustrate the programmatic options that Kedro provides.

Let's look again at the example pipeline from the [pipeline introduction documentation](./pipeline_introduction.md#how-to-build-a-pipeline), which computes the variance of a set of numbers:

Expand Down
2 changes: 1 addition & 1 deletion kedro/framework/cli/catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ def resolve_patterns(metadata: ProjectMetadata, env: str) -> None:
context = session.load_context()

catalog_config = context.config_loader["catalog"]
credentials_config = context.config_loader.get("credentials", None)
credentials_config = context._get_config_credentials()
data_catalog = DataCatalog.from_config(
catalog=catalog_config, credentials=credentials_config
)
Expand Down
13 changes: 12 additions & 1 deletion kedro/io/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,18 @@
VERSION_KEY = "version"
HTTP_PROTOCOLS = ("http", "https")
PROTOCOL_DELIMITER = "://"
CLOUD_PROTOCOLS = ("s3", "s3n", "s3a", "gcs", "gs", "adl", "abfs", "abfss", "gdrive")
CLOUD_PROTOCOLS = (
"abfs",
"abfss",
"adl",
"gcs",
"gdrive",
"gs",
"oss",
"s3",
"s3a",
"s3n",
)


class DatasetError(Exception):
Expand Down
1 change: 1 addition & 0 deletions tests/framework/cli/test_catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -543,6 +543,7 @@ def test_catalog_resolve(
"catalog": fake_catalog_config,
"credentials": fake_credentials_config,
}
mocked_context._get_config_credentials.return_value = fake_credentials_config
mocked_context.catalog = DataCatalog.from_config(
catalog=fake_catalog_config, credentials=fake_credentials_config
)
Expand Down

0 comments on commit 3399c37

Please sign in to comment.