From ec3e49db9b88ac74048b530843e302377ce467c0 Mon Sep 17 00:00:00 2001 From: Jo Stichbury Date: Wed, 11 Oct 2023 11:36:00 +0100 Subject: [PATCH 1/4] Move project setup to advanced section Signed-off-by: Jo Stichbury --- docs/source/index.rst | 11 +++++------ docs/source/kedro_project_setup/index.md | 2 +- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index 5850f15f76..30d7d483ca 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -92,11 +92,6 @@ Welcome to Kedro's documentation! :maxdepth: 2 :caption: Kedro projects - kedro_project_setup/index.md - -.. toctree:: - :maxdepth: 2 - configuration/index.md .. toctree:: @@ -113,6 +108,11 @@ Welcome to Kedro's documentation! :maxdepth: 2 :caption: Advanced usage + kedro_project_setup/index.md + + .. toctree:: + :maxdepth: 2 + extend_kedro/index.md .. toctree:: @@ -120,7 +120,6 @@ Welcome to Kedro's documentation! hooks/index.md - .. toctree:: :maxdepth: 2 diff --git a/docs/source/kedro_project_setup/index.md b/docs/source/kedro_project_setup/index.md index 2b3f882950..609b4d32b8 100644 --- a/docs/source/kedro_project_setup/index.md +++ b/docs/source/kedro_project_setup/index.md @@ -1,4 +1,4 @@ -# Kedro project setup +# Project setup ```{toctree} :maxdepth: 1 From b8be11c48f3106a0d913fdc12cc565e2f58dbb24 Mon Sep 17 00:00:00 2001 From: Jo Stichbury Date: Wed, 11 Oct 2023 13:25:13 +0100 Subject: [PATCH 2/4] =?UTF-8?q?Update=20index.rst=20=F0=9F=A4=A6=E2=80=8D?= =?UTF-8?q?=E2=99=80=EF=B8=8F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/source/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index 30d7d483ca..6a73aaddfd 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -110,7 +110,7 @@ Welcome to Kedro's documentation! kedro_project_setup/index.md - .. toctree:: +.. toctree:: :maxdepth: 2 extend_kedro/index.md From 15be279bb6549055045ad02159c52ce72190b5fb Mon Sep 17 00:00:00 2001 From: Jo Stichbury Date: Thu, 12 Oct 2023 10:19:15 +0100 Subject: [PATCH 3/4] Remove secret scan kryptonite Signed-off-by: Jo Stichbury --- .../add_kedro_to_a_notebook.ipynb | 112 +++++++++--------- .../add_kedro_to_a_notebook.md | 7 +- 2 files changed, 59 insertions(+), 60 deletions(-) diff --git a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb index 89bd373bca..f5c95ddd8b 100644 --- a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb +++ b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb @@ -2,7 +2,15 @@ "cells": [ { "cell_type": "markdown", - "id": "51a33241", + "id": "c60fadc1", + "metadata": {}, + "source": [ + " " + ] + }, + { + "cell_type": "markdown", + "id": "8e8161b9", "metadata": {}, "source": [ "# Add Kedro features to a notebook\n", @@ -11,7 +19,7 @@ "\n", "It starts with a notebook example which does NOT use Kedro. It then explains how to convert portions of the code to use Kedro features while remaining runnable within a notebook. For that part of the example, you need to have [set up Kedro](../../get_started/install.md).\n", "\n", - ">**NOTE**: If you want to experiment with the code in a notebook, you can find it in the [`notebook-example` folder on GitHub](https://github.com/kedro-org/kedro/tree/4727aab2f5ab0210f264f5bfb477b66c6327fbbe/docs/source/notebooks_and_ipython/notebook-example). Be sure to download the entire folder, or clone the entire repo, because [`add_kedro_to_spaceflights_notebook.ipynb`](https://github.com/kedro-org/kedro/blob/4727aab2f5ab0210f264f5bfb477b66c6327fbbe/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb) relies upon files stored in the `notebook-example` folder.\n", + ">**NOTE**: If you want to experiment with the code in a notebook, you can find it in the [`notebook-example` folder on GitHub](https://github.com/kedro-org/kedro/tree/main/docs/source/notebooks_and_ipython/notebook-example). Be sure to download the entire folder, or clone the entire repo, because the `add_kedro_to_spaceflights_notebook.ipynb` notebook relies upon files stored in the `notebook-example` folder.\n", "\n", "## Kedro spaceflights\n", "\n", @@ -24,7 +32,7 @@ }, { "cell_type": "markdown", - "id": "a74a1d51", + "id": "0ab066eb", "metadata": {}, "source": [ "### The notebook example\n", @@ -34,7 +42,7 @@ { "cell_type": "code", "execution_count": null, - "id": "192ff93b", + "id": "2d1e2829", "metadata": {}, "outputs": [], "source": [ @@ -48,7 +56,7 @@ { "cell_type": "code", "execution_count": null, - "id": "42b3e71b", + "id": "eae11bc4", "metadata": {}, "outputs": [], "source": [ @@ -71,7 +79,7 @@ { "cell_type": "code", "execution_count": null, - "id": "b1714db3", + "id": "3a68464f", "metadata": {}, "outputs": [], "source": [ @@ -104,7 +112,7 @@ { "cell_type": "code", "execution_count": null, - "id": "3de887cc", + "id": "21d1d666", "metadata": {}, "outputs": [], "source": [ @@ -117,7 +125,7 @@ }, { "cell_type": "markdown", - "id": "562acc6b", + "id": "7c688dcc", "metadata": {}, "source": [ "## Use Kedro for data processing\n", @@ -149,7 +157,7 @@ { "cell_type": "code", "execution_count": null, - "id": "03535931", + "id": "efcc49c0", "metadata": {}, "outputs": [], "source": [ @@ -174,7 +182,7 @@ }, { "cell_type": "markdown", - "id": "b340ab9d", + "id": "72d01559", "metadata": {}, "source": [ "The rest of the spaceflights notebook code for data processing and model evaluation from above can now run as before.\n", @@ -188,7 +196,7 @@ { "cell_type": "code", "execution_count": null, - "id": "f581d2c4", + "id": "5fcef8f7", "metadata": {}, "outputs": [], "source": [ @@ -197,7 +205,7 @@ }, { "cell_type": "markdown", - "id": "d0993fb8", + "id": "97670a51", "metadata": {}, "source": [ "[Good software engineering practice](https://towardsdatascience.com/five-software-engineering-principles-for-collaborative-data-science-ab26667a311) suggests that we extract *‘magic numbers’* into named constants. These could be defined at the top of a file or in a utility file, in a format such as yaml." @@ -205,7 +213,7 @@ }, { "cell_type": "markdown", - "id": "51afaf8d", + "id": "a512a5ae", "metadata": {}, "source": [ "```yaml\n", @@ -222,7 +230,7 @@ { "cell_type": "code", "execution_count": null, - "id": "34f2ab2c", + "id": "672c2869", "metadata": {}, "outputs": [], "source": [ @@ -235,7 +243,7 @@ { "cell_type": "code", "execution_count": null, - "id": "96d8b2d6", + "id": "9861a8b9", "metadata": {}, "outputs": [], "source": [ @@ -246,7 +254,7 @@ { "cell_type": "code", "execution_count": null, - "id": "fdf3a16e", + "id": "9ef0ca25", "metadata": {}, "outputs": [], "source": [ @@ -265,7 +273,7 @@ { "cell_type": "code", "execution_count": null, - "id": "eae818a1", + "id": "236c087f", "metadata": {}, "outputs": [], "source": [ @@ -276,7 +284,7 @@ { "cell_type": "code", "execution_count": null, - "id": "ed65fee4", + "id": "a1fea8f2", "metadata": {}, "outputs": [], "source": [ @@ -289,7 +297,7 @@ }, { "cell_type": "markdown", - "id": "b632a2d6", + "id": "3f217026", "metadata": {}, "source": [ "The rest of the model evaluation code can now run as before." @@ -298,7 +306,7 @@ { "cell_type": "code", "execution_count": null, - "id": "e4c301ae", + "id": "d2387d66", "metadata": {}, "outputs": [], "source": [ @@ -315,7 +323,7 @@ }, { "cell_type": "markdown", - "id": "c7317607", + "id": "2e848f6f", "metadata": {}, "source": [ "### Use a configuration file for all \"magic values\"\n", @@ -343,7 +351,7 @@ { "cell_type": "code", "execution_count": null, - "id": "4338a498", + "id": "ecd7f57f", "metadata": {}, "outputs": [], "source": [ @@ -359,7 +367,7 @@ { "cell_type": "code", "execution_count": null, - "id": "8774448c", + "id": "8a41b6d7", "metadata": {}, "outputs": [], "source": [ @@ -370,7 +378,7 @@ { "cell_type": "code", "execution_count": null, - "id": "2edb954f", + "id": "a5a0210c", "metadata": {}, "outputs": [], "source": [ @@ -383,7 +391,7 @@ }, { "cell_type": "markdown", - "id": "8cc56162", + "id": "01fa2ce7", "metadata": {}, "source": [ "The rest of the model evaluation code can now run as before." @@ -392,7 +400,7 @@ { "cell_type": "code", "execution_count": null, - "id": "94d8c207", + "id": "c5f2c766", "metadata": {}, "outputs": [], "source": [ @@ -409,7 +417,7 @@ }, { "cell_type": "markdown", - "id": "3713621c", + "id": "94917c25", "metadata": {}, "source": [ "## Use Kedro configuration\n", @@ -422,7 +430,7 @@ { "cell_type": "code", "execution_count": null, - "id": "2642417f", + "id": "12259d26", "metadata": {}, "outputs": [], "source": [ @@ -434,7 +442,7 @@ { "cell_type": "code", "execution_count": null, - "id": "a2df4ee3", + "id": "ae0e0c83", "metadata": {}, "outputs": [], "source": [ @@ -448,7 +456,7 @@ { "cell_type": "code", "execution_count": null, - "id": "5053a3cf", + "id": "56da68cb", "metadata": {}, "outputs": [], "source": [ @@ -461,7 +469,7 @@ }, { "cell_type": "markdown", - "id": "4ef0edec", + "id": "74f90788", "metadata": {}, "source": [ "The rest of the model evaluation code can now run as before." @@ -470,7 +478,7 @@ { "cell_type": "code", "execution_count": null, - "id": "973987b8", + "id": "d6dd3e52", "metadata": {}, "outputs": [], "source": [ @@ -487,7 +495,7 @@ }, { "cell_type": "markdown", - "id": "83434503", + "id": "28d643c2", "metadata": {}, "source": [ "### Use Kedro's configuration loader to load the Data Catalog\n", @@ -497,7 +505,7 @@ { "cell_type": "code", "execution_count": null, - "id": "905d4644", + "id": "492b6181", "metadata": {}, "outputs": [], "source": [ @@ -520,7 +528,7 @@ }, { "cell_type": "markdown", - "id": "14acf4c7", + "id": "317293da", "metadata": {}, "source": [ "It's also possible to use Kedro's `OmegaConfigLoader`configuration loader to initialise the Data Catalog.\n", @@ -531,7 +539,7 @@ { "cell_type": "code", "execution_count": null, - "id": "0b8b5c08", + "id": "df31cfaa", "metadata": {}, "outputs": [], "source": [ @@ -554,7 +562,7 @@ }, { "cell_type": "markdown", - "id": "8101818e", + "id": "a653ae92", "metadata": {}, "source": [ "## Where next?\n", @@ -567,7 +575,7 @@ { "cell_type": "code", "execution_count": null, - "id": "97482d33", + "id": "cc48efc8", "metadata": {}, "outputs": [], "source": [ @@ -596,7 +604,7 @@ { "cell_type": "code", "execution_count": null, - "id": "4e72c93f", + "id": "78c537ff", "metadata": {}, "outputs": [], "source": [ @@ -647,7 +655,7 @@ { "cell_type": "code", "execution_count": null, - "id": "7fbf8c6c", + "id": "f47c4177", "metadata": {}, "outputs": [], "source": [ @@ -657,7 +665,7 @@ }, { "cell_type": "markdown", - "id": "334eee68", + "id": "1084a880", "metadata": {}, "source": [ "In truth, this code is not much more maintainable than previous versions.\n", @@ -670,7 +678,7 @@ { "cell_type": "code", "execution_count": null, - "id": "b71f452e", + "id": "6e98fb75", "metadata": {}, "outputs": [], "source": [ @@ -758,7 +766,7 @@ { "cell_type": "code", "execution_count": null, - "id": "222ded65", + "id": "9a86c8d3", "metadata": {}, "outputs": [], "source": [ @@ -779,7 +787,7 @@ }, { "cell_type": "markdown", - "id": "a524adab", + "id": "c47834b9", "metadata": {}, "source": [ "And that's it. The notebook code has been refactored into a series of functions. Let's reproduce it all in one big notebook cell for reference. Compare it to the notebook code at the top of this page that began this example." @@ -788,7 +796,7 @@ { "cell_type": "code", "execution_count": null, - "id": "1ed899e0", + "id": "19feea6d", "metadata": {}, "outputs": [], "source": [ @@ -912,24 +920,12 @@ ], "metadata": { "jupytext": { - "formats": "ipynb,md" + "formats": "md,ipynb" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.16" } }, "nbformat": 4, diff --git a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md index b682e318fe..1d47468a49 100644 --- a/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md +++ b/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.md @@ -1,7 +1,7 @@ --- jupyter: jupytext: - formats: ipynb,md + formats: md,ipynb text_representation: extension: .md format_name: markdown @@ -13,13 +13,16 @@ jupyter: name: python3 --- + + + # Add Kedro features to a notebook This page describes how to add Kedro features incrementally to a notebook. It starts with a notebook example which does NOT use Kedro. It then explains how to convert portions of the code to use Kedro features while remaining runnable within a notebook. For that part of the example, you need to have [set up Kedro](../../get_started/install.md). ->**NOTE**: If you want to experiment with the code in a notebook, you can find it in the [`notebook-example` folder on GitHub](https://github.com/kedro-org/kedro/tree/4727aab2f5ab0210f264f5bfb477b66c6327fbbe/docs/source/notebooks_and_ipython/notebook-example). Be sure to download the entire folder, or clone the entire repo, because [`add_kedro_to_spaceflights_notebook.ipynb`](https://github.com/kedro-org/kedro/blob/4727aab2f5ab0210f264f5bfb477b66c6327fbbe/docs/source/notebooks_and_ipython/notebook-example/add_kedro_to_a_notebook.ipynb) relies upon files stored in the `notebook-example` folder. +>**NOTE**: If you want to experiment with the code in a notebook, you can find it in the [`notebook-example` folder on GitHub](https://github.com/kedro-org/kedro/tree/main/docs/source/notebooks_and_ipython/notebook-example). Be sure to download the entire folder, or clone the entire repo, because the `add_kedro_to_spaceflights_notebook.ipynb` notebook relies upon files stored in the `notebook-example` folder. ## Kedro spaceflights From efeaba2d4e643fb4bb501f29e5118bff00823b11 Mon Sep 17 00:00:00 2001 From: Ankita Katiyar Date: Thu, 12 Oct 2023 12:16:08 +0100 Subject: [PATCH 4/4] Ignore lint_only by merge gatekeeper Signed-off-by: Ankita Katiyar --- .github/workflows/merge-gatekeeper.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/merge-gatekeeper.yml b/.github/workflows/merge-gatekeeper.yml index cccbe479b4..6d3e82eac9 100644 --- a/.github/workflows/merge-gatekeeper.yml +++ b/.github/workflows/merge-gatekeeper.yml @@ -24,4 +24,4 @@ jobs: token: ${{ secrets.GITHUB_TOKEN }} timeout: 3600 interval: 30 - ignored: 'runner / vale,ci/circleci: win_e2e_tests-3.7,ci/circleci: win_pip_compile-3.9,ci/circleci: win_e2e_tests-3.9,ci/circleci: win_pip_compile-3.8,ci/circleci: lint-3.7,ci/circleci: win_pip_compile-3.7,ci/circleci: pip_compile-3.7,ci/circleci: e2e_tests-3.7,ci/circleci: win_unit_tests-3.7,ci/circleci: win_unit_tests-3.9,ci/circleci: e2e_tests-3.8,ci/circleci: win_unit_tests-3.10,ci/circleci: win_pip_compile-3.10,ci/circleci: win_unit_tests-3.8,ci/circleci: e2e_tests-3.9,ci/circleci: unit_tests-3.10,ci/circleci: unit_tests-3.8,ci/circleci: e2e_tests-3.10,ci/circleci: lint-3.8,ci/circleci: unit_tests-3.9,ci/circleci: unit_tests-3.7,ci/circleci: win_e2e_tests-3.10,ci/circleci: pip_compile-3.8,ci/circleci: pip_compile-3.10,ci/circleci: win_e2e_tests-3.8,ci/circleci: lint-3.9,ci/circleci: pip_compile-3.9,ci/circleci: lint-3.10,build_code,ci/circleci: check-updated-files,regular' + ignored: 'lint_only, runner / vale,ci/circleci: win_e2e_tests-3.7,ci/circleci: win_pip_compile-3.9,ci/circleci: win_e2e_tests-3.9,ci/circleci: win_pip_compile-3.8,ci/circleci: lint-3.7,ci/circleci: win_pip_compile-3.7,ci/circleci: pip_compile-3.7,ci/circleci: e2e_tests-3.7,ci/circleci: win_unit_tests-3.7,ci/circleci: win_unit_tests-3.9,ci/circleci: e2e_tests-3.8,ci/circleci: win_unit_tests-3.10,ci/circleci: win_pip_compile-3.10,ci/circleci: win_unit_tests-3.8,ci/circleci: e2e_tests-3.9,ci/circleci: unit_tests-3.10,ci/circleci: unit_tests-3.8,ci/circleci: e2e_tests-3.10,ci/circleci: lint-3.8,ci/circleci: unit_tests-3.9,ci/circleci: unit_tests-3.7,ci/circleci: win_e2e_tests-3.10,ci/circleci: pip_compile-3.8,ci/circleci: pip_compile-3.10,ci/circleci: win_e2e_tests-3.8,ci/circleci: lint-3.9,ci/circleci: pip_compile-3.9,ci/circleci: lint-3.10,build_code,ci/circleci: check-updated-files,regular'