SinaraML Creating new pipeline tutorial

Preamble

In this tutorial you will learn what is the Pipeline Step Template and how to use it to create new ML Pipeline

Important

Following commands should be executed in the running SinaraML server terminal.

Prerequisites

Configured and running SinaraML Server.

Pipeline Step Template

Pipeline step template is a starting point to create your own pipeline step. You should use it to create new pipelines. Creating pipeline from examples are not recommended since examples can be outdated and use old version if the SinaraML Library. Pipeline Step template on other hand always kept up to date.

Tip

If you need to use code from examples, then copy individual cells from example notebooks.

Create new ML Pipeline

Pipeline represents series of pipeline steps or just steps connected together by inputs and outputs. Pipeline step placed in a single git repository named as {pipeline name}-{step name}. To create single step you should clone Pipeline Step Template repository with new name and push it to new git repo.

For pipeline named my_pipeline and step named my_first_step Clone step template with following command:

git clone --recursive --depth 1 https://github.com/4-DS/pipeline-step_template.git my_pipeline-my_first_step

Step will be created in the new my_pipeline-my_first_step directory.

Now you can open step directory and configure step:

1. delete file prepare_data_for_template.ipynb
2. rename file do_step.ipynb to actual substep name
3. open /params/step_params.json file for editing and change substep_name field value - it should be equal to the new name of the substep
4. open renamed substep notebook and configure substep interface
5. you are ready to develop substep code
6. debug substep code in interactive mode
7. run pipeline step in job mode by running following command in terminal:

python step.dev.py

Repeat step creation procedure from Step Template for all steps in your new pipeline.