Skip to content

Commit

Permalink
(blog) Your Private App Store for Data Pipelines (#353)
Browse files Browse the repository at this point in the history
* blog draft

* Update 2023-07-12-your-private-app-store-for-data-pipelines.png

* Update 2023-07-12-your-private-app-store-for-data-pipelines.png
  • Loading branch information
anna-geller authored Jul 12, 2023
1 parent f78b530 commit d58124a
Show file tree
Hide file tree
Showing 3 changed files with 65 additions and 0 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
---
title: "Your Private App Store for Data Pipelines"
description: "New feature: Data Orchestration Blueprints to Kickstart Your Next Flow"
date: 2023-07-12T14:00:00
category: Solutions
author:
name: Anna Geller
image: "ageller"
image: /blogs/2023-07-12-your-private-app-store-for-data-pipelines.png
---


Data professionals are exposed to a growing number of tools in the data, AI, and engineering ecosystem. If you combined the logo of every tool used by a typical data team, you would likely get a picture resembling an app store. Amidst a wide variety of such apps, it can be challenging to integrate *existing* applications with *new* ones while retaining institutional knowledge and staying adaptable to change. We’ve introduced a **new feature** in the Kestra UI called **Blueprints** to help you implement repeatable workflow patterns and share knowledge within your organization.

## Why Blueprints

### The first step is always the hardest

Blueprints are helpful because they make it **easier to get started**. Often the biggest hurdle to solving a challenging problem is just taking the first step. Blueprints help by providing a scaffold (…blueprint?) for a specific use case or integration.

### More time for work that is actually fun and rewarding

Many data management tasks are increasingly **standardized**. It doesn’t make much sense to write glue code yet again only to concurrently trigger several data ingestion syncs, then start a dbt or Spark transformation job, configure alert notifications, or invoke some process via a third-party API and report on their completion status. Instead of spending time writing imperative code to glue those standardized tasks together, you can just declare them in a YAML config, validate they work as expected, and move on to the next tasks quickly. Less time spent on *boring* glue code, more time left for *fun* code that adds value to your business. Blueprints further optimize that process by providing ready-to-use examples to help you iterate even faster.

### A better place for docs than docs

Blueprints serve as a natural extension to the docs. Instead of searching online resources for configuration or real-life examples, they are now available in a place where you need them most — right next to your code editor.

## What are Blueprints

Blueprints is a curated, organized, and searchable catalog of ready-to-use examples designed to help you kickstart your workflow. Each Blueprint combines code and documentation and can be assigned several tags for organization and discoverability.

All blueprints are validated and documented so that they *just work*. You can easily customize and integrate them into your new or existing flows with a single click on the “Use” button.

![blueprintsDemo](/blogs/2023-07-12-your-private-app-store-for-data-pipelines/blueprintsDemo.png)


## Getting Started with Blueprints

First, start Kestra locally by downloading the [docker-compose file](https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml) and running `docker compose up -d`. Then [navigate to the UI](http://localhost:8080/).

Blueprints are accessible from two places in the UI:

1. The left navigation **sidebar**
2. A dedicated **tab** in the code editor named *“Source and blueprints”,* showing your source code and Blueprints side by side.

![blueprint2](/blogs/2023-07-10-release-0-10-blueprints-worker-groups-scripts/blueprint2.png)

Once you are on the Blueprints page, you can:

1. **Search** blueprints for a specific use case or integration, e.g., Snowflake, BigQuery, DuckDB, Slack, ETL, ELT, Pandas, GPU, Git, Python, Docker, Redis, MongoDB, dbt, Airbyte, Fivetran, etc.
2. **Filter** by a tag, e.g., filter for *Docker*, to see various ways to run containers in your flow. Or filter for *Notifications* to see several options for configuring alerts on success or failure.

## How to contribute to Blueprints

Blueprints, available out of the box, cover only the most common and standardized tasks. There are two ways to extend Blueprints with more nuanced workflow examples tailored to your needs.

1. **Community Blueprints:** use the [dedicated issue template](https://github.com/kestra-io/kestra/issues/new?assignees=&labels=blueprint&projects=&template=blueprint.yml) on GitHub to contribute a new Blueprint. We’ll then either add your Blueprint right away or leave comments to discuss and refine the flow together.
2. **Organization Blueprints:** while Community Blueprints are visible to everyone, the Enterprise Edition allows adding private fine-grained Blueprints available only to your organization. You can use them to share, centralize, and document commonly used workflows in your team. The combination of custom plugins and Organization Blueprints covers a wide range of integrations and workflow patterns tailored to your needs.

Blueprints facilitate fast feedback loops and knowledge sharing. No need to wait for the next release or implement a CI/CD workflow. Once the Blueprint is published, it’s available within the UI right away. No upgrade or restart is required. This way, you can quickly share institutional knowledge with your team. By combining custom plugins and blueprints, you can effectively build a private App Store for your data workflows.

## Next steps

We’re constantly expanding the collection of Blueprints. We hope you’ll enjoy using Blueprints as much as we enjoy creating them. Which use cases and integrations would you like to see covered next? If you have any suggestions about new Blueprints or improvements to the existing ones, submit a [GitHub issue](https://github.com/kestra-io/kestra/issues/new?assignees=&labels=blueprint&projects=&template=blueprint.yml) or ask us [via Slack](https://kestra.io/slack).
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

1 comment on commit d58124a

@vercel
Copy link

@vercel vercel bot commented on d58124a Jul 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Successfully deployed to the following URLs:

kestra-io – ./

kestra-io-kestra.vercel.app
kestra-io-git-main-kestra.vercel.app
kestra-io.vercel.app

Please sign in to comment.