Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

blog post dremio partnership #1039

Merged
merged 3 commits into from
Mar 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions content/blogs/2024-03-14-dremio-partnership.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
title: "Announcing Dremio Partnership: Orchestrate Your Data Lakehouse with Kestra"
description: "We are proud to announce our partnership with Dremio, a data lakehouse platform that simplifies big data analytics."
date: 2024-03-14T08:00:00
category: Company News
author:
name: Emmanuel Darras
image: "edarras"
image: /blogs/2024-03-14-dremio-partnership.jpg
---

**Today, we’re thrilled to announce a strategic partnership between [Kestra](https://github.com/kestra-io/kestra) and [Dremio](https://www.dremio.com/) to simplify data workflows and accelerate time-to-delivery for data teams! This collaboration will empower data professionals to leverage the combined strengths of Kestra's declarative workflow engine and Dremio's Data lakehouse capabilities.**

## About Dremio

Dremio is a data lakehouse platform that simplifies big data analytics. It allows you to directly access data from various sources, such as Postgres, S3 and Azure Data Lake Storage, without needing to copy or move the data. Its key features include a fast query engine, a semantic layer to help manage and share data, a catalog for Iceberg tables, and reflections — a market-leading query acceleration technology that delivers sub-second query response times. Designed to work with SQL and common BI tools, Dremio provides self-service analytics and data management for BI workloads with the best price performance and lowest cost.

## Kestra's Integration with Dremio

Integrating Dremio with Kestra offers a powerful solution for analytical workflows and scenarios involving complex data transformations and business-critical operations.

Kestra provides a declarative workflow engine that orchestrates Dremio's data access and management capabilities. This allows data professionals to build complex workflows without intricate coding, while easily querying data directly from various sources through Dremio's lakehouse architecture.

Together, we reduce unnecessary coding overhead and eliminate the need for complex ETL pipelines. The intuitive interfaces focus on the user experience, and the rich plugin ecosystems of both tools further amplify their synergy, accelerating development cycles and empowering data professionals to deliver insights faster.

![plugin](/blogs/2024-03-14-dremio-partnership/schema.png)

### Practical Application of the Kestra-Dremio Integration

As part of our partnership, we've recently launched the Dremio and Arrow Flight SQL plugins to orchestrate your data lakehouse workflows. These plugins empower users to automate complex data processes, including:

- Trigger a workflow when there is an anomaly in the Dremio database or when certain KPIs fall outside of an expected range.
- Run SQL queries against Dremio databases to fetch data for downstream processing in Python, R, or other languages supported by Kestra.
- Query data from Dremio and process it in a Python script


## What's Next

Kestra and Dremio can help data practitioners to be more productive by simplifying data access, automating data workflows, and improving data quality.

If you want to learn more about Kestra and Dremio capabilities, you can read this article about [Data lakehouse orchestration](https://kestra.io/blogs/2023-12-07-dremio-kestra-integration) with Kestra, Dremio, dbt and Python.

You can also kickstart your Kestra & Dremio journey with our [Community Blueprints](https://kestra.io/blueprints?page=1&size=24&q=dremio).


If you have any questions, reach out via [Slack](https://kestra.io/slack) or open a [GitHub issue](https://github.com/kestra-io/kestra).

If you like the project, give us a [GitHub star](https://github.com/kestra-io/kestra) and join [the community](https://kestra.io/slack).
Binary file added public/blogs/2024-03-14-dremio-partnership.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.