-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
chore(docker): try a better startup process with docker-compose
- Loading branch information
1 parent
7ef9173
commit 796bf20
Showing
4 changed files
with
244 additions
and
37 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,24 +1,226 @@ | ||
| <p align="center"> | ||
| <a href="https://www.kestra.io"> | ||
| <img width="460" src="https://kestra.io/logo.svg" alt="Kestra workflow orchestrator" /> | ||
| </a> | ||
| </p> | ||
|
|
||
| # Kestra | ||
|  | ||
|  | ||
|  | ||
|  | ||
|  | ||
| [](https://codecov.io/gh/kestra-io/kestra) | ||
|  | ||
| <h1 align="center" style="border-bottom: none"> | ||
| Kestra, Infinitely scalable open source orchestration & scheduling platform. <br> | ||
| </h1> | ||
|
|
||
| <div align="center"> | ||
| <a href="/kestra-io/kestra/blob/develop/LICENSE"><img src="https://img.shields.io/github/license/kestra-io/kestra?style=flat-square" alt="License" /></a> | ||
| <a href="https://github.com/kestra-io/kestra/pulse"><img src="https://img.shields.io/github/commit-activity/m/kestra-io/kestra?style=flat-square" alt="Commits-per-month"></a> | ||
| <a href="/kestra-io/kestra/stargazers"><img src="https://img.shields.io/github/stars/kestra-io/kestra.svg?style=flat-square" alt="Github star" /></a> | ||
| <a href="/kestra-io/kestra/releases"><img src="https://img.shields.io/github/tag-pre/kestra-io/kestra.svg?style=flat-square" alt="Last Version" /></a> | ||
| <a href="https://hub.docker.com/r/kestra/kestra"><img src="https://img.shields.io/docker/pulls/kestra/kestra.svg?style=flat-square" alt="Docker pull" /></a> | ||
| <a href="https://artifacthub.io/packages/helm/kestra/kestra"><img src="https://img.shields.io/badge/Artifact%20Hub-kestra-417598?style=flat-square&logo=artifacthub" alt="Artifact Hub" /></a> | ||
| <a href="https://kestra.io"><img src="https://img.shields.io/badge/Website-kestra.io-192A4E?style=flat-square" alt="Kestra infinitely scalable orchestration and scheduling platform"></a> | ||
| <a href="https://discord.gg/5RgZmkW"><img src="https://img.shields.io/discord/903344083391631471?label=Discord&style=flat-square" alt="Discord"></a> | ||
| <a href="/kestra-io/kestra/discussions"><img src="https://img.shields.io/github/discussions/kestra-io/kestra?style=flat-square" alt="Github discussions"></a> | ||
| <a href="https://twitter.com/kestra_io"><img src="https://img.shields.io/twitter/follow/kestra_io?style=flat-square" alt="Twitter" /></a> | ||
| <a href="https://app.codecov.io/gh/kestra-io/kestra"><img src="https://img.shields.io/codecov/c/github/kestra-io/kestra?style=flat-square&token=It6L7BTaWK" alt="Code Cov" /></a> | ||
| <a href="/kestra-io/kestra/actions"><img src="https://img.shields.io/github/workflow/status/kestra-io/kestra/Main/develop?style=flat-square" alt="Github Actions" /></a> | ||
| </div> | ||
|
|
||
| <br /> | ||
|
|
||
| <p align="center"> | ||
| <img width="460" src="https://kestra.io/logo.svg" alt="Kestra workflow orchestrator" /> | ||
| <a href="https://kestra.io/"><b>Website</b></a> • | ||
| <a href="https://twitter.com/kestra_io"><b>Twitter</b></a> • | ||
| <a href="https://www.linkedin.com/company/kestra/"><b>Linked In</b></a> • | ||
| <a href="https://discord.gg/NMG39WKGth"><b>Discord</b></a> • | ||
| <a href="https://kestra.io/docs/"><b>Documentation</b></a> | ||
| </p> | ||
|
|
||
| > The modern, scalable orchestrator & scheduler open source platform. | ||
| <br /> | ||
|
|
||
| <p align="center"><img src="https://kestra.io/video.gif" alt="modern data orchestration and scheduling platform " width="640px" /></p> | ||
|
|
||
|
|
||
| ## What is Kestra ? | ||
| Kestra is an infinitely scalable orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines. | ||
|
|
||
| - 🔀 **Any kind of workflow**: Workflows can start simple and progress to more complex systems with branching, parallel, dynamic tasks, flow dependencies | ||
| - 🎓 **Easy to learn**: Flows are in simple, descriptive language defined in YAML;u don't need to be a developer to create a new flow. | ||
| - 🔣 **Easy to extend**: Plugins are everywhere in Kestra, many are available from the Kestra core team, but you can create one easily. | ||
| - 🆙 **Any triggers**: Kestra is event-based at heart—you can trigger an execution from API, schedule, detection, events | ||
| - 💻 **A rich user interface**: The built-in web interface allows you to create, run, and monitor all your flows—no need to deploy your flows, just edit them. | ||
| - ⏩ **Enjoy infinite scalability**: Kestra is built around top cloud native technologies—scale to millions of executions stress-free. | ||
|
|
||
| **Example flow:** | ||
|
|
||
| ```yaml | ||
| id: my-first-flow | ||
| namespace: my.company.teams | ||
|
|
||
| inputs: | ||
| - type: FILE | ||
| name: uploaded | ||
|
|
||
| tasks: | ||
| - id: "archive" | ||
| type: "io.kestra.plugin.gcp.gcs.Upload" | ||
| description: "Archive the file on Google Cloud Storage bucket" | ||
| from: "{{ inputs.uploaded }}" | ||
| to: "gs://my_bucket/archives/{{ execution.id }}.csv" | ||
| - id: "csvReader" | ||
| type: "io.kestra.plugin.serdes.csv.CsvReader" | ||
| from: "{{ inputs.uploaded }}" | ||
| - id: fileTransform | ||
| type: io.kestra.plugin.scripts.nashorn.FileTransform | ||
| description: This task will anonymize the contactName with a custom nashorn script (javascript over jvm). This show that you able to handle custom transformation or remapping in the ETL way | ||
| from: "{{ outputs.csvReader.uri }}" | ||
| script: | | ||
| if (row['contactName']) { | ||
| row['contactName'] = "*".repeat(row['contactName'].length); | ||
| } | ||
| - id: avroWriter | ||
| type: io.kestra.plugin.serdes.avro.AvroWriter | ||
| description: This file will convert the file from Kestra internal storage to avro. Again, we handling ETL since the conversion is done by Kestra before loading the data in BigQuery. This allow you to have some control before loading and to reject wrong data as soon as possible. | ||
| from: "{{ outputs.fileTransform.uri }}" | ||
| schema: | | ||
| { | ||
| "type": "record", | ||
| "name": "Root", | ||
| "fields": | ||
| [ | ||
| { "name": "contactTitle", "type": ["null", "string"] }, | ||
| { "name": "postalCode", "type": ["null", "long"] }, | ||
| { "name": "entityId", "type": ["null", "long"] }, | ||
| { "name": "country", "type": ["null", "string"] }, | ||
| { "name": "region", "type": ["null", "string"] }, | ||
| { "name": "address", "type": ["null", "string"] }, | ||
| { "name": "fax", "type": ["null", "string"] }, | ||
| { "name": "email", "type": ["null", "string"] }, | ||
| { "name": "mobile", "type": ["null", "string"] }, | ||
| { "name": "companyName", "type": ["null", "string"] }, | ||
| { "name": "contactName", "type": ["null", "string"] }, | ||
| { "name": "phone", "type": ["null", "string"] }, | ||
| { "name": "city", "type": ["null", "string"] } | ||
| ] | ||
| } | ||
| - id: load | ||
| type: io.kestra.plugin.gcp.bigquery.Load | ||
| description: Simply load the generated from avro task to BigQuery | ||
| avroOptions: | ||
| useAvroLogicalTypes: true | ||
| destinationTable: kestra-prd.demo.customer_copy | ||
| format: AVRO | ||
| from: "{{outputs.avroWriter.uri }}" | ||
| writeDisposition: WRITE_TRUNCATE* | ||
| - id: aggregate | ||
| type: io.kestra.plugin.gcp.bigquery.Query | ||
| description: Aggregate some data from loaded files | ||
| createDisposition: CREATE_IF_NEEDED | ||
| destinationTable: kestra-prd.demo.agg | ||
| sql: | | ||
| SELECT k.categoryName, p.productName, c.companyName, s.orderDate, SUM(d.quantity) AS quantity, SUM(d.unitPrice * d.quantity * r.exchange) as totalEur | ||
| FROM `kestra-prd.demo.salesOrder` AS s | ||
| INNER JOIN `kestra-prd.demo.orderDetail` AS d ON s.entityId = d.orderId | ||
| INNER JOIN `kestra-prd.demo.customer` AS c ON c.entityId = s.customerId | ||
| INNER JOIN `kestra-prd.demo.product` AS p ON p.entityId = d.productId | ||
| INNER JOIN `kestra-prd.demo.category` AS k ON k.entityId = p.categoryId | ||
| INNER JOIN `kestra-prd.demo.rates` AS r ON r.date = DATE(s.orderDate) AND r.currency = "USD" | ||
| GROUP BY 1, 2, 3, 4 | ||
| timePartitioningField: orderDate | ||
| writeDisposition: WRITE_TRUNCATE | ||
| ``` | ||
| ## Getting Started | ||
| To get a local copy up and running, please follow these simple steps. | ||
| ### Prerequisites | ||
| Make sure you have already installed: | ||
| - [Docker](https://docs.docker.com/engine/install/) | ||
| - [Docker Compose](https://docs.docker.com/compose/install/) | ||
| ### Launch Kestra | ||
| - Download the compose file [here](https://github.com/kestra-io/kestra/blob/develop/docker-compose.yml) | ||
| - Run `docker-compose up -d` | ||
| - Open `http://localhost:8080` on your browser | ||
| - Follow [this tutorial](https://kestra.io/docs/getting-started/) to create your first flow. | ||
| - Read the [documentation](https://kestra.io/docs/) to understand how to | ||
| - [Develop your flows](https://kestra.io/docs/developer-guide/) | ||
| - [Deploy Kestra](https://kestra.io/docs/administrator-guide/) | ||
| - Use our [terraform provider](https://kestra.io/docs/terraform/) | ||
| - Develop your [own plugins](https://kestra.io/docs/plugin-developer-guide/) | ||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
| ## Plugins | ||
| Kestra is built on [plugin systems](https://kestra.io/plugins/). You can find your plugin to interact with your provider; alternatively, you can follow [simple steps](https://kestra.io/docs/plugin-developer-guide/) to develop your own plugin. Here are the official plugins that are available: | ||
|
|
||
| <table> | ||
| <tr> | ||
| <td><a href="https://kestra.io/plugins/plugin-aws#s3">Amazon S3</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-serdes#avro">Avro</a></td> | ||
| <td><a href="https://kestra.io/plugins/core/tasks/scripts/io.kestra.core.tasks.scripts.Bash">Bash</a></td> | ||
| </tr> | ||
| <tr> | ||
| <td><a href="https://kestra.io/plugins/plugin-gcp#bigquery">Big Query</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-serdes#csv">CSV</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-jdbc-clickhouse">ClickHouse</a></td> | ||
| </tr> | ||
| <tr> | ||
| <td><a href="https://kestra.io/plugins/plugin-elasticsearch">ElasticSearch</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-notifications#mail">Email</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-gcp#gcs">Google Cloud Storage</a></td> | ||
| </tr> | ||
| <tr> | ||
| <td><a href="https://kestra.io/plugins/plugin-googleworkspace#drive">Google Drive</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-googleworkspace#sheets">Google Sheets</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-scripts-groovy">Groovy</a></td> | ||
| </tr> | ||
| <tr> | ||
| <td><a href="https://kestra.io/plugins/plugin-fs#http">Http</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-serdes#json">JSON</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-scripts-jython">Jython</a></td> | ||
| </tr> | ||
| <tr> | ||
| <td><a href="https://kestra.io/plugins/plugin-kafka">Kafka</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-kubernetes">Kubernetes</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-jdbc-sqlserver">Microsoft SQL Server</a></td> | ||
| </tr> | ||
| <tr> | ||
| <td><a href="https://kestra.io/plugins/plugin-mongodb">MongoDb</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-jdbc-mysql">MySQL</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-scripts-nashorn">Nashorn</a></td> | ||
| </tr> | ||
| <tr> | ||
| <td><a href="https://kestra.io/plugins/core/tasks/scripts/io.kestra.core.tasks.scripts.Node">Node</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-crypto#openpgp">Open PGP</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-jdbc-oracle">Oracle</a></td> | ||
| </tr> | ||
| <tr> | ||
| <td><a href="https://kestra.io/plugins/plugin-jdbc-postgres">Postgres</a></td> | ||
| <td><a href="https://kestra.io/plugins/core/tasks/scripts/io.kestra.core.tasks.scripts.Python">Python</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-jdbc-redshift">Redshift</a></td> | ||
| </tr> | ||
| <tr> | ||
| <td><a href="https://kestra.io/plugins/plugin-fs#sftp">SFTP</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-singer">Singer</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-notifications#slack">Slack</a></td> | ||
| </tr> | ||
| <tr> | ||
| <td><a href="https://kestra.io/plugins/plugin-jdbc-vectorwise">Vectorwise</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-jdbc-vertica">Vertica</a></td> | ||
| <td><a href="https://kestra.io/plugins/plugin-serdes#xml">XML</a></td> | ||
| </tr> | ||
| </table> | ||
|
|
||
|
|
||
|
|
||
| This list is growing quickly as we are actively building more plugins, and we welcome contributions! | ||
|
|
||
| ## Roadmap | ||
|
|
||
|  | ||
| See the [open issues](https://github.com/kestra-io/kestra/issues) for a list of proposed features (and known issues) or look at the [project board](https://github.com/orgs/kestra-io/projects/2). | ||
|
|
||
| ## Documentation | ||
| * The official Kestra documentation can be found under: [kestra.io](https://kestra.io) | ||
|
|
||
| ## License | ||
| Apache 2.0 © [Kestra Technologies](https://kestra.io) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters