Skip to content

Commit

Permalink
fix(docs): keep getting started high level (#410)
Browse files Browse the repository at this point in the history
point to detailed troubleshooting guide only if needed
  • Loading branch information
anna-geller authored Aug 4, 2023
1 parent afba2e2 commit b3d7939
Show file tree
Hide file tree
Showing 8 changed files with 230 additions and 35 deletions.
49 changes: 20 additions & 29 deletions content/docs/01.getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,23 @@
title: 🚀 Getting started
---

This document will guide you through setting up Kestra in a few minutes and creating your first flow.
This page shows how to start Kestra and create your first flow.

## Prerequisites

Make sure the following is installed:
- Docker, [Docker Desktop](https://docs.docker.com/get-docker/) is the easiest way to use it.
Make sure that Docker is installed in your environment. We recommend [Docker Desktop](https://docs.docker.com/get-docker/).

## Run Kestra
## Start Kestra

### Download Docker Compose file

Download the Docker Compose file for Kestra using the following command:
Download the Docker Compose file using the following command:

```bash
curl -o docker-compose.yml https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml
```

If you don't have curl installed, you can download the [Docker Compose file](https://github.com/kestra-io/kestra/blob/develop/docker-compose.yml) and save it as `docker-compose.yml`.
If you don't have `curl` installed, you can download the [Docker Compose file](https://github.com/kestra-io/kestra/blob/develop/docker-compose.yml) manually and save it as `docker-compose.yml`.

### Start your server

Expand All @@ -29,48 +28,40 @@ Make sure the Docker is up and running. Then, run the following command:
docker compose up -d
```

Open [http://localhost:8080](http://localhost:8080) in your browser.
Open [http://localhost:8080](http://localhost:8080) in your browser to launch the UI.

### Networking in this docker compose

The above docker-compose file doesn't specify anything regarding the network of Kestra's service.
> If you want to extend your Docker Compose file, modify container networking, or if you have any other issues using this Docker Compose file, check the [Troubleshooting Guide](14.troubleshooting.md).
This means that as it is, you won't be able to access any of your localhost exposed services (eg. another docker container with a mapped port) even if you're able to reach it from your machine.

If you plan to use local services and reach them from Kestra, you have four options:
- Depending on your docker implementation, you should be able to use the `host.docker.internal` hostname or `172.17.0.1` to reach your host machine from inside Kestra's container.
- By default, your container will be in a `default` network. You can add your other services in the above `docker-compose.yml` file and use the services' alias (keys from `services`) to reach them.
- You can create a new network and add your services to it. Then you can add this network to the `networks` section of the `kestra` service. With this, you will have access via `localhost` to all your exposed ports.
- You can use the `host` network mode for the `kestra` service. This will make your container use your host network and you will be able to reach all your exposed ports. This means you have to change the `services.kestra.environment.KESTRA_CONFIGURATION.datasources.postgres.url` to `jdbc:postgresql://localhost:5432/kestra`. This is the easiest way but it can be a security risk.

## Create your first flow

In the **Flow** menu, click the "Create" button and paste the following flow:
Navigate to **Flows** in the left menu, then click the "Create" button and paste the following configuration to create your first flow:

```yaml
id: getting-started
namespace: io.kestra.tutorial
namespace: dev
tasks:
- id: log-task
- id: hello
type: io.kestra.core.tasks.log.Log
message: Hello World !
message: Hello World!
```
When you're done, click on the **Save** button to save it and start your first execution via the **Create Execution** button.
Click on **Save** and start your first execution via the **Create Execution** button.
![Your first flow saved](/docs/getting-started/saved.png)
::next-link
[For a better introduction to Kestra, follow our Tutorial](./02.tutorial/index.md)
[For a more detailed introduction to Kestra, check our Tutorial](./02.tutorial/index.md)
::
## Next Steps
Now, you're ready to use Kestra! :rocket:
Congrats on your first flow! :rocket:
As next steps, we suggest reading the following documentation in this order:
- Follow the [tutorial](./02.tutorial/index.md) and create your first flow with all the Kestra possibilities!
- Learn Kestra [concepts](./03.concepts/index.md).
- Read the [Developer Guide](./05.developer-guide/index.md) to understand how to build your own flows.
- Look at [Plugins](../plugins/index.md) to perform some real tasks.
- [Deploy](./09.administrator-guide/index.md) your Kestra instance to real environments.
Next, we suggest following the documentation in this order:
- Check the [tutorial](./02.tutorial/index.md)
- Learn core [concepts](./03.concepts/index.md)
- Read the [Developer Guide](./05.developer-guide/index.md) for more advanced concepts
- Check the available [Plugins](../plugins/index.md) to integrate with external systems and automate your tasks
- [Deploy](./09.administrator-guide/index.md) your Kestra instance to production environments.
2 changes: 1 addition & 1 deletion content/docs/02.tutorial/index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: 🤓 Tutorial
title: 📚 Tutorial
---

Welcome to this tutorial on Kestra! Kestra is an open-source workflow engine and job scheduler that allows you to automate and manage complex workflows efficiently.
Expand Down
2 changes: 1 addition & 1 deletion content/docs/04.user-interface-guide/index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: 🖥️ User Interface Guide
title: 📊 User Interface Guide
---

Kestra comes with a rich web user interface located by default on port 8080.
Expand Down
2 changes: 1 addition & 1 deletion content/docs/08.architecture.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: 📝 Architecture
title: 🏗️ Architecture
---

## The Kestra internal components
Expand Down
2 changes: 1 addition & 1 deletion content/docs/12.api-guide.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: 📝 Api Guide
title: 💻 API Guide
rightBar: false
editLink: false
---
Expand Down
2 changes: 1 addition & 1 deletion content/docs/13.contributing.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: 🔧 Contributing
title: 🤝 Contributing
---

You can contribute to Kestra in many ways depending on your skills and interests.
Expand Down
204 changes: 204 additions & 0 deletions content/docs/14.troubleshooting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
---
title: 🔧 Troubleshooting
---

Something doesn't work as expected? Check out this page to help you troubleshoot.


## Support us on social medias

- Star Kestra on [GitHub](https://github.com/kestra-io/kestra).
- Follow us on [Twitter](https://twitter.com/kestra_io) and [LinkedIn](https://www.linkedin.com/company/kestra).
- Get in touch with us on [Slack](https://kestra.io/slack).


### Networking in Docker Compose

The docker-compose file provided as part of the [Getting Started](01.getting-started.md) guide doesn't configure networking for the Kestra containers. This means that you won't be able to access any services exposed via `localhost` on your local machine (e.g., another Docker container with a mapped port). Your machine and Docker container use a different network. To use a locally exposed service from Kestra container, you can use the `host.docker.internal` hostname or `172.17.0.1`. The `host.docker.internal` address allows you to reach your host machine's services from Kestra's container.

Alternatively, you can leverage Docker network. By default, your Kestra container will be placed in a `default` network. You can add your custom services to the `docker-compose.yml` file provided by Kestra and use the services' alias (keys from `services`) to reach them. Even better would be if you create a new network e.g. network `kestra_net` and add your services to it. Then you can add this network to the `networks` section of the `kestra` service. With this, you will have access via `localhost` to all your exposed ports. The example below shows how you can add `iceberg-rest`, `minio` and `mc` (i.e. Minio client) to your Kestra Docker Compose file.

```yaml
version: "3"

volumes:
postgres-data:
driver: local
kestra-data:
driver: local

networks:
kestra_net:

services:
postgres:
image: postgres
volumes:
- postgres-data:/var/lib/postgresql/data
environment:
POSTGRES_DB: kestra
POSTGRES_USER: kestra
POSTGRES_PASSWORD: k3str4
healthcheck:
test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
interval: 30s
timeout: 10s
retries: 10
networks:
kestra_net:

iceberg-rest:
image: tabulario/iceberg-rest
ports:
- 8181:8181
environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=password
- AWS_REGION=us-east-1
- CATALOG_WAREHOUSE=s3://warehouse/
- CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
- CATALOG_S3_ENDPOINT=http://minio:9000
networks:
kestra_net:

minio:
image: minio/minio
container_name: minio
environment:
- MINIO_ROOT_USER=admin
- MINIO_ROOT_PASSWORD=password
- MINIO_DOMAIN=minio
networks:
kestra_net:
aliases:
- warehouse.minio
ports:
- 9001:9001
- 9000:9000
command: ["server", "/data", "--console-address", ":9001"]

mc:
depends_on:
- minio
image: minio/mc
container_name: mc
networks:
kestra_net:
environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=password
- AWS_REGION=us-east-1
entrypoint: >
/bin/sh -c "
until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
/usr/bin/mc rm -r --force minio/warehouse;
/usr/bin/mc mb minio/warehouse;
/usr/bin/mc policy set public minio/warehouse;
tail -f /dev/null
"
kestra:
image: kestra/kestra:latest-full
pull_policy: always
entrypoint: /bin/bash
# Note that this is meant for development only. Refer to the documentation for production deployments of Kestra which runs without a root user.
user: "root"
command:
- -c
- /app/kestra server standalone --worker-thread=128
volumes:
- kestra-data:/app/storage
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/kestra-wd:/tmp/kestra-wd
environment:
KESTRA_CONFIGURATION: |
datasources:
postgres:
url: jdbc:postgresql://postgres:5432/kestra
driverClassName: org.postgresql.Driver
username: kestra
password: k3str4
kestra:
server:
basic-auth:
enabled: false
username: admin
password: kestra
repository:
type: postgres
storage:
type: local
local:
base-path: "/app/storage"
queue:
type: postgres
tasks:
tmp-dir:
path: /tmp/kestra-wd/tmp
url: http://localhost:8080/
ports:
- "8080:8080"
- "8081:8081"
depends_on:
postgres:
condition: service_started
networks:
kestra_net:
```
Finally, you can also use the `host` network mode for the `kestra` service. This will make your container use your host network and you will be able to reach all your exposed ports. This means you have to change the `services.kestra.environment.KESTRA_CONFIGURATION.datasources.postgres.url` to `jdbc:postgresql://localhost:5432/kestra`. This is the easiest way but it can be a security risk. See the example below using `network_mode`.

```yaml
version: "3"
volumes:
kestra-data:
driver: local
services:
kestra:
image: kestra/kestra:latest-full
pull_policy: always
entrypoint: /bin/bash
network_mode: host
environment:
JAVA_OPTS: "--add-opens java.base/java.nio=ALL-UNNAMED"
NODE_OPTIONS: "--max-old-space-size=4096"
KESTRA_CONFIGURATION: |
datasources:
postgres:
url: jdbc:postgresql://localhost:5432/kestra
driverClassName: org.postgresql.Driver
username: kestra
password: k3str4
kestra:
server:
basic-auth:
enabled: false
username: admin
password: kestra
anonymous-usage-report:
enabled: true
repository:
type: postgres
storage:
type: local
local:
base-path: "/app/storage"
queue:
type: postgres
tasks:
tmp-dir:
path: /tmp/kestra-wd/tmp
scripts:
docker:
volume-enabled: true
defaults: # just one example to show global taskDefaults
- type: io.kestra.plugin.airbyte.connections.Sync
url: http://host.docker.internal:8000/
username: airbyte
password: password
url: http://localhost:8080/
variables:
env-vars-prefix: "" # to avoid requiring KESTRA_ prefix on env vars
```
2 changes: 1 addition & 1 deletion content/docs/index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: Overview
title: 📓 Overview
---

Kestra is an open-source, event-driven orchestrator that simplifies data operations and improves collaboration between engineers and business users. By bringing Infrastructure as Code best practices to data pipelines, Kestra allows you to build reliable workflows and manage them with confidence.
Expand Down

1 comment on commit b3d7939

@vercel
Copy link

@vercel vercel bot commented on b3d7939 Aug 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Successfully deployed to the following URLs:

kestra-io – ./

kestra-io.vercel.app
kestra-io-git-main-kestra.vercel.app
kestra-io-kestra.vercel.app

Please sign in to comment.