-
Notifications
You must be signed in to change notification settings - Fork 68
/
apache.yml
77 lines (69 loc) · 3.66 KB
/
apache.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
commands:
create-admin:
args: users create --username admin --firstname FIRST_NAME --lastname LAST_NAME
--role Admin --email admin@example.org
description: Create an admin user.
ui:
args: webserver
description: Start the Airflow webserver.
docs: https://docs.meltano.com/guide/orchestration
domain_url: https://airflow.apache.org/
label: Airflow
logo_url: /assets/logos/orchestrators/airflow.png
maintenance_status: active
name: airflow
namespace: airflow
next_steps: |
_The version of Airflow currently installed with Meltano (2.1.2) requires that Python be at version 3.9 or lower. Upgrading the Airflow version is tracked in [this issue](https://github.com/meltano/hub/issues/658)._
1. Use the [meltano schedule](https://docs.meltano.com/reference/command-line-interface#schedule) command to create pipeline schedules in your project, to be run by Airflow.
1. If you're running Airflow for the first time in a new environment, create an admin user:
```sh
meltano invoke airflow:create-admin
# This is equivalent to `airflow users create` with some arguments in the Airflow documentation
```
1. Launch the Airflow UI and log in using the username/password you created:
```sh
meltano invoke airflow:ui
```
By default, the UI will be available at at [`http://localhost:8080`](http://localhost:8080). You can change this using the `webserver.web_server_port` setting documented below.
1. Start Scheduler or execute Airflow commands directly using the instructions in [the Meltano docs](https://docs.meltano.com/guide/orchestration#starting-the-airflow-scheduler).
pip_url: apache-airflow==2.10.1 --constraint
https://raw.githubusercontent.com/apache/airflow/constraints-2.10.1/constraints-${MELTANO__PYTHON_VERSION}.txt
repo: https://github.com/apache/airflow
requires:
files:
- name: files-airflow
variant: meltano
settings:
- env: AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION
kind: boolean
label: Pause DAGs at Creation
name: core.dags_are_paused_at_creation
value: false
- env: AIRFLOW__CORE__DAGS_FOLDER
label: DAGs Folder
name: core.dags_folder
value: $MELTANO_PROJECT_ROOT/orchestrate/dags
- env: AIRFLOW__CORE__LOAD_EXAMPLES
kind: boolean
label: Load Examples
name: core.load_examples
value: false
- env: AIRFLOW__CORE__PLUGINS_FOLDER
label: Plugins Folder
name: core.plugins_folder
value: $MELTANO_PROJECT_ROOT/orchestrate/plugins
- env: AIRFLOW__CORE__SQL_ALCHEMY_CONN
label: SQL Alchemy Connection
name: core.sql_alchemy_conn
value: sqlite:///$MELTANO_PROJECT_ROOT/.meltano/orchestrators/airflow/airflow.db
- env: AIRFLOW__WEBSERVER__WEB_SERVER_PORT
kind: integer
label: Webserver Port
name: webserver.web_server_port
value: 8080
settings_group_validation: []
settings_preamble: |
Meltano [centralizes the configuration](https://docs.meltano.com/guide/configuration) of all of the plugins in your project, including Airflow's. This means that if the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html) tells you to put something in `airflow.cfg`, you can use `meltano config`, `meltano.yml`, or environment variables instead, and get the benefits of Meltano features like [environments](https://docs.meltano.com/concepts/environments).
Any setting you can add to `airflow.cfg` can be added to `meltano.yml`, manually or using `meltano config`. For example, `[core] executor = SequentialExecutor` becomes `meltano config airflow set core executor SequentialExecutor` on the CLI, or `core.executor: SequentialExecutor` in `meltano.yml`. Config sections indicated by `[section]` in `airflow.cfg` become nested dictionaries in `meltano.yml`.
variant: apache