Airflow systemd

Install airflow server as systemd daemon under linux. This run with your regular linux user.

Requirements

• miniconda
• mariadb >= 10.2.7

Setup airflow

Step 1: Clone repo.

$ cd ~
$ git clone https://github.com/adrianmarino/airflow-systemd.git
$ mv airflow-systemd airflow
$ cd airflow

Step 2: Install mysql dev tools.

$ sudo apt install libmysqlclient-dev

Step 3: Create conda environment required to run airflow.

$ conda env create -f environment.yml

Step 4: Copy service file user level systemd config path:

$ cp airflow.service ~/.config/systemd/user/

Step 5: Import required common shell env variables:

$ echo "source ~/airflow/.shell.airflowrc" >> ~/.bashrc 
or
$ echo "source ~/airflow/.shell.airflowrc" >> ~/.zshrc

Step 6: Replace __HOME__ with your home path.

$ sed -i 's/__HOME__/YOUR_HOME_PATH/g' $HOME/airflow/airflow.cfg

Step 7: Refresh systemd daemon with updated config.

$ systemctl --user daemon-reload

Step 8: Start service on boot.

$ systemctl --user enable airflow

Step 9: Create database and user used by airflow server to access to database schema.

CREATE DATABASE airflow_db;
CREATE USER airflow_user;
GRANT ALL PRIVILEGES ON airflow_db.* TO 'airflow_user'@localhost IDENTIFIED BY 'airflow_pass';
FLUSH PRIVILEGES;

Step 10: Config db connection string under airflow.cfg:

sql_alchemy_conn = mysql://airflow_user:airflow_pass@localhost/airflow_db

Step 11: Initialize db:

$ airflow db init

Step 12: Create an admin airflow webserver user:

$ airflow users create \
  --username username \
  --firstname Fistname \
  --lastname LastName \
  --role Admin \
  --email my.email@gmail.com

Step 13: Check miniconda home path in config.conf.

CONDA_PATH="/opt/miniconda3"

Step 14: Start airflow as systemd daemon.

$ systemctl --user start airflow

Step 15: create a ~/airflow/dags directory where will all dags be stored.