Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bundle src code and plugin into docker image #53

Merged
merged 4 commits into from
Mar 20, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,33 @@
FROM apache/airflow:2.5.1

USER root

# Copy Ro-dou core files from the host Docker context
COPY src /opt/airflow/dags/ro_dou

# Install Git
RUN apt-get update && \
apt-get install -y git && \
rm -rf /var/lib/apt/lists/*

# Clone the FastETL GitHub repository and copy specified files
RUN git clone https://github.com/economiagovbr/FastETL.git /tmp/repo-FastETL && \
mkdir -p /opt/airflow/plugins/FastETL/hooks && \
mkdir -p /opt/airflow/plugins/FastETL/custom_functions/utils && \
cp /tmp/repo-FastETL/hooks/dou_hook.py /opt/airflow/plugins/FastETL/hooks/dou_hook.py && \
cp -r /tmp/repo-FastETL/custom_functions/utils/* /opt/airflow/plugins/FastETL/custom_functions/utils/

# Remove Git and clean up package cache
RUN apt-get remove -y git && \
apt-get autoremove -y && \
rm -rf /tmp/repo-ro-dou && \
rm -rf /tmp/repo-FastETL

USER airflow

# Install additional Airflow dependencies
RUN pip install --no-cache-dir --user 'apache-airflow[microsoft.mssql,google_auth]'

# Copy and install requirements.txt
COPY requirements.txt /
RUN pip install --no-cache-dir -r /requirements.txt
7 changes: 1 addition & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,10 @@ isso é necessário ter o **Docker Compose** na versão 1.29 ou maior
clonar o repositório no seu computador, acesse o diretório pela linha de
comando e execute os comandos a seguir:

```bash
make install-deps
```
Este comando baixa as dependências necessárias.

```bash
make run
```
Este comando baixa as imagens docker e sobe todos os contêineres do Ro-dou.
Este comando baixa as imagens docker necessárias, faz build da contêiner do Ro-dou e sobe tudo.

O Airflow pode demorar alguns minutos para se configurar a primeira vez. Após
isso ele estará disponível em http://localhost:8080/. Para se autenticar
Expand Down
7 changes: 2 additions & 5 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,12 +71,9 @@

AIRFLOW_CONN_EXAMPLE_DATABASE_CONN: postgresql://airflow:airflow@postgres:5432/airflow
volumes:
- ./src:/opt/airflow/dags/ro_dou
- ./src:/opt/airflow/dags/ro_dou # for development purpose
- ./dag_confs:/opt/airflow/dags/ro_dou/dag_confs
- ./tests:/opt/airflow/tests

- ./FastETL/hooks/dou_hook.py:/opt/airflow/plugins/FastETL/hooks/dou_hook.py
- ./FastETL/custom_functions/utils/:/opt/airflow/plugins/FastETL/custom_functions/utils/
- ./tests:/opt/airflow/tests # for test purpose
depends_on:
postgres:
condition: service_healthy
Expand Down