Topic: Apache Airflow set-up on docker with AWS S3
Steps executed:
- Docker configuration for Airflow
- Docker configuration for Airflow’s extended image
- Docker configuration for AWS
- Executing docker image to create container
- DAG and Tasks creation in Airflow
- Executing DAG from Airflow UI
- Accessing S3 bucket / objects using AWS CLI
Technology Stack used: Docker, Airflow, Python, AWS SDK , AWS CLI and AWS S3