This repository houses an ETL pipeline that processes music data sourced from a music application. The pipeline retrieves data from logs and files, transforms it, and loads it into a star schema in a PostgreSQL database
The main objective of this database is to provide the music app with a reliable and consistent source of data that can be used to answer various business questions related to the music preferences and habits of its users. By leveraging the data stored in logs and files, this database allows to gain insights into the types of songs and artists that are popular among its users
Data is retrieved from JSON files using file reading with PySpark.
Data is then transformed using PySpark to clean, format, and normalize the data.
Finally, the data is loaded into a PostgreSQL database using a star schema for improved query performance.
To execute the ETL pipeline, you must have PySpark and PostgreSQL dependencies installed.
Create a .env file :
# postgres
DB_NAME=database
DB_USER=user
DB_PASS=password
Enter this command :
Make run