ETL for Song Play Analysis

This project aims to transform raw song play data and load them into traditional database, in this case, Postgres for later analysis. This is also used to satisfied with Data Modeling with Postgres project under Data Engineer Nanodegree Program.

Prerequisite

• conda
• Docker
• dataset gathered from Million Song Dataset

Local Setup

1. Bootstrap Python and dependencies

   $ ./bootstrap_env_via_conda.sh

2. Spin up localized instance of Postgres DB

   $ ./respawn_db.sh

3. Initialize related tables

   $ python ./create_tables.py

4. place dataset under ./data directory

Running ETL

$ python etl.py

Verifying ETL Result

$ jupyter notebook

# then walk through `test.ipynb` notebook

Note

• In case of something wrong in local database, use respawn_db.sh to re-initialize new one.