Sparkify-data-lake

This is project 4 of Udacitys Data Engineering Nanodegree. In this project Spark is used to pull data from a s3 bucket. Temporary tables are then created, and data is uploaded to another bucket written in .parquet format.

Starting the program

1. Add global aws config values in dl.cfg
2. Execute "etl.py".