Skip to content

Latest commit

 

History

History
84 lines (70 loc) · 3.39 KB

README.md

File metadata and controls

84 lines (70 loc) · 3.39 KB
Raw

Big Data Management: Data-Driven Architecture Project using Spark

This repository contains the implementation of two critical backbones of a data-driven architecture: the Data Management Backbone and the Data Analysis Backbone. The project involves setting up a structured data lake with defined zones and performing either descriptive or predictive analysis.

Project Overview

This project focuses on creating a data-driven architecture using Apache Spark. It involves setting up a data lake with structured zones on the local file system, processing raw data, and performing analysis.

Environment SetUp

The following guide will aid in setting up PySpark on Mac (for help with Windows setup, please head to: https://www.machinelearningplus.com/pyspark/install-pyspark-on-windows/).

Spark Mac

  1. Open terminal
  2. Execute following command (make sure to have Homebrew installed) brew install openjdk
  3. For successful installation check run the following comands: 
    java -version
whereis java
  4. Setting up Java_Home environment in shell profile (e.g., ~/bashrc or ~/.zshrc) by running: 
    export JAVA_HOME=/usr/libexec/java_home

    source ~/.bashrc
  5. Installing Apache Spark brew install apache-spark -> for path info brew info apache-spark
  6. Setting Environment Variables (replace version with the installed Spark version) 
    export SPARK_HOME=/usr/local/Cella/apache-spark/<version>/libexec
export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=python3
source ~/.bashrc
  7. Install PySpark Python Package 
    pip install pyspark
pyspark --version

Data Management Backbone

Landing Zone

Stores raw data ingested into the data lake in a structured or semi-structured format. This includes data directly extracted from source systems with minimal transformation.

  • Implementation: This would be implemented in a Distributed File System in a real-world scenarion but for the project goal it will be done in my local file system.

Formatted Zone

Stores data in a standardized format according to a canonical data model. Data is potentially enriched and in a consumption-ready form. -Implementation: Implemented using Parquet files for efficient storage and schema enforcement on the local file system.

Exploitation Zone

Contains processed and refined data optimized for analysis, such as features and KPIs.

  • Implementation:* Implemented using Parquet and CSV files for efficient storage on the local file system.

Data Analysis Backbone

Descriptive Analysis and Dashboarding

Descriptive Analysis: Performed exploratory data analysis (EDA) on the data in the Exploitation Zone to summarize and understand the data.

Dashboarding: Created interactive dashboards using tools like Tableau, Power BI, or Jupyter Notebooks with matplotlib/seaborn.

How to navigate the repository

├── Documents
│   ├── BigData_Spark_notebook.ipynb
│   └── BigData_Spark_report.pdf
├── LandingZone
│   ├── cultural-sites
│   ├── income
│   └── price_opendata
├── FormattedZone
│   ├── CulturalSites
│   ├── Income
│   └── PriceOpenData
├── ExploitationZone
│   ├── CulturalSites
│   ├── Income
│   ├── Price_Income
│   └── PriceOpenData
└── README.md