Overview

The purpose of this project is develop and end to end solution using various tools that can stream stock market data in real time that can be used for reporting and analysis

Technology

Python - Functions for Producer and Consumer

Apache Kafka - Used to stream stock market data

AWS S3 - Data Lake/Object Storage

Amazon EC2 - Computing instance to run scripts

AWS Glue - Crawler to fetch new data as its added to S3

Amazon Athena - Serverless analytics service to run SQL

Architecture

Design/Development Process

1. Create a new EC2 instance to install Java & Kafka on
2. Create new Kafka Topic, and start Producer & Consumer
3. Develop Python scripts to pass data for each stock ticker to a data frame using pandas and upload to S3 in JSON format
4. Design AWS Glue Crawler to crawl the bucket in the data lake to check for new data in real time
5. Utilize Athena to confirm data counts are increasing on tables as data streams in real time to AWS S3 & gets ingested from the Glue Crawler