This project is designed to fetch historical stock data for a predefined list of companies and store it in a MySQL database. It utilizes Python, Spark, and the Alpha Vantage API to perform data extraction, transformation, and loading (ETL) tasks.
- Fetch historical stock data using the Alpha Vantage API.
- Process and transform data using Apache Spark.
- Store the transformed data in a MySQL database.
- Configurable list of companies and date range for data fetching.
- Python 3.12 or higher
- Apache Spark
- MySQL Database
- An API key from Alpha Vantage
- Clone the repository to your local machine.
- Install the required Python dependencies:
- For Installing poetry :
pip install poetry
- For Installing packages defined in poetry.toml file :
poetry install
- For Installing poetry :
- Kindly Do Changes in the config.yml files related to the database, companies date range and API Key.
- Run the following command to execute the ETL process:
- For historical data:
poetry run python historical_dump.py
- For day-to-day data (Note: Day -1 data requires a premium version API key):
poetry run python daily_dump.py
- For historical data: