💠 Introduction

This project demonstrates a real-time data pipeline using Kafka for messaging between two databases. It consists of two demos:

1. demo-db-to-kafka: Produces student data from a source database and publishes it to a Kafka topic named student.
2. demo-kafka-to-db: Consumes student data from the student topic and inserts it into a destination database.

Workflow Overview

1. Every 2 seconds, 5 students records are produced from the source database and published to the Kafka topic in a batch.
2. The batch is then consumed and inserted into the destination database as soon as the batch is received from the Kafka topic.

Visual Representation

• Left window: Shows the Kafka consumer listening for incoming data.
• Right window: Displays the destination database inserting the consumed data.

Students.data.sent.from.DB.to.another.DB.through.Kafka.mp4

⚠ Prerequisites

Ensure the following tools are installed to run the project smoothly:

• Java 21.0.3: Required to build and run the Spring Boot applications.
• Apache Maven 3.9.8: Used for dependency management and building the project.
• Spring Boot 3.3.3: Framework for building the Kafka producer and consumer demos.
• Docker 26.1.4: Necessary to set up MySQL, Zookeeper, and Kafka services in containers.

⚡ Running the Project

Follow these steps to run the project locally:

1. Clone the repository:

   git clone https://github.com/Ahmad-AlDeeb/demo-db-to-kafka-to-db.git
   

2. Navigate to the project directory:

   cd demo-db-to-kafka-to-db
   

3. Start Docker services: Make sure Docker is running, then start the required services (MySQL, Zookeeper, Kafka) using:

   docker-compose up -d
   

4. Build and run the producer demo:

   mvn -f ./demo-db-to-kafka clean install spring-boot:run
   

5. Build and run the consumer demo: Open another terminal in the same directory and run:

   mvn -f ./demo-kafka-to-db clean install spring-boot:run
   

6. Stop the services: After testing, stop both demos by pressing CTRL + C in each terminal.