Yet-Another-Kafka-YAK2.0

YAK2.0 can also be termed as Yet Another Kafka 2.0.

This project aims to create Kafka like software or platform, for producers and consumers.

Table of Contents

Introduction

Prerequisites and Techstack

Steps for execution

Usage

Introduction

• In this project we have used Python and RabbitMQ to mimic the functionalities of Kafka.

• We have successfully implemented producers, consumers, 3 brokers, topic log, etc...

• This project can also handle multiple or any random number of producers and consumers.

  (back to top)

  
  

Prerequisites and Techstack

• Python

• RabbitMQ

• Kafka concept

  (back to top)

  
  

Steps for execution

1. Clone the Yet-Another-Kafka--YAK-2.0 repository

    git clone https://github.com/ankitacoder3/Yet-Another-Kafka-YAK2.0.git

2. Start RabbitMQ in terminal/command prompt. By typing

   rabbitmq-server.bat 

3. Open a new terminal and start consumer. By typing

   python consumer.py test n

   Or

   python consumer.py test --from-beginning

4. Open another new terminal and start producer. By typing

   python producer1.py test 101

5. Check the terminal where consumer is running, to check for messages received.
6. Now, open the 'broker' folder, which is under 'YAK' folder.
7. In that folder you can check the 'topic-log' file for the log activity.
8. In the same folder, check the '101' folder (where 101 is the key of leader broker).
9. Open the 'test-lead' file (where test is the topic). You can see that the message has been appended in file.
10. Hence, the messages have been successfully sent, received and stored. A file also maintains all the activities done as log.

(back to top)

Usage

• To run multiple producers and consumers, open a new terminal for each new consumer and producer.

• To run a consumer type the following:

  • To view the recent message sent

  python consumer.py <topic_name> n

  • To view all messages sent from beginning or starting of topic

  python consumer.py <topic_name> --from-beginning

• To run a producer type the following

  python producers1.py <topic_name> <key_of_leader_broker>

  where, key_of_leader_broker = 101 or 102 or 103

  (back to top)

  
  

Thank You.