Skip to content

Latest commit

 

History

History
132 lines (99 loc) · 6.25 KB

README.md

File metadata and controls

132 lines (99 loc) · 6.25 KB
Raw

Nest Logo

A progressive Node.js framework for building efficient and scalable server-side applications.

NPM Version Package License NPM Downloads CircleCI Coverage Discord Backers on Open Collective Sponsors on Open Collective Donate us Support us Follow us on Twitter

Kafka Producer Template

This repository serves as a template for a Kafka producer, offering flexible and powerful ways to send messages to Kafka topics. It supports three distinct methods of sending messages, making it a versatile solution for various data processing scenarios.

Features

  1. Broadcast: The message is sent to all partitions of a Kafka topic without binding to a specific key. This is used when you need to send a message to all consumers without partitioning by queues.

  2. Queueing by Key: The message is sent with a specific key, ensuring its delivery to a designated partition of the topic. This ensures sequential processing of messages with the same key and is used to manage tasks that require ordered processing in a queue.

  3. Rate-limited Sending: The ability to send messages at a specified rate limit. This approach is useful when you need to control the load on Kafka brokers or limit the processing speed for an even distribution of the load.

How to Use

Clone the repository and configure the connection to your Kafka broker. Choose and configure the desired method for sending messages (broadcast, queueing by key, or rate-limited sending). Integrate the template into your application for quick and efficient Kafka message handling.

BroadCast

Each application instance should have its own static consumer group, ensuring they can read from a single partition independently. This design allows for scalability by increasing the number of partitions if needed, though it's not a strict requirement for functionality.

flowchart TD
    P["Producer"] --> K["Kafka"]
    K -->|group Id 1| A["Consumer replica 1"]
    K -->|group Id 2| B["Consumer replica 2"]
    K -->|group Id 3| C["Consumer replica 3"]
Loading

Queueing by Key

...

Project setup

  1. Install dependencies
$ npm install
  1. Create .env file by .env.example
KAFKAJS_NO_PARTITIONER_WARNING=1
NODE_ENV=local
HOST=localhost
...

Compile and run the project

# development
$ npm run start

# watch mode
$ npm run start:dev

# production mode
$ npm run start:prod

Run tests

# unit tests
$ npm run test

# e2e tests
$ npm run test:e2e

# test coverage
$ npm run test:cov

Docker build & run

# build docker image
$ docker build -t nestjs-kafka-producer .

# run image with .env file
$ docker run --env-file .env -d -p 3000:3000 nestjs-kafka-producer

Resources

Check out a few resources that may come in handy when working with NestJS:

  • Visit the NestJS Documentation to learn more about the framework.
  • For questions and support, please visit our Discord channel.
  • To dive deeper and get more hands-on experience, check out our official video courses.
  • Visualize your application graph and interact with the NestJS application in real-time using NestJS Devtools.
  • Need help with your project (part-time to full-time)? Check out our official enterprise support.
  • To stay in the loop and get updates, follow us on X and LinkedIn.
  • Looking for a job, or have a job to offer? Check out our official Jobs board.

Support

Nest is an MIT-licensed open source project. It can grow thanks to the sponsors and support by the amazing backers. If you'd like to join them, please read more here.

Stay in touch

License

Nest is MIT licensed.