A progressive Node.js framework for building efficient and scalable server-side applications.
This repository serves as a template for a Kafka producer, offering flexible and powerful ways to send messages to Kafka topics. It supports three distinct methods of sending messages, making it a versatile solution for various data processing scenarios.
-
Broadcast: The message is sent to all partitions of a Kafka topic without binding to a specific key. This is used when you need to send a message to all consumers without partitioning by queues.
-
Queueing by Key: The message is sent with a specific key, ensuring its delivery to a designated partition of the topic. This ensures sequential processing of messages with the same key and is used to manage tasks that require ordered processing in a queue.
-
Rate-limited Sending: The ability to send messages at a specified rate limit. This approach is useful when you need to control the load on Kafka brokers or limit the processing speed for an even distribution of the load.
Clone the repository and configure the connection to your Kafka broker. Choose and configure the desired method for sending messages (broadcast, queueing by key, or rate-limited sending). Integrate the template into your application for quick and efficient Kafka message handling.
Each application instance should have its own static consumer group, ensuring they can read from a single partition independently. This design allows for scalability by increasing the number of partitions if needed, though it's not a strict requirement for functionality.
flowchart TD
P["Producer"] --> K["Kafka"]
K -->|group Id 1| A["Consumer replica 1"]
K -->|group Id 2| B["Consumer replica 2"]
K -->|group Id 3| C["Consumer replica 3"]
...
- Install dependencies
$ npm install- Create .env file by .env.example
KAFKAJS_NO_PARTITIONER_WARNING=1
NODE_ENV=local
HOST=localhost
...# development
$ npm run start
# watch mode
$ npm run start:dev
# production mode
$ npm run start:prod# unit tests
$ npm run test
# e2e tests
$ npm run test:e2e
# test coverage
$ npm run test:cov# build docker image
$ docker build -t nestjs-kafka-producer .
# run image with .env file
$ docker run --env-file .env -d -p 3000:3000 nestjs-kafka-producerCheck out a few resources that may come in handy when working with NestJS:
- Visit the NestJS Documentation to learn more about the framework.
- For questions and support, please visit our Discord channel.
- To dive deeper and get more hands-on experience, check out our official video courses.
- Visualize your application graph and interact with the NestJS application in real-time using NestJS Devtools.
- Need help with your project (part-time to full-time)? Check out our official enterprise support.
- To stay in the loop and get updates, follow us on X and LinkedIn.
- Looking for a job, or have a job to offer? Check out our official Jobs board.
Nest is an MIT-licensed open source project. It can grow thanks to the sponsors and support by the amazing backers. If you'd like to join them, please read more here.
- Author - Kamil Myśliwiec
- Website - https://nestjs.com
- Twitter - @nestframework
Nest is MIT licensed.