Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[KafkaJS] Support single span for bulk producer message send #2236

Open
SeanReece opened this issue May 23, 2024 · 0 comments
Open

[KafkaJS] Support single span for bulk producer message send #2236

SeanReece opened this issue May 23, 2024 · 0 comments

Comments

@SeanReece
Copy link

SeanReece commented May 23, 2024

Is your feature request related to a problem? Please describe

When using producer.send or producer.sendBatch to send multiple messages, a span is created for every message sent. This creates a lot of noise when bulk sending messages (We frequently send 1000s of events in a single bulk call), and this is not how kafkaJS produces messages, which seems to be only split by topic/partition/broker.

See here for message sending logic in KafkaJS which is used for send and sendBatch: https://github.com/tulios/kafkajs/blob/master/src/producer/sendMessages.js#L17

Example

image

Describe the solution you'd like to see

I think this could either be configurable to only add spans to each call to producer.send (maybe include a messageCount attribute)

Example

instrumentations: [
    new KafkaJsInstrumentation({ groupProducerSend: true }),
],

OR the logic could be changed to wrap broker.produce (is this possible?) which would more closely represent how KafkaJS interacts with the broker.

Additional context

const producer = kafka.producer()

await producer.connect()
await producer.send({
    topic: 'topic-name',
    messages: [
        { key: 'key1', value: 'hello world' },
        { key: 'key2', value: 'hey hey!' }
    ],
})

The above snippet would create 2 spans currently, even though there is only 1 call to send and KafkaJS will send these to the broker in a single request.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants