-
Notifications
You must be signed in to change notification settings - Fork 714
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactors such that Kafka bundles spans and shares more code #146
Conversation
@adriancole This pr has same content as #143 which was reverted. I think we should try to find out why the Travis build fails prior merging. wdyt? |
This is the error I see in the logs:
|
3b5f471
to
f24bc75
Compare
This rewrites KafkaSpanCollector to bundle multiple spans into the same Kafka message. Basically, as many messages that come in one second will send in the same message. This change resulted in several times higher throughput in Yelp's kafka+cassandra Zipkin architecture. Flushing can be controlled via KafkaSpanCollector.Config.flushInterval Incidentally, this shares a lot of code with HttpSpanCollector, which should reduce the amount of bugs and maintenance around queue-based collection.
f24bc75
to
8e5c3d9
Compare
sorry.. some leftovers. should be good now! |
Refactors such that Kafka bundles spans and shares more code
This change is released as part of brave 3.5.0 |
@kristofa @adriancole - I don't believe it is necessary to front the |
@ewhauser ack. the main point is to move span reporting off the client thread, which has a side-effect of protecting it from exceptions and what-not as well. There's no intention to limit implementations to synchronous invocation, although what you've noted surely deserves a comment in the code. |
ex. there's several exceptions that could crash the calling thread: this queue is implicitly protecting the caller from this sort of thing (although it isn't well tested). I can update the code to make this point more explicit. |
This rewrites KafkaSpanCollector to bundle multiple spans into the same
Kafka message. Basically, as many messages that come in one second will
send in the same message. This change resulted in several times higher
throughput in Yelp's kafka+cassandra Zipkin architecture.
Flushing can be controlled via KafkaSpanCollector.Config.flushInterval
Incidentally, this shares a lot of code with HttpSpanCollector, which
should reduce the amount of bugs and maintenance around queue-based
collection.