-
Notifications
You must be signed in to change notification settings - Fork 714
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactors such that Kafka bundles spans and shares more code #143
Conversation
Good that we try to share more code between collectors. The queueing and submitting logic was implemented multiple times and can easy contain subtle bugs. Makes sense to abstract and re-use. |
3a504cd
to
f813664
Compare
@kristofa PTAL |
PS the code would be less complex if we mandated those using Kafka to |
I think it is fine to ask people who use Kafka to upgrade to 1.35+ collectors. We can add it in the release notes. In that case we can remove the |
I also like the approach with |
f813664
to
40c81b9
Compare
ready to go |
40c81b9
to
ac5ca34
Compare
This rewrites KafkaSpanCollector to bundle multiple spans into the same Kafka message. Basically, as many messages that come in one second will send in the same message. This change resulted in several times higher throughput in Yelp's kafka+cassandra Zipkin architecture. Flushing can be controlled via KafkaSpanCollector.Config.flushInterval Incidentally, this shares a lot of code with HttpSpanCollector, which should reduce the amount of bugs and maintenance around queue-based collection.
ac5ca34
to
74cc5af
Compare
Not sure why travis build is unhappy. Tests run fine in my local environment. Will merge. |
Refactors such that Kafka bundles spans and shares more code
@adriancole After merge master build is failing. There was a warning about missing version for maven-shade-plugin which I fixed but I can't see build log for latest 2 builds... (latest one: https://travis-ci.org/openzipkin/brave/builds/113873969) any idea what is wrong? |
I can't reproduce build failure in my local environment. |
This rewrites KafkaSpanCollector to bundle multiple spans into the same
Kafka message. Basically, as many messages that come in one second will
send in the same message. This change resulted in several times higher
throughput in Yelp's kafka+cassandra Zipkin architecture.
Flushing can be controlled via KafkaSpanCollector.Config.flushInterval
Incidentally, this shares a lot of code with HttpSpanCollector, which
should reduce the amount of bugs and maintenance around queue-based
collection.