Support tracing in Cassandra itself

[yurishkuro]

It would be nice if this module not only wrapped Cassandra calls in the client-side spans, but would also inject the opentracing span context into Cassandra requests so that tracing of the Cassandra backend was also possible, e.g. as was done in this blog post.

[malafeev]

As I understand from https://github.com/thelastpickle/cassandra-zipkin-tracing
It requires development of plugin (custom tracing class).
Each Cassandra node needs to be started with 2 jvm options:

JVM_OPTS = 
"-Dcassandra.custom_tracing_class=io.opentracing.contrib.cassandra.CustomTracing"
"-Dcassandra.custom_query_handler_class=org.apache.cassandra.cql3.CustomPayloadMirroringQueryHandler"

It will work only starting with Cassandra 3.4. Current is 3.10

[yurishkuro]

That's my understanding as well. Hopefully the plugin can be purely OpenTracing, so that it would work with any tracer. It could use service loader.

[malafeev]

service loader requires adding file under "META-INF/service" with interface name (as filename) and implementation (in file content).
it sounds like user will need to build jar himself and before that prepare and put file under META-INF/service.
for me it looks not user friendly :)

[objectiser]

@malafeev The cassandra custom tracing class simply needs to use https://github.com/opentracing-contrib/java-tracerresolver - this uses service loader under the covers, so as long as the tracer provides the TracerResolver impl (and any relevant config) it will just work by adding the dependency to the cassandra server.

[malafeev]

looks like LightStep tracer doesn't provide TracerResolver. @bhs ?

[yurishkuro]

Not sure about Lightstep. Jaeger does support tracer resolver

[bhs]

@malafeev, LightStep's tracer does not provide TracerResolver, but that is not intentional and easily remedied... we just haven't had a customer asking for it.

[malafeev]

I faced one issue: payload is sent using ByteBuffer.
But BINARY format support is not implemented properly in opentracing java api (I don't know size of byte buffer to allocate):
opentracing/opentracing-java#95

tracer.inject(span.context(), Builtin.BINARY, ByteBuffer.allocate(????????));

[bhs]

@malafeev just wanted to make sure you were aware of opentracing/opentracing-java#99 (which should be part of the 1.0 release, I'd argue).

[tedsuo]

Latest BINARY proposal: opentracing/opentracing-java#276