client stops working after failing to send messages that are too large

[xificurC]

Requirement - what kind of business use case are you trying to solve?

Sending Jaeger spans.

Problem - what in Jaeger blocks you from solving the requirement?

After a few unsuccessful sends the client stops sending any spans. I've got these error messages by logging the exception as explained in #653 .

There's a couple of these

io.jaegertracing.internal.exceptions.SenderException: ThriftSender received a span that was too large, size = 109837, max = 64967
        at io.jaegertracing.thrift.internal.senders.ThriftSender.append(ThriftSender.java:58) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at com.gratex.bl.upm2.config.JaegerRemoteReporter$AppendCommand.execute(JaegerRemoteReporter.java:139) ~[classes/:1.0.1]
        at com.gratex.bl.upm2.config.JaegerRemoteReporter$QueueProcessor.run(JaegerRemoteReporter.java:174) [classes/:1.0.1]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

NOTE: JaegerRemoteReporter is RemoteReporter + the 1 line for the log.error

After a while I get errors like this:

io.jaegertracing.internal.exceptions.SenderException: Failed to flush spans.
        at io.jaegertracing.thrift.internal.senders.ThriftSender.append(ThriftSender.java:76) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at com.gratex.bl.upm2.config.JaegerRemoteReporter$AppendCommand.execute(JaegerRemoteReporter.java:139) ~[classes/:1.0.1]
        at com.gratex.bl.upm2.config.JaegerRemoteReporter$QueueProcessor.run(JaegerRemoteReporter.java:174) [classes/:1.0.1]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: io.jaegertracing.internal.exceptions.SenderException: Could not send 1 spans
        at io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:66) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:110) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thrift.internal.senders.ThriftSender.append(ThriftSender.java:73) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        ... 3 more
Caused by: org.apache.thrift.transport.TTransportException: Message size too large: 1 > 65000
        at io.jaegertracing.thrift.internal.reporters.protocols.ThriftUdpTransport.write(ThriftUdpTransport.java:132) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at org.apache.thrift.protocol.TCompactProtocol.writeByteDirect(TCompactProtocol.java:486) ~[libthrift-0.12.0.jar:0.12.0]
        at org.apache.thrift.protocol.TCompactProtocol.writeByteDirect(TCompactProtocol.java:493) ~[libthrift-0.12.0.jar:0.12.0]
        at org.apache.thrift.protocol.TCompactProtocol.writeFieldBeginInternal(TCompactProtocol.java:260) ~[libthrift-0.12.0.jar:0.12.0]
        at org.apache.thrift.protocol.TCompactProtocol.writeFieldBegin(TCompactProtocol.java:242) ~[libthrift-0.12.0.jar:0.12.0]
        at io.jaegertracing.thriftjava.Tag$TagStandardScheme.write(Tag.java:920) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thriftjava.Tag$TagStandardScheme.write(Tag.java:809) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thriftjava.Tag.write(Tag.java:709) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thriftjava.Span$SpanStandardScheme.write(Span.java:1360) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thriftjava.Span$SpanStandardScheme.write(Span.java:1144) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thriftjava.Span.write(Span.java:1028) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thriftjava.Batch$BatchStandardScheme.write(Batch.java:473) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thriftjava.Batch$BatchStandardScheme.write(Batch.java:407) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thriftjava.Batch.write(Batch.java:344) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.agent.thrift.Agent$emitBatch_args$emitBatch_argsStandardScheme.write(Agent.java:1056) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.agent.thrift.Agent$emitBatch_args$emitBatch_argsStandardScheme.write(Agent.java:1018) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.agent.thrift.Agent$emitBatch_args.write(Agent.java:969) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:71) ~[libthrift-0.12.0.jar:0.12.0]
        at org.apache.thrift.TServiceClient.sendBaseOneway(TServiceClient.java:66) ~[libthrift-0.12.0.jar:0.12.0]
        at io.jaegertracing.agent.thrift.Agent$Client.send_emitBatch(Agent.java:70) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.agent.thrift.Agent$Client.emitBatch(Agent.java:63) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:64) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:110) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thrift.internal.senders.ThriftSender.append(ThriftSender.java:73) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        ... 3 more

io.jaegertracing.internal.exceptions.SenderException: Failed to flush spans.
        at io.jaegertracing.thrift.internal.senders.ThriftSender.append(ThriftSender.java:76) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at com.gratex.bl.upm2.config.JaegerRemoteReporter$AppendCommand.execute(JaegerRemoteReporter.java:139) ~[classes/:1.0.1]
        at com.gratex.bl.upm2.config.JaegerRemoteReporter$QueueProcessor.run(JaegerRemoteReporter.java:174) [classes/:1.0.1]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: io.jaegertracing.internal.exceptions.SenderException: Could not send 8 spans
        at io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:66) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:110) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thrift.internal.senders.ThriftSender.append(ThriftSender.java:73) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        ... 3 more
Caused by: org.apache.thrift.transport.TTransportException: Message size too large: 1 > 65000
        at io.jaegertracing.thrift.internal.reporters.protocols.ThriftUdpTransport.write(ThriftUdpTransport.java:132) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at org.apache.thrift.protocol.TCompactProtocol.writeByteDirect(TCompactProtocol.java:486) ~[libthrift-0.12.0.jar:0.12.0]
        at org.apache.thrift.protocol.TCompactProtocol.writeMessageBegin(TCompactProtocol.java:206) ~[libthrift-0.12.0.jar:0.12.0]
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:70) ~[libthrift-0.12.0.jar:0.12.0]
        at org.apache.thrift.TServiceClient.sendBaseOneway(TServiceClient.java:66) ~[libthrift-0.12.0.jar:0.12.0]
        at io.jaegertracing.agent.thrift.Agent$Client.send_emitBatch(Agent.java:70) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.agent.thrift.Agent$Client.emitBatch(Agent.java:63) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:64) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:110) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        at io.jaegertracing.thrift.internal.senders.ThriftSender.append(ThriftSender.java:73) ~[jaeger-thrift-1.0.0.jar:1.0.0]
        ... 3 more

Proposal - what do you suggest to solve the problem or improve the existing situation?

A few failing messages shouldn't cause the whole client to fall down. I suspect some internal byte buffer doesn't get reinitialized correctly.

Also, I'd like to have the message size configurable via an environment variable.

Any open questions to address

None. Thank you!