OpenTracing support

[whiskeysierra]

Detailed Description

Add OpenTracing integration that e.g. allows to add retry count as a tag to spans.

Context

Instrumenting the underlying Apache Http Client is a good enough solution for most, but a deeper integration allows for more sophisticated use cases (like retry count).

Possible Implementation

• riptide-opentracing module
  • OpenTracingPlugin
  • plugin that creates a span for one logical request
  • OpenTracingRetryListener which set's a retry count tag
  • Support for span decorator?
    • Tags https://github.com/opentracing/specification/blob/master/semantic_conventions.md
  • Operation Name generator?
    • Type-safe attributes to be passed from call-site to plugins (transported as part of request arguments)
• riptide-spring-boot-autoconfigure
  • plugin needs to be registered directly after/before timeout plugin (auto configuration)
  • instrument executor and scheduler using opentracing-concurrent
  • instrument Apache Http Client using jaeger-apachehttpclient without dependency to jaeger!
    • Propagate tags?
  • Set client id as tag?
  • Degree of configurability?

Example Configuration

riptide:
  defaults:
    tracing:
      enabled: true
      tags:
        account: ${CDP_TARGET_INFRASTRUCTURE_ACCOUNT}
        zone: ${CDP_TARGET_REGION}
        artifact_version: ${CDP_BUILD_VERSION}
        deployment_id: ${CDP_DEPLOYMENT_ID}
      propagate-flow-id: true
        
  clients:
    exchange-rate:
      tracing:
        tags:
          peer.service: exchange-rate-service
        propagate-flow-id: false

Your Environment

• Version used:
• Link to your project:

[tkrop]

I'm not sure about the value of retry count - OpenTracing needs aggregate over spans anyhow, making a retry count optical visible. However, a consistent higher level operation name and an idempotency key to distinguish different requests could be a value. Riptide does not provide an operation name, does it? Anyhow, it should be easy to calculate a consistent operation name from the path template - something that is difficult from the HTTP client itself.

[whiskeysierra]

OpenTracing needs aggregate over spans anyhow, making a retry count optical visible

See also https://github.bus.zalan.do/SRE/opentracing/issues/6.
Visible yes, but indistinguishable from sequential requests to the same endpoint.

Anyhow, it should be easy to calculate a consistent operation name from the path template

I can imagine proving something for simplistic templates, but for more complex ones it might not be as easy to generate a consistent operation name.

[tkrop]

See also https://github.bus.zalan.do/SRE/opentracing/issues/6.

Interesting. Overlooked it. How do you correlate two spans that are retries, when you concurrently issue other requests to the same endpoint - as we did long time ago in our project? May be this issue solves only have the problem. Should all requests with retries be wrapped into an enclosing span for this remote request?

[whiskeysierra]

How do you correlate two spans that are retries, when you concurrently issue other requests to the same endpoint

Either by doing this:

retries be wrapped into an enclosing span

or

an idempotency key to distinguish different requests

[whiskeysierra]

Requires zalando/opentracing-toolbox#204

[whiskeysierra]

Instead of using an interceptor for Apache Http Client, maybe we can just register our new OpenTracingPlugin twice. Once for enclosing span and a second time after the failsafe plugin in case we have retries enabled.

[whiskeysierra]

#450 would introduce a more fine-grained approach to error handling which might change which of those exceptions we treat as an error (error tag set to true). For example a response being translated into an exception within the routing tree might not be what OpenTracing considers an error.