Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the HTTP header format proposal for TraceContext propagation. #1

Merged
merged 2 commits into from
Sep 6, 2017
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Add the HTTP header format proposal for TraceContext propagation.
  • Loading branch information
Bogdan Drutu committed Mar 31, 2017
commit 8257a889e7427c20420db44ccc0aaa80a41f32ed
88 changes: 88 additions & 0 deletions HTTP_HEADER_FORMAT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Trace Context HTTP Header Format
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which systems are committed to support this? Which systems would you like to support this? Might be helpful to @-mention people from the latter so we can have any debates before this is merged.

Copy link

@codefromthecrypt codefromthecrypt Apr 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

putting in 2p eventhough the question was for @bogdandrutu :)

TL;DR; I'd expect tracing systems which maintain all of their tracers to make a more top-down decision, but yeah not currently the case in zipkin as this trace context is compatible with zipkin's wire format.

Currently, some zipkin-compatible tracers support vendor-specific or not widely used formats. Those types of tracers will have an easier time with this since zipkin's trace context isn't inherently incompatible with this specification (at the moment). I'd expect the cross-section of google+zipkin users to be first to ask, as it is likely google will land some variant of this first (grpc, cloud services and stackdriver instrumentation).

Similar to other things that happen, when that demand occurs it is usually in a repo or two. For example, our first requests for StackDriver and X-Ray trace support came from sleuth issues list. Tracers run independently and can move to support something sooner or later. I often ping people across tracers on things like this so that they can weigh-in before organic demand hits.

Regardless, support or not support lies in the scope of each tracer to decide until there are server implications like the trace context is incompatible or too wide to store in zipkin.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the current format (that is compatible with ZIpkin and Google) we try to make at least these systems to work with this format. Having a common place for the specs (maybe some simple implementation in multiple languages) is one of the goal.

Anyone who is interested in using this format is welcome to join the effort and send patches/PRs etc.


Date: 31/03/2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see why this is here, though it will be maintenance causing.. wonder if there's a badge to auto-include the git timestamp?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.


A trace context header is used to pass trace context information across systems
for a HTTP request. Our goal is to share this with the community so that various
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hear a lot of pain around tracing things like messages buses / kafka / etc. Might be worth intentionally decoupling from HTTP and making this about any messaging tech that has room for metadata/headers.

tracing and diagnostics products can operate together, and so that services can
pass context through them, even if they're not being traced (useful for load
balancers, etc.)

# Format

## Header name

`Trace-Context`
Copy link

@costinm costinm May 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'Via' is a standard header with pretty similar semantics. It is also in hpack.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trace-Context can be mistaken with something carrying the baggage or context properties. This is id. We decided to use Request-Id for correlation protocol in .NET. If we will be able to converge on format - it may be a good one to use.

Request-Id is quite descriptive for the header purpose I think.


## Field value

`base16(<version><version_format>)`

The value will be US-ASCII encoded (which is UTF-8 compliant).

### Version

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Version or variant? maybe mention in docs this could be used by variants of the spec? or just wait and see..

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(As a random, interested passerby:)

The problem with having an explicit version field is that you risk breaking all receivers as soon as you increment the field, which means a user can't do a staged rollout of a system that uses a new version of the protocol.

A better strategy would be to reserve some of the options bits and then requiring current implementations to set them to zero and ignore any trailing data. This way, you can evolve your protocol gradually.


Is a 1-byte representing a uint8 value.

### Version = 0

#### Format

`<trace-id><span-id><trace-options>`

All fields are required.

#### Trace-id

Is the ID of the whole trace forest. It is represented as a 16-bytes array,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The room in 🇩🇪 seemed to favor a variable-length field here... I thought that was for good reason, too: 128 bits of precision is completely unnecessary for the purposes of most tracing systems, yet insufficient for others we heard about that needed 192. Why does this spec need to try and get this right? (We can set a hard max at 32 bytes or something if you are concerned about unbounded allocations)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fwiw As a unit, zipkin usually implement things a lot later than the first request after significant cross-project interest exists, and there are hands to do it. Ex 128bit trace ids was implemented probably 2 years after the first sites started forking zipkin to accomplish this. This happened after a couple more recent glitches occurred including users of AWS lambda not being able to reuse the uuid there, as well zipkin users who have hybrid sites that employ stackdriver (which is 128bit). This topic has been discussed at length on other forums, so not trying to replicate it here. Suffice to say that 128bit is definitely in use in zipkin sites.

To make things less conjectury, it would be nice to have a matrix of tracing systems and their trace context requirements someplace, maybe on a google doc or otherwise.

In practice, Zipkin has never been asked directly to support a bit length higher than 128. However, we have been asked for more "squishy" things like opaque strings. Also, some tracers like sleuth were asked for lambda, which has an interesting timestamp+identifier trace id format. Having a matrix around can make things more objective as people can see if all of the systems they care about can work with a spec or not. It won't be about most, rather what site is running. For example, 9/10 is meaningless if that last one is the only one they use.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we accept variable size context we should definitely have a max size for both trace_id/span_id if 32 bytes fits all the systems then we should probably change to that. Do you know any system that is willing to change to a common format and uses more than 128bits? As @adriancole mentioned for the moment the systems that are interested in using this format are Zipkin and Google and both are fine with 128bits.

One option to move forward is to make v0 having fixed sizes and we can define the v1 later when we have a clear system that wants to use this and requires more than 128bits.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also filed an issue to keep track of who wants/is willing to use this format.

Please add your project here #4. Add any extra requirements.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tracelytics uses 160-bit trace IDs currently, and we are interested in a common format. FWIW AWS X-Ray uses a 128-bit trace ID format consisting of a 64-bit timestamp concatenated (using a hyphen) to a 96-bit unique ID, both encoded in hex.

e.g., `0x4bf92f3577b34da6a3ce929d0e0e4736`. All bytes 0 is considered invalid.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is 0x prefix intended here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.


Implementation may decide to completely ignore the trace-context if the trace-id
is invalid.

#### Span-id

Is the ID of the caller span (parent). It is represented as a 8-bytes array,
e.g., `0x00f067aa0ba902b7`. All bytes 0 is considered invalid.

Implementation may decide to completely ignore the trace-context if the span-id
is invalid.

#### Trace-options
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's odd to me that we support a versioning bit, yet have this 4-byte thing we only have 1 bit specified for... we could alternatively just have a single byte for version 0 and skip all of the endianness discussion.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was expecting this to fill up (the 4-byte options) with things like what you proposed (sampling probability). Uber also suggested an extra bit for deferred sampling decision.

I am trying to get for the moment the minimum requirement.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am trying to collect data about how many bytes we need. So far the list contains 3 (based on Jager requirements).

If the list is less than 8 we can definitely go with 1 byte for the options in v0.

Should we have the sampling probability that @bhs proposed as a separate field?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. 1-byte used for options.


Controls tracing options such as sampling, trace level etc. It is a 4-bytes
representing a 32-bit unsigned integer in little-endian order. The least

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

java's default is big-endian.. there are a lot of java programmers

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ps to elaborate on this, there's a couple things in play. On one hand, if you have only one flag, the endianness isn't an issue. Most users have sampled set as a single bit, and that's the only flag used in zipkin routinely.

On the other hand, the platform default encoding of java as big-endian results in many not being aware of endian-ness in general. Mistakes coding anything binary have been difficult in zipkin, resulting in json actually. While the case endianness is "little" elsewhere, anectodally I've noticed those who have little endian platforms already know how to do big endian. Main thing is that they get annoyed when endian isn't documented. Ex when folks were doing thrift encoding I remember someone rather annoyed that thrift tbinary format was big endian without saying so (because it was implicitly that in java).

food for thought

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

java's default is big-endian.. there are a lot of java programmers

not only that, but big-endian is also "network byte order". Wikipedia says:

Big-endian is the most common format in data networking; fields in the protocols of the Internet protocol suite, such as IPv4, IPv6, TCP, and UDP, are transmitted in big-endian order. For this reason, big-endian byte order is also referred to as network byte order.

If I can write the 32bit flags number as 0x00000001, it seems more natural to have 00000001 in the header.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know that network byte order is big-endian. But probably 99% of the CPU in used are little-endian.

Let's vote here #3 for this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think @bhs proposal to reduce this to 1 byte is an alternative option. Updated the issue with this possibility.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. 1-byte used for options.

significant bit provides recommendation whether the request should be traced or
not (1 recommends the request should be traced, 0 means the caller does not make
a decision to trace and the decision might be deferred). The flags are
recommendations given by the caller rather than strict rules to follow for 3
reasons:

1. Trust and abuse.
2. Bug in caller
3. Different load between caller service and callee service might force callee
to down sample.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming sampling logic may differ much between vendors - why place sampling flag to the trace context? Why not pass it as a separate header that is subject for vendor-specific logic? In some cases to resolve the issues you describe - extension libraries will need more than just a sampling flag. Most probably some additional information from baggage or other headers.

You may also have multi-tier sampling. Take an example of local agent mode that @bogdandrutu demo-ed for census. You may want to sample data that you send to backend with sampling rate 0.01 and locally - 0.1. So you'd need more than one bit to carry both sampling decisions.

Another consideration - properties of Trace-Context belongs to the new span you create. So developer or extension library can access those fields by taking the "running" span details. You can use those identifiers for the non-tracing needs. Sampling flag will not be one of a span properties (will it?). So you have no access to the original value of that flag. And you may not need any options as you may only care about the identifiers. So having options in Trace-Context looks inconsistent.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SergeyKanzhelev consistent sampling across the whole trace is very important characteristic. If the sampling decision is not propagated and the trace spans multiple implementations t1,...,tN, then each implementation will have to make a sampling decision over and over, and the probability that you will capture the whole trace becomes very small, p1 * ... * pN.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yurishkuro if you are making a sampling decision based on traceid you will have min(p1, p2, .. pN) number of full traces, not the multiplication. If all probabilities match - all the traces will be collected fully,

When you do sampling you may need to estimate the count of spans exhibit certain properties based on sampled data. In case of statistical sampling you may just multiple the raw count of spans to sampling percentage to get statistically accurate number.

If sampling decision forced from above without the information on sampling percentage of originator - you cannot do this type of estimations.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

statistical sampling on every layer is just one example of different type of sampling you may want to implement and you will need more than a bit of informaiton

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you are making a sampling decision based on traceid...
When you do sampling you may need to estimate the count of spans

Having the sampling bit does not prevent you from passing extra data or making more elaborate decisions, but it's not part of the proposed standard. NOT having the sampling bit pretty much guarantees that the trace will be broken, unless every implementation makes the decision based on the exact same formula, like traceId < p * 2^128. which is again not a part of the proposed standard.

Sampling bit is a recommendation. If a service can respect it and handle the volume - great. If it cannot respect it 100%, maybe it can respect it for "more important" spans like RPCs, and shed the load by dropping in-process spans & metrics. Essentially it does not put any restrictions on how you want to implement sampling, but still provides a way to achieve consistent sampling across the stack, if the volume allows it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is precisely my question. Do you think it will be typical for libraries to trust this flag? Or every vendor and library will implement own logic so this flag will never be used? Should this standard define the flag or let customers decide on sampling algorithm across services in their org as a decision separate from the data correlation protocol?

Beauty of sampling flag is ability to implement solutions like forced data collection for period of time or specific span name. Also it removes the need to synchronize the sampling decision algorithm.

On negative side - services looses control of the data volume and statistical accuracy of collected data.

Protocol is optimized for a single team owning many components with relatively similar load. It is not always the case. Every component may be owned by a team which want to play nice and contribute to the overall correlation story. But have a bigger priority to fully control telemetry volume and distribution collected from this component.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On negative side - services looses control of the data volume and statistical accuracy of collected data.

I don't agree with this assessment - the flag is recommendation, an implementation does not have to respect it if it thinks it can do a better job.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it will be typical for libraries to trust this flag?

Yes, I think it's a very common implementation to simply trust the flag, in the absence of other knowledge about the system.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. I'm not against the flag that will force all layers to record a specific trace. It is very useful for troubleshooting scenarios.
  2. I also agree that collecting the entire trace as often as possible is very useful.
  3. It is unavoidable that downstream services will collect more telemetry than was sampled in by front door. So there will be incomplete traces.

What I want to avoid is the situation when you heavily rely on implementation detail of upstream component sampling algo. Also I want to make sure there is an easy to replicate on any language mechanism to control the flood of telemetry in case of non-matching load patterns. Third, for many Application Insights scenarios we need to keep the sampling percentage that was used to sample telemetry out so we can run statistical algorithms on telemetry and recognize patterns. So single bit will not generally work for us as a universal sampling mechanism.

I'd propose to have a debug flag with the semantic of occasional forceful tracing across the system. Sampling flag then can be vendor specific.


The behavior of other bits is currently undefined.

#### Examples of HTTP headers

*Valid sampled Trace-Context:*

```
Value = 004bf92f3577b34da6a3ce929d0e0e473600f067aa0ba902b701000000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO this is over-optimized / pre-optimized for performance. In the context of an RPC, parsing these things is going to be a joke from an overhead standpoint... I'd rather have an id format that's human-readable, or at least human-parseable (without counting characters by hand, that is).

E.g., this:

Value = 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01000000

Copy link

@codefromthecrypt codefromthecrypt Apr 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

made comment elsewhere but I also agree a delimited format is near required from an operator POV. Especially as people cross-reference trace ids in other systems. Like splunk search shows something where one system logs the trace id as a logging key. Copy/paste this into zipkin or another system. There's also the "curl"ability which is "can a user work only having curl?" Straightforwardness is important for adoption, or rejection prevention.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is not unreasonable to add an extra character especially for the HTTP/String version to make this more readable.

If this is an agreement I think we should do it this way. I will start few issues where we can vote for certain things to get all these concerns/issues/questions resolved and CC interested people.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please vote/comment here: #2

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Used a delimiter between all fields.

Version = 0 (0x00)
TraceId = 0x4bf92f3577b34da6a3ce929d0e0e4736

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0x prefix will likely lead to encoding bugs as people will almost certainly copy/paste if we give opportunity to

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

SpanId = 0x00f067aa0ba902b7
TraceOptions = 0x1 // sampled
```

*Valid not-sampled Trace-Context:*

```
Value = 004bf92f3577b34da6a3ce929d0e0e473600f067aa0ba902b700000000
Version = 0 (0x00)
TraceId = 0x4bf92f3577b34da6a3ce929d0e0e4736
SpanId = 0x00f067aa0ba902b7
TraceOptions = 0x0 // not-sampled

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make an example that breaks with endian, especially if we use little endian

Copy link

@reta reta Apr 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adriancole This is terrific initiative. The proposal is looking quite reasonable. I think the only debatable piece is the presence of Version. Iе would better use your suggestion, like Variant, or Encoding, or Scheme. To give an example, I would like to see something like that: Z<traceid><spanid><options> where Z would tell me that this is Zipkin encoding scheme. For HTrace it could be H<spandId><options> (there is no separate Trace Id, it is encoded in Span Id, at least in 4.1). It is the same 8 bits but using characters instead of 0. Not sure if it makes sense to you guys. What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No more endianness.

I like the idea of maybe having multiple supported systems, but one of the main purpose is to make all these systems work together so having separate encodings for each system will not be that much of a win.

```