Simplified span2 format

[codefromthecrypt]

This is a proposal for a simpler json representation for tracers to use when reporting spans. This is a simplification of scope from what was formerly known as Zipkin v2 #939

Scope

The scope is only write, and only json. For example, this doesn't imply dropping support for using the same span ID across client and server RPC calls. It does imply converting this to the existing model at the point of collection.

Format

The following is the description of the simplified fields. Note that in the case of traceId it is fixed width vs variable. For 64-bit trace ids, simply pad left with 0s.

{
  "traceId": 16or32lowerHexCharacters,
  "parentId": 16lowerHexCharactersOrAbsentForRoot,
  "id": 16lowerHexCharacters,
  "kind": enum(CLIENT|SERVER|Absent),
  "name": stringOrAbsent,
  "timestamp": uint53EpochMicrosOrAbsentIfIncomplete,
  "duration": uint53MicrosOrAbsentIfIncomplete,
  "localEndpoint": existingEndpointTypeOrAbsent,
  "remoteEndpoint": existingEndpointTypeOrAbsent,
  "annotations": [
    {"timestamp": uint53EpochMicros, "value": string},
    ...
  ],
  "tags": {
    string: string,
    ...
  },
  "debug": trueOrAbsent,
  "shared": trueOrAbsent
}

Span name and timestamps are allowed to be missing to support late data, which is an edge case when spans are reported twice due to timeouts. Like previous versions, an empty span or service name defers the decision (usually to the other side of a shared span).

This format applied to a typical client span would look like this:

{
  "kind": "CLIENT",
  "traceId": "00000000000000005af7183fb1d4cf5f",
  "parentId": "6b221d5bc9e6496c",
  "id": "352bff9a74ca9ad2",
  "name": "query",
  "timestamp": 1461750040359130,
  "duration": 63874,
  "localEndpoint": {
    "serviceName": "zipkin-server",
    "ipv4": "172.19.0.3",
    "port": 9411
  },
  "remoteEndpoint": {
    "serviceName": "mysql",
    "ipv4": "172.19.0.2",
    "port": 3306
  },
  "tags": {
    "sql.query": "select distinct foo from bar"
  }
}

Impact on tracers

Tracers are the primary customers of this format, and in doing so they can have a closer alignment between field names and common operations. For example, the following is a "turing complete" span api which has almost direct mapping to this format.

start()
name(String name)
kind(Kind kind)
remoteEndpoint(Endpoint endpoint)
annotate(String value)
tag(String key, String value)
finish()
flush() <-- for reporting spans that didn't start and finish locally

It is important to remind that this format does not in any way drop support for the existing one. So old tracers can continue to operate.

Impact on servers

Collectors will decode and convert the new spans into the existing span model, backfilling "cs" and "cr" annotations and the "sa" binary annotation to make it appear as if it were sent in the old format. In order to do so, they need to know how to read the format. Choosing only json makes this simpler.

For http, we can add an endpoint or accept a media type header more specific than json. We could also choose a heuristic route. Kafka and similar systems which don't have headers would need a heuristic approach.

A heuristic approach would be to look for new fields. For example, startTimestamp or finishTimestamp will always be present in spans, so this can be used to decide which parsing rules to apply.

FAQ

Why just json?

This effort is geared at making tracer development simpler, and having the least scope possible to accomplish that. While thrift was supported in the old model, and has its merits, it is not a great format for people getting started, and it is very hard to debug. Those looking for reduced size of data can compress the spans before they are sent. Those who really like thrift can use the old model.

Why not special-case the tracer's endpoint?

The tracer's endpoint (localEndpoint) doesn't change from operation to operation, so you could reasonably ask why this isn't uploaded separate from the span data. The reason is that it lowers the scope of work. Many tracers report by adding spans to a queue flushed occasionally to a message producer. By keeping each span self-contained, this type of logic can be reused with no changes at all.

Why not change the query api, too?

This format is simpler because it removes the need to add the endpoint of the tracer to each annotation or tag. This simplification is possible in tracers as they have a one-one relationship with a traced service. At query time, a span can include both a client and a server, so it cannot be represented in this format.

Why not just remove support for the old model?

#939 started as an effort to change the semantics of spans to single-host, not just for upload, but also processing, storage and query. After a year of discussion, it is clear this is not a practical or pragmatic first step. Reasons include the benefits of the current model and a general lack of interest. This is a tactical alternative that provides many of the wins without the work of a big bang refactor.

What about async spans?

We recently started supporting async/one-way spans via "cs" -> "sr" annotations. Notice the span begins and ends with the request side of the RPC (there's no response sent by the server or received by the client). This translates to the new model as the following: clientSideSpan.start().flush() serverSideSpan.start().flush().

[codefromthecrypt]

cc @openzipkin/instrumentation-owners @openzipkin/core

[bvillanueva-mdsol]

@adriancole what does Absent mean in "kind"? What kind of use case it stands for? Will it mean that we put 'Absent' as the value or put nothing at all?

[bvillanueva-mdsol]

@adriancole We were talking about async traces with @jcarres-mdsol . Is it safe to assume that if finishTimestamp is empty with kind as Client, an async process is being started with that span?

[codefromthecrypt]

@adriancole <https://github.com/adriancole> what does Absent mean in "kind"? What kind of use case it stands for? Will it mean that we put 'Absent' as the value or put nothing at all?

sorry I was trying to hint you can either leave out the field "key" or set it to null. Kind is used to indicate the span is a client or server RPC span. This helps in translation to the old format, and means the user doesn't need do add annotations like "cs" and "cr". It also makes obvious what remoteAddress is (if client, the remoteAddress is server) local spans (ex ones that never leave the host), would leave kind out.

[codefromthecrypt]

We were talking about async traces with @jcarres-mdsol . Is it safe to assume that if finishTimestamp is empty with kind as Client, an async process is being started with that span?

The semantics of one-way spans are currently the same as RPC, except there's no response. This was the least efforts means we could come up with. There's an example of this in a recent pull request #1497

To do this in using the new format in a backwards compatible way, the key is being able to flush a span (report it to zipkin without attempting to calculate duration locally).

ex. span.kind(client).start() --> translates into a "cs" annotation if you flush at this point
-- propagated to the server then..
ex. span.kind(server).start() --> translates into an "sr" annotation if you flush at this point

Let me know if this doesn't answer your question

[bvillanueva-mdsol]

Thanks for the clarification @adriancole

[eirslett]

Note that 64 bit integers are tricky in JavaScript, which doesn't have them.
Interestingly, it's possible to represent the number of milliseconds since 1970 in only ~52 bits, while JavaScript can represent whole numbers up to 53 bits. (Just a coincidence) So it will silently work until we eventually encounter a "year 2000 problem".
It might be safer to store timestamps as strings (possibly compact/encoded), even if that could impact performance?

[codefromthecrypt]

Remember, 64 bit integers are tricky in JavaScript, which doesn't have them. Interestingly, it's possible to represent the number of milliseconds since 1970 in only ~52 bits, while JavaScript can represent whole numbers up to 53 bits. (Just a coincidence) So it will silently work until we eventually encounter a "year 2000 problem". It might be safer to store timestamps as strings (possibly compact/encoded), even if that could impact performance?

I hear you, and thanks for reminding about this. String dates are indeed idiomatic. Worth re-stating the rationale, since this issue is fairly special case. For example, it is making a shortcut form of the existing model (and would need to port without lossiness to the old one) Here's some things to think about. Everything currently uses the epoch micros, so it directly translates across. For example, when querying back (or looking at stored data) you would see the exact numeric form reported). Date formats usually don't have microsecond precision, either. Using a date format opens up to people in the N languages with uncoordinated tracers putting in an incorrect format (which we would immediately unwind it back into an epoch time). We'd get all the fun with people accidentally putting wrong timezone etc which I'm sure many of us are familiar with. On 53bits. that's a known constraint wrt timestamps (our ids don't use numeric form for this reason). Ex our current json parser guards on 53bits which if I'm not mistaken puts the Y2K problem at the year 2255. While that's a punt for sure, since data is usually retained at a few days, sometime between now and 2255 I'm sure we'd have a new overall format. Status quo does include the typical problem of people making ad-hoc or new tracers accidentally put epochMillis where they meant epochMicros. This is very easy to detect and fix. Heck we could probably even log it since when this mistake occurs the timestamps would look like they are from 1970. In other words, we can help make developing easier by making a development mode of zipkin which would crash POST on upload lint problems, such as 1970 dates. Thoughts?

[codefromthecrypt]

ps will change proposal to 53bit as that's a very good point regardless!

[codefromthecrypt]

@eirslett and anyone else just circling back.. would any of the below help on the date thing?

It is certainly the case that "timestamp" implies seconds since epoch. Would replacing "timestamp" with "micros" or adding a suffix clear up enough ambiguity to be worth its characters?
rename startTimestamp to startMicros or startTimestampMicros
rename finishTimestamp to finishMicros or finishTimestampMicros

[codefromthecrypt]

ps I had an offline request somewhere about the size impact of this. Spans that include more than one annotation or tag will be smaller than before due to de-duplication of the local endpoint.

It won't be as small as using single-character field names, but it isn't optimizing for that anyway.

The compression of json vs the same data compressed in thrift is an exercise we could do, but I don't think size is the major feature here, rather simplicity.

[shakuzen]

Would replacing "timestamp" with "micros" or adding a suffix clear up enough ambiguity to be worth its characters?

I personally think unambiguous field names are best, so startTimestampMicros and finishTimestampMicros sound good to me. This way, the field names largely document themselves.

I don't see much value in optimizing for size here, especially considering compression. The marginal performance that I presume would be gained would not be worth the loss in clarity. And if such performance were a significant factor, JSON is already the wrong choice.

[codefromthecrypt]

thinking about picking this up

[basvanbeek]

One thing to think about... maybe do this format in proto3 as it supports a canonical encoding in JSON so we have a simple roll your own JSON model but at the same time can handle higher performance binary encoding/decoding.

[codefromthecrypt]

One thing to think about... maybe do this format in proto3 as it supports a canonical encoding in JSON so we have a simple roll your own JSON model but at the same time can handle higher performance binary encoding/decoding.

interesting idea. regardless we'd want a json schema, too (ex as we sortof do in the openapi spec)

[codefromthecrypt]

So first wave will be implementing the collection side, and a storage impl for elasticsearch. This can flex as we review the implementation.

Note: We aim to reuse hex encoding of IDs as they exist today (in lower-hex). This makes sure existing log integration patterns don't break (ex logging usually adds trace ID to context and that is lower-hex which is same as json and headers). This is particularly useful in storage backends that contain both logging and tracing data (ex Elasticsearch).

Note part deux: usually, collectors receive single-host spans, but we'll need to address shared spans sooner or later. Kafka pipelines exist (theoretically) that send merged traces directly to storage (ex via zipkin-sparkstreaming). Also, people sometimes save off json (which has merged spans) and POST it later. While not a data format issue, it is an integration issue that will bite us similarly to how it bit stackdriver when a storage layer like ES stores single-host spans. https://github.com/GoogleCloudPlatform/stackdriver-zipkin/blob/master/translation/src/main/java/com/google/cloud/trace/zipkin/translation/SpanTranslator.java

I'll do my best to keep all operational notes like above close to the code, so that nothing is surprising.

[SergeyKanzhelev]

What do you think of making this JSON two-layered - envelope that has a basic correlation fields and versioned&typed payload. Something along the lines:

{
  "traceId": 32lowerHexCharacters,
  "parentId": 16lowerHexCharactersOrAbsentForRoot,
  "kind": enum(SPAN_CLIENT|SPAN_SERVER|SPAN_Local|ANNOTATION),
  "debug": trueOrAbsent,
  "data": 
    {
      "id": 16lowerHexCharacters,
      "name": string,
      "finishTimestamp": uint53EpochMicros,
      "localEndpoint": existingEndpointType,
      "remoteEndpoint": existingEndpointTypeOrAbsent,
      "tags": {
        string: string,
        ...
      },
      "annotations": [
        {"timestamp": uint53EpochMicros, "value": string},
        ...
      ],
    }
}

This way you can solve some scenarios like:

• You can send annotation separately from span without the need to specify the end time:

  {
    "traceId": 32lowerHexCharacters,
    "parentId": 16lowerHexCharactersOrAbsentForRoot,
    "kind": ANNOTATION,
    "debug": trueOrAbsent,
    "data": 
    {
        "name": string,
        "localEndpoint": existingEndpointType,
        "tags": {
          string: string,
          ...
        },
  

• Some annotations can be forced out of span in future so they could be indexed and queried easier
• Versioning of span types may be easier in future. Just encode version in kind field
• Other types like traces/exceptions/etc. may be supported as a payload in future. Those types will be correlated with traces without the need to change UI and tools. If type is unknown - tool can simply show JSON

[codefromthecrypt]

What do you think of making this JSON two-layered - envelope that has a basic correlation fields and versioned&typed payload. Something along the lines:

First thoughts were that this design was to reduce nesting, not add to it :) That said, at least this nesting will not introduce a collection to search through. Will comment more below, and regardless appreciate your interest.

This way you can solve some scenarios like: - You can send annotation separately from span without the need to specify the end time: in-flight work has both timestamps optional, as it was an oversight to

make them mandatory (for tags, not annotations, but anyway yeah). Reason is that there are use cases where data happens after the fact (even if it is exceptional and usually related to timeouts). Adding another layer of nesting doesn't make this easier or harder, imho.

- Some annotations can be forced out of span in future so they could be indexed and queried easier yeah that was an oversight in the sketch, not the impl. It doesn't require

a data envelope to handle.

- Versioning of span types may be easier in future. Just encode version in kind field The kind field is a shortcut currently used to eliminate bookend

annotations of "cs" "sr" etc. That's why it is an enum, not a open type. Encoding other data into that field would not be supported. For versioning, I think it is useful to define a schema and possibly a proto3 mapping.

- Other types like traces/exceptions/etc. may be supported as a payload in future. We wouldn't remove fields, though possibly add ones. I would be

unsurprised if we had an exception type at some point (in the span). There's no likelihood of defining a separate "trace" type in zipkin (as opposed to a list of spans). Regardless, there's no issue tracking future changes in json schema and/or proto3.

- Those types will be correlated with traces without the need to change UI and tools. If type is unknown - tool can simply show JSON There are clear benefits for having this a straight-forward data type,

including natural objects when code is generated from it. Unknown fields within this type can be skipped, as they are now, but changing this from a data type to be a generic carrier is beyond the intent. Data type currently fits the model of existing tracers who post some or all of a span as a unit. There's no expectation that they will start sending arbitrary data, and it is a bit early to guess how they would. Meanwhile, escalating this change from a data type to a trace-stained-carrier will put it at risk of not completing (again). TL;DR; let's keep this as a data type and fork any discussion off about a coordinated trace-identifier scoped carrier as a separate issue or thread. Let's make sure there any evaluation of such is diversely bought-into as we don't have enough hands to maintain things that aren't.

[SergeyKanzhelev]

@adriancole thanks for clarification. My understanding of schema was wrong I thought only these fields marked "Absent" can be omitted:

"kind": enum(CLIENT|SERVER|Absent),
"remoteEndpoint": existingEndpointTypeOrAbsent,
"debug": trueOrAbsent

So I wasn't clear on how you can report a span name for the runaway annotation. The intend with proposed envelope was to extract fields strictly required for correlation one level up and put all situation-specific fields inside the data object. Using kind you can strongly type those data objects further.

Also that proposal didn't change any data formats (except extended up kind to be a string) so I thought it may be incorporated now and make versioning easier in future.

Anyway, I see your point that this proposal doesn't solve any major problems you have immediate plans for and creates unnecessary complexity and overhead.

I really like how this schema simplify semantic of span!

[codefromthecrypt]

@adriancole <https://github.com/adriancole> thanks for clarification. My understanding of schema was wrong I thought only these fields marked "Absent" can be omitted:

heh you are right about the absent part, just that I didn't update the issue after working on it yesterday and noticing the missing absent on the timestamps. Will do now!

[codefromthecrypt]

updated about absent

[SergeyKanzhelev]

What about the name? Will you be able to report it when you have remote annotation only? Or it will be just empty?

[codefromthecrypt]

good point. we can accept absent span name and use the same semantics as what we currently do for empty (which is defer the naming choice). updated

[codefromthecrypt]

ok made a tracking issue for this #1644

[yurishkuro]

@adriancole what about the old format that was blocking openapi ?

[codefromthecrypt]

@adriancole <https://github.com/adriancole> what about the old format that was blocking openapi ?

binary annotations. openapi v2.0 cannot deal with oneOf values. openapi 3 can, but there's no codegen for it

[codefromthecrypt]

nearing the finish line here. One remaining thing left is honing the java api for span recording. We have an opportunity to make it easier and also match the schema 1-1. That would include the following

• Change IDs to validated strings
  • This doesn't mean we don't validate.. it is still hex 16 or 32 chars
  • This should be provably more efficient than repeatedly converting back/forth from longs
  • We can tell the difference between an ID that was propagated as 128 bit from one that had all high bits unset (helpful for looking at instrumentation)
  • Storage backends that keep fixed binary or uint64 can continue to (we'd just convert on the way in)
• Change IPs to validated strings
  • this is cheaper than repeatedly doing so and also packing and unpacking InetAddress instances
  • since we don't repeat endpoints anymore, it isn't as necessary to use integer encoding
  • This is a lot easier to read and less to debug than byte arrays or integers

By doing the above, the java struct looks exactly like the openapi struct. This makes generated code in any language look just like our native code, which means porting should be easier. This also makes everything easier to troubleshoot, as there are no conversions between types that don't always "toString" well.

Before, zipkin had to take special attention to reduce memory size as things were repeated a lot. While IDs are definitely longer this way (16 chars is more memory than a long), this seems a reasonable tradeoff. Moreover, if such was a big deal, the instrumentation could make a custom struct easier with v2 than today anyway.

cc @nicmunroe @anuraaga @shakuzen @jplock @llinder @devinsba

[codefromthecrypt]

oh yeah one more thing on ^^. It makes it gson, jackson, moshi friendly: very hard to get serialization wrong.

[codefromthecrypt]

On the java struct having validated strings, seems sensible so will try it out probably in a PR against the elasticsearch WIP.

[codefromthecrypt]

started on trace identifiers as string here #1721

[codefromthecrypt]

java api change to validated strings for IP and IDs resulted in 2x efficiency gains in benchmarks. #1721 It is incidentally also easier to write your own codec for. However, when we write to storage, we should still use the formal library as it normalizes IPv6, which makes string comparison more likely to work.

[codefromthecrypt]

I've thought about it and the "v2 library" will be able to encode v1 format (at least json). This is important as for example if this is used in brave or similar, we want tracers to be able to upgrade independently of the system. Ex Encoder.LEGACY_JSON would spit out the old format (with no dependency on v1 jars). This will happen after #1726

[jorgheymans]

@adriancole it's late, i don't have the energy to scrutinize 77 comments of this issue, but i have a hunch that this issue can be closed given v2 being out in the open for quite a while. Unless you object, i'll close it tomorrow.

[codefromthecrypt]

late or not. go for close. appreciate your usual scrutiny but agree this one would be a day of effort to summarize as.. we should close it ;)

[jorgheymans]

🎵 Another one bites the dust 🎵

@jcchavezs and at your request here's the Mercury emoji ☿️