how to search annotations or binaryAnnotations in kibana?

[dragontree101]

i send trace info to es5, and i want to search annotations or binaryAnnotations from kibana.

i have one span json like this:

{
  "_index": "zipkin-2016-11-17",
  "_type": "span",
  "_id": "AVhw8k-GQpNFwf9ty7hN",
  "_score": null,
  "_source": {
    "traceId": "0f56baad2910df69",
    "timestamp_millis": 1479363860752,
    "annotations": [
      {
        "endpoint": {
          "ipv4": "10.8.0.2",
          "port": 18088,
          "serviceName": "first-example-dev"
        },
        "value": "cs",
        "timestamp": 1479363860752000
      },
      {
        "endpoint": {
          "ipv4": "10.8.0.2",
          "port": 18088,
          "serviceName": "first-example-dev"
        },
        "value": "cr",
        "timestamp": 1479363860903752
      }
    ],
    "parentId": "20c251aac430bf39",
    "duration": 151752,
    "binaryAnnotations": [
      {
        "endpoint": {
          "ipv4": "10.8.0.2",
          "port": 19091,
          "serviceName": "thrift"
        },
        "value": true,
        "key": "sa"
      }
    ],
    "name": "secondadd",
    "collector_timestamp_millis": "2016-11-17T06:21:18.086+0000",
    "id": "f83891ef110158ef",
    "timestamp": 1479363860752000
  },
  "fields": {
    "timestamp_millis": [
      1479363860752
    ],
    "collector_timestamp_millis": [
      1479363678086
    ]
  },
  "sort": [
    1479363678086
  ]
}

i want to search annotations's endpoint's serviceName is first-example-dev, and i direct search first-example-dev from kibana, no result :(, and i search annotations:first-example-dev also nothing.

how can i search this content?

[codefromthecrypt]

we aren't setup to support Kibana as a first class citizen, since there's really no direct relationship here. We are focused on supporting the zipkin UI. You might have more luck going through normal Kibana support channels and reporting back anything interesting you find.

[codefromthecrypt]

just in case.. @openzipkin/elasticsearch does anyone happen to know what this might imply? I know that in zipkin itself annotation searches are nested terms queries

[mansu]

@adriancole I think @dragontree101 is asking for example ES queries on how to search for spans from the kibana UI. I think this is a valid use case that we should document since most people who use ES will also use Kibana. In practice, I found that Kibana is a better UI to search for traces esp when they are broken and don't show up the UI. We are currently doing the same, happy to share what we come up with shortly.

[codefromthecrypt]

@mansu well the point in suggesting reliable channels for Kibana questions isn't a question about if it is a valid use case, or not. Rather, we haven't yet established anything near reliability for a place to come to for Kibana support. We neither document, run or test it for starters! Only thing we have right now is a note about timestamps (and an ad-hoc test I did once to see if maybe it worked or not). Even if we did support Kibana, we'd want the community to know where to get more help, or help when one of those who know Kibana aren't looking.

Anyway...

I'm glad you're interested in helping, as that'd be the first step towards supporting something sustainably. If you come up with something, let's put it in the README in a section for Kibana. Then, maybe @bsideup and/or I can help make sure it is tested. That way, we don't accidentally break it, which would be sad.

Sound good!

[dragontree101]

thanks @adriancole and @mansu
i both use zipkin's ui and kibana ui, i think zipkin's ui find one trace very good.
but i put some other import info into annotation or binaryAnnotation, so i want to aggregation some annotation or binaryAnnotation 's key or value in spans in kibana, because of i think kibana's aggregation better than zipkin's ui.
:)

[codefromthecrypt]

well, it could only be better if you use elasticsearch :) ps whatever isn't good should at least be noted for improvement in with a screen shot of what's better (since good is subjective and not always guessable) https://github.com/openzipkin/zipkin-ui/issues

Anyway, in case it helps, here's a complex query that the http implementation sends on the wire. It might help you figure out what to do (or not).

ex.

QueryRequest.builder().serviceName("service")
    .spanName("methodcall")
    .addAnnotation("custom")
    .addBinaryAnnotation("BAH", "BEH").build();

becomes this http request:

POST zipkin/span/_search?allow_no_indices=true&expand_wildcards=open&ignore_unavailable=true
...

{
  "size" : 0,
  "query" : {
    "bool" : {
      "must" : {
        "match_all" : { }
      },
      "filter" : {
        "bool" : {
          "must" : [ {
            "range" : {
              "timestamp_millis" : {
                "from" : 0,
                "to" : 1479693481467,
                "include_lower" : true,
                "include_upper" : true
              }
            }
          }, {
            "bool" : {
              "should" : [ {
                "nested" : {
                  "query" : {
                    "term" : {
                      "annotations.endpoint.serviceName" : "service"
                    }
                  },
                  "path" : "annotations"
                }
              }, {
                "nested" : {
                  "query" : {
                    "term" : {
                      "binaryAnnotations.endpoint.serviceName" : "service"
                    }
                  },
                  "path" : "binaryAnnotations"
                }
              } ]
            }
          }, {
            "term" : {
              "name" : "methodcall"
            }
          }, {
            "nested" : {
              "query" : {
                "bool" : {
                  "must" : [ {
                    "term" : {
                      "annotations.value" : "custom"
                    }
                  }, {
                    "term" : {
                      "annotations.endpoint.serviceName" : "service"
                    }
                  } ]
                }
              },
              "path" : "annotations"
            }
          }, {
            "nested" : {
              "query" : {
                "bool" : {
                  "must" : [ {
                    "term" : {
                      "binaryAnnotations.key" : "BAH"
                    }
                  }, {
                    "term" : {
                      "binaryAnnotations.value" : "BEH"
                    }
                  }, {
                    "term" : {
                      "binaryAnnotations.endpoint.serviceName" : "service"
                    }
                  } ]
                }
              },
              "path" : "binaryAnnotations"
            }
          } ]
        }
      }
    }
  },
  "aggregations" : {
    "traceId_agg" : {
      "terms" : {
        "field" : "traceId",
        "size" : 10,
        "order" : {
          "timestamps_agg" : "desc"
        }
      },
      "aggregations" : {
        "timestamps_agg" : {
          "min" : {
            "field" : "timestamp_millis"
          }
        }
      }
    }
  }
}

[sammypbaird]

@adriancole , thanks for the sample elasticsearch query. I was having trouble querying the serviceName too, until I realized from your code snippet that the annotations are stored as a nested object in elasticsearch, which requires a different type of query. One question I had though was why the serviceName is stored for every annotation. I would assume serviceName is applicable to the whole span, and should be stored as a field on the span. Under what situations can you have different serviceNames for the same span?

@dragontree101 , here's a simpler query that you can use on elasticsearch to find spans that match a specific serviceName (in this example, I'm searching for spans with a serviceName of jdbc):

{
  "query": {
    "bool": {
      "should": [
        {
          "nested": {
            "query": {
              "term": {
                "annotations.endpoint.serviceName": "jdbc"
              }
            },
            "path": "annotations"
          }
        }
      ]
    }
  }
}

That is what worked for me.

[codefromthecrypt]

@adriancole <https://github.com/adriancole> , thanks for the sample elasticsearch query. I was having trouble querying the serviceName too, until I realized from your code snippet that the annotations are stored as a nested object in elasticsearch, which requires a different type of query. One question I had though was why the serviceName is stored for every annotation. I would assume serviceName is applicable to the whole span, and should be stored as a field on the span. Under what situations can you have different serviceNames for the same span?

In an RPC span, you have a different serviceName for the client and the server. Also, there's a special case for "ca"(Client Address) and "sa" (Server Address) binary annotations. our api contract is such that service name is a primary filter, so it is like.. I want a trace where service A added annotation foo=bar. http://zipkin.io/zipkin-api/#/paths/%252Ftraces Hope that answers the question.

[sammypbaird]

Sorry, I meant the raw spans (i.e. the spans you get when you query a trace from the zipkin api and pass in the parameter "raw"). Each raw span looks like they always have the same serviceName (in the raw trace, the client and server belong to different spans). I would imagine that the serviceName would exist at the same level as the span name, but instead it's populated for each annotation in the raw span, which makes querying from elasticsearch more difficult. I'm sure there is a reason for this, just wondering why.

[codefromthecrypt]

this is historical See #939 for a proposal to use a new model.

[codefromthecrypt]

closing as it is antiquated especially now we have a different model