Exemplars prototype

[cnnradams]

This PR introduces exemplars into the SDK (for details on exemplars, see OTEP #113)

These change are (of course) experimental, as no major backend supports "statistical" exemplars and there is no concrete SDK spec for exemplars yet (only an OTEP). So any feedback is very much appreciated :)

Built off of the views API prototype: #596

[jmacd]

This looks great!

My approval here should carry only so much weight because I'm not very familiar with the Python SDK but this code generally looks readable and feels familiar. I have been thinking about how precisely I would integrate sampling in the SDK, and came up with two possibilities. One of the possibilities is to implement exemplars in the Aggregators themselves, as you've done, an d another possibility is to implement something in the Accumulator that might support exemplars in a less Aggregator-dependent way. Of course, you're also showing why we want the Aggregator to see the Update, so that Histograms can select exemplars by bucket.

To get this behavior w/ the first approach is straight forward, though it requires the Aggregator API to receive the excluded labels (as you have done), whereas up until now the Aggregators only know about numbers and math--labels are orthogonal to Aggregator state.

I thought about the other possibility (that exemplars can be Accumulator functionality) because it would be nice if we could compose something with any other aggregator to make this work. Imagine a class, named ExemplarAggregator, that can be used to select Aggregators in parallel any other aggregation (e.g., Histogram, Exact, ...). This is less straight forward, because now there's some kind of dependency between exemplar selection and an independent aggregator, and that's why I'm approving your approach.

For the record though, I put up a draft PR that begins this other approach here: open-telemetry/opentelemetry-go#1023. The idea is that the Accumulator can be told which dimensions to include (thus which to exclude) for each metric instrument, and that it therefore knows when to call the sampling manager. See: https://github.com/open-telemetry/opentelemetry-go/pull/1023/files#diff-4c3de179542bac3c5ceacd2090215160R225. I haven't worked out the details.

[lzchen]

Are you not able to just link to the python file instead?

[aabmass]

You can also do something like this if you want it inlined in the docs.

.. literalinclude:: basic_trace.py
     :language: python
     :lines: 1-

[cnnradams]

did literalinclude of the python examples, would have preferred a link to the jupyter notebook but this will work for now

[lzchen]

Nit: Can we rename basic_meter to metrics and put this folder under that in the examples?

[aabmass]

Looking at the algorithm Josh linked, I don't think merging like this would give a uniform sample. In the extreme case, imagine k = 1 and you have two samplers; sampler1's rand_count = 1 (population size of 1) and sampler2 finishes with rand_count = 10. sampler1's single value would be weighted much more strongly than any of the values sampler2 saw.

E.g. sampler1 see value [0], sampler2 sees values [1, 2, 3, ..., 10] but keeps only k = 1 sample, say, [6]. Randomly sampling k = 1 values from [0, 6] would give [0] 50% of the time, even though it was one of 11 values sampled in the whole population.

I think there is a similar issue when arg_count > k. The wiki page says each item is sampled with probability k/n; if the samplers being merged saw different n, then you would end up giving more weight to the sampler with smaller n.

[cnnradams]

as discussed offline, just keeping the second set's exemplars with that assumption based on the current SDK implementation. Might need to change in the future but for now that keeps this much simpler

[aabmass]

I love the Jupyter Notebook! I think it would be awesome to add nbsphinx to the docs setup so you could include the notebook directly in the docs, if others are cool with that.

[cnnradams]

I love the Jupyter Notebook! I think it would be awesome to add nbsphinx to the docs setup so you could include the notebook directly in the docs, if others are cool with that.

I actually tried to use nbsphinx, and found that the notebook was awful looking inline, due to its margins being the full page but the docs being only half a page. Maybe if I put it on its own page it would be different?

[aabmass]

I actually tried to use nbsphinx, and found that the notebook was awful looking inline, due to its margins being the full page but the docs being only half a page. Maybe if I put it on its own page it would be different?

Ya, not too sure, ive never used it myself. If nothing else works, you could probably render it to HTML then link or embed that somehow?

[aabmass]

A few more small nits

[aabmass]

If you know the exact keys and types, try a TypedDict

[cnnradams]

We don't know the exact types - it may change depending on what aggregators need to know

[aabmass]

You mean for custom aggregators?

[cnnradams]

the values already used are int, str, bool, list, and there is no restriction on passing in something like a float to the config dict

[aabmass]

I don't follow you. Are you saying you would use the same key with different value types depending on the aggregator?

[jmacd]

I left a lengthy remark on this topic here:
open-telemetry/opentelemetry-specification#617 (comment)

[cnnradams]

Closing for now.