Metric Exemplars SDK Specification

[jsuereth]

Fixes #1797

Changes

Add Exemplar specification and customization to the metrics specification.

Related issues #1260 #617

Related oteps #113

Notes

• This exposes two extensible interfaces. It's fine to drop these if they are contentious. The key important piece we must keep is the ability to turn OFF exemplars and the ability to limit exemplars to only measurements with associated sampled spans on context.
• This diverges from the Exemplar OTEP in that it is less restrictive on behavior of the exemplar reservoir (currently). Additionally the aggregators available when the exemplar OTEP was written are not yet specified. We can refine this in a follow on PR as necessary. We should align any sampling decisions w/ the future of tracing sampling.
• There are still some TODOs for discussion.

[jmacd]

LGTM

[jmacd]

FYI I have been working on trace sampling lately, and there is a lot of alignment between that work and this work. I have made claims that we can use weighted sampling to implement a hard rate limit on spans inside a tail sampler, and by extension I claim to know how to do the same for histogram exemplars. For a histogram aggregator outputting high-resolution histogram buckets you might not want one exemplar per bucket, and with weighted sampling we can produce a fixed number (i.e., using a reservoir) of exemplars with an equal expected number of spans per bucket.

This simple approach uses two passes, documented in the README of the https://github.com/lightstep/varopt repository. The more complicated approach can do this approximately in a single pass, which I also discussed briefly in today's Sampling SIG. See this draft. I will elaborate on this in a prototype exemplar reservoir tail sampler based on Varopt.

[jsuereth]

@jmacd I agree I think we can do way better than status quo, especially for high-fidelity histograms. I re-tweaked the wording to leave room for an exemplar reservoir sampling specification when those land.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[reyang]

LGTM, I've left some minor comments.

[aabmass]

Same comment as above

[aabmass]

This should be marked optional right?