Merge pull request #1 from huyan0/master

Add  support for configurable OTLP export strategy with cumulative default strategy

text/metrics/0126-Configurable-Metric-Aggregations.md

@@ -0,0 +1,102 @@
+# A Proposal For SDK Support for Configurable Batching and Aggregations (Basic Views)
+
+Add support to the default SDK for the ability to configure Metric Aggregations.
+
+## Motivation
+
+OpenTelemetry's architecture separates the concerns of instrumentation and operation. The Metric Instruments
+provided by the Metric API are all defined to have a default aggregation. And, by default, aggregations are
+performed with all Labels being used to define a unit of aggregation. Although this is a good default
+configuration for the SDK to provide, more configurability is needed.
+
+There are 3 main use-cases that this proposal is intended to address:
+
+1) The application developer/operator wishes to use an aggregation other than the default provided by the SDK
+for a given instrument or set of instruments.
+2) An exporter author wishes to inform the SDK what "Temporality" (delta vs. cumulative) the resulting metric
+data points represent. "Delta" means only metric recordings since the last reporting interval are considered
+in the aggregation, and "Cumulative" means that all metric recordings over the lifetime of the Instrument are
+considered in the aggregation.
+3) The application developer/operator wishes to constrain the cardinality of labels for metrics being reported
+to the metric vendor/backend of choice.
+
+## Explanation
+
+I propose a new feature for the default SDK, available on the interface of the SDK's MeterProvider implementation, to configure
+the batching strategies and aggregations that will be used by the SDK when metric recordings are made. This is the beginnings
+of a "Views" API, but does not intend to implement the full View functionality from OpenCensus.
+
+The basic API has two parts.
+
+* InstrumentSelector - Enables specifying the selection of one or more instruments for the configuration to apply to.
+  - Selection options include: the instrument type (Counter, ValueRecorder, etc), and a regex for instrument name.
+  - If more than one option is provided, they are considered additive.
+  - Example: select all ValueRecorders whose name ends with ".duration".
+* View - configures how the batching and aggregation should be done.
+  - 3 things can be specified: The aggregation (Sum, MinMaxSumCount, Histogram, etc), the "temporality" of the batching,
+    and a set of pre-defined labels to consider as the subset to be used for aggregations.
+    - Note: "temporality" can be one of "DELTA" and "CUMULATIVE" and specifies whether the values of the aggregation
+      are reset after a collection is done or not, respectively.
+  - If not all are specified, then the others should be considered to be requesting the default.
+  - Examples:
+    - Use a MinMaxSumCount aggregation, and provide delta-style batching.
+    - Use a Histogram aggregation, and only use two labels "route" and "error" for aggregations.
+    - Use a quantile aggregation, and drop all labels when aggregating.
+
+In this proposal, there is only one View associated with each selector.
+
+As a concrete example, in Java, this might look something like this:
+
+```java
+ // get a handle to the MeterSdkProvider (note, this is concrete name of the default SDK class in java, not a general SDK)
+ MeterSdkProvider meterProvider = OpenTelemetrySdk.getMeterProvider();
+
+ // create a selector to select which instruments to customize:
+ InstrumentSelector instrumentSelector = InstrumentSelector.newBuilder()
+  .instrumentType(InstrumentType.COUNTER)
+  .build();
+
+ // create a configuration of how you want the metrics aggregated:
+ View view =
+      View.create(Aggregations.minMaxSumCount(), Temporality.DELTA);
+
+ //register the configuration with the MeterSdkProvider
+ meterProvider.registerView(instrumentSelector, view);
+```
+
+## Internal details
+
+This OTEP does not specify how this should be implemented in a particular language, only the functionality that is desired.
+
+A prototype with a partial implementation of this proposal in Java is available in PR form [here](https://github.com/open-telemetry/opentelemetry-java/pull/1412)
+
+## Trade-offs and mitigations
+
+This does not intend to deliver a full "Views" API, although it is the basis for one. The goal here is
+simply to allow configuration of the batching and aggregation by operators and exporter authors.
+
+This does not intend to specify the exact interface for providing these configurations, nor does it
+consider a non-programmatic configuration option.
+
+## Prior art and alternatives
+
+* Prior Art is probably mostly in the [OpenCensus Views](https://opencensus.io/stats/view/) system.
+* Another [OTEP](https://github.com/open-telemetry/oteps/pull/89) attempted to address building a Views API.
+
+## Open questions (to be resolved in an official specification)
+
+1. Should custom aggregations be allowable for all instruments? How should an SDK respond to a request for a non-supported aggregation?
+2. Should the requesting of DELTA vs. CUMULATIVE be only available via an exporter-only API, rather than generally available to all operators?
+3. Is regex-based name matching too broad and dangerous? Would the alternative (having to know the exact name of all instruments to configure) be too onerous?
+4. Is there anything in this proposal that would make implementing a full Views API (i.e. having multiple, named aggregations per instrument) difficult?
+5. How should an exporter interact with the SDK for which it is configured, in order to change aggregation settings?
+6. Should the first implementation include label reduction, or should that be done in a follow-up OTEP/spec?
+7. Does this support disabling an aggregation altogether, and if so, what is the interface for that?
+8. What is the precedence of selectors, if more than one selector can apply to a given Instrument?
+
+## Future possibilities
+
+What are some future changes that this proposal would enable?
+
+- A full-blown views API, which would allow multiple "views" per instrument. It's unclear how an exporter would specify which one it wanted, or if it would all the generated metrics.
+- Additional non-programmatic configuration options.

text/metrics/0131-otlp-export-behavior.md

@@ -0,0 +1,43 @@
+# OTLP Exporters Configurable Export Behavior
+
+ Add support for configurable export behavior in OTLP exporters.
+
+ The expected behavior required are 1) exporting cumulative values since start time by default, and 2) exporting delta values per collection interval when configured.
+
+## Motivation
+
+1. **Export behavior should be configurable**: Metric backends such as Prometheus, Cortex and other backends supporting Prometheus time-series that ingest data from the Prometheus remote write API, require cumulative values for cumulative metrics and additive metrics, per collection interval. In order to export metrics generated by the SDK using the Collector, incoming values from the SDK should be cumulative values. Note than in comparison, backends like Statsd expect delta values for each collection interval. To support different backend requirements, OTLP metric export behavior needs to be configurable, with cumulative values exported as a default. See discussion in [#731](https://github.com/open-telemetry/opentelemetry-specification/issues/731).
+2. **Cumulative export should be the default behavior since it is more reliable**: Cumulative export also addresses the problem of missing delta values for an UpDownCounter. The final consumer of the UpDownCounter metrics is almost always interested in the cumulative value. If the Metrics SDK exports deltas and allows the consumer aggregate cumulative values, then any deltas lost in-transit will lead to inaccurate final values. This loss may impact the condition on which an alert is fired or not. On the other hand, exporting cumulative values guarantees only resolution is lost, but the value received by the final consumer will be correct eventually.
+    1. *Note:* The [Metrics SIG](https://docs.google.com/document/d/1LfDVyBJlIewwm3a0JtDtEjkusZjzQE3IAix8b0Fxy3Y/edit#heading=h.fxqkpi2ya3br) *July 23 and July 30 meetings concluded that cumulative export behavior is more reliable.* For example, Bogdan Drutu in [#725](https://github.com/open-telemetry/opentelemetry-specification/issues/725) notes “When exporting delta values of an UpdownCounter instrument, the export pipeline becomes a single point of failure for the alerts, any dropped "delta" will influence the "current" value of the metric in an undefined way."
+
+## Explanation
+
+In order to support Prometheus backends using cumulative values as well as other backends that use delta values, the SDK  needs to be configurable and support an OTLP exporter which handles both cumulative values by default and delta values for export. The implication is that the OTLP metric protocol should support both cumulative and delta reporting strategies.
+
+Users should be allowed to declare an environment variable or configuration field that determines this setting for OTLP exporters.
+
+## Internal details
+
+OTLP exporters can report using the behavior it needs to the Metrics SDK. The SDK can merge the previous state of metrics with current value and return the appropriate values to the exporter.  
+
+Configurable export behavior is already coded in the Metrics Processor component in the [Go SDK](https://github.com/open-telemetry/opentelemetry-go/pull/840). However, this functionality is hardcoded today and would need to rewritten to handle user-defined configuration. See the OTLP metrics definition in [PR #193](https://github.com/open-telemetry/opentelemetry-proto/pull/193), which support both export behaviors.
+
+## Trade-offs and mitigations
+
+ High memory usage: To support cumulative exports, the SDK needs to maintain state for each cumulative metrics. This means users with high-cardinality metrics can experience high memory usage.
+
+The high-cardinality metrics use case could be addressed by adding the metrics aggregation processor in the Collector. This would enable the Collector, when configured as an Agent, to support converting delta OTLP to Cumulative OTLP. This functionality requires a single agent for each metric-generating client so that all delta values of a metric are converted by the same Collector instance.
+
+## Prior art and alternatives
+
+A discussed solution is to convert deltas to cumulative in the Collector both as an agent and as a standalone service. However, supporting conversion in the Collector when it is a standalone service requires implementation of a routing mechanism across all Collector instances to ensure delta values of the same cumulative metric are aggregated by the same Collector instance.
+
+## Open questions
+
+As stated in the previous section, delta to cumulative conversion in the Collector is needed to support Prometheus type backends. This may be necessary in the Collector in the future because the Collector may also accept metrics from other sources that report delta values. On the other hand, if sources are reporting cumulative values, cumulative to delta conversion is needed to support Statsd type backends.
+
+The future implementation for conversions in the Collector is still under discussion. There is a proposal is to add a [Metric Aggregation Processor](https://github.com/open-telemetry/opentelemetry-collector/issues/1422) in the Collector which recommends a solution for delta to cumulative conversion.
+
+## Future possibilities
+
+A future improvement that could be considered is to support a dynamic configuration from a configuration server that determines the appropriate export strategy of OTLP clients at startup.