Add exporter API and more details about SDK implementation

specification/library-guidelines.md

@@ -17,15 +17,15 @@ The document does not attempt to describe a language library API. For API specs
 4. Language library implementation must be clearly separated into wire protocol-independent parts that implement common logic (e.g. batching, tag enrichment by process information, etc.) and protocol-dependent telemetry exporters. Telemetry exporters must contain minimal functionality, thus enabling vendors to easily add support for their specific protocol to the language library.
 
 
-## Language Library Generic Design
+# Language Library Generic Design
 
 Here is a generic design for a language library (arrows indicate calls):
 
 ![Language Library Design Diagram](img/library-design.png)
 
-### Expected Usage
+## Expected Usage
 
-The OpenTelemetry Language Library will be composed of 2 packages: API package and SDK package.
+The OpenTelemetry Language Library is composed of 2 packages: API package and SDK package.
 
 Third-party libraries and frameworks that want to be instrumented in OpenTelemetry-compatible way will have a dependency on the API package. The developers of these third-party libraries will add calls to telemetry API to produce telemetry data.
 
@@ -34,48 +34,55 @@ Applications that use third-party libraries that are instrumented with OpenTelem
 In order to enable telemetry the application must take a dependency on the OpenTelemetry SDK, which implements the delivery of the telemetry. The application must also configure exporters so that the SDK knows where and how to deliver the telemetry. The details of how exporters are enabled and configured are language specific.
 
 
-### API and Minimal Implementation
+## API and Minimal Implementation
 
 The API package is a self-sufficient dependency, in the sense that if the end-user application or a third-party library depends only on it and does not plug a full SDK implementation then the application will still build and run without failing, although no telemetry data will be actually delivered to a telemetry backend.
 
 This self-sufficiency is achieved the following way.
 
-The API dependency will contain a minimal implementation of the API. When no other implementation is explicitly included in the application no telemetry data will be collected. Here is what active components will look like in this case:
+The API dependency contains a minimal implementation of the API. When no other implementation is explicitly included in the application no telemetry data will be collected. Here is what active components look like in this case:
 
 ![Minimal Operation Diagram](img/library-minimal.png)
 
 It is important that values returned from this minimal implementation of API are valid and do not require the caller to perform extra checks (e.g. createSpan() method should not fail and should return a valid non-null Span object). The caller should not need to know and worry about the fact that minimal implementation is in effect. This minimizes the boilerplate and error handling in the instrumented code.
 
 It is also important that minimal implementation incurs as little performance penalty as possible, so that third-party frameworks and libraries that are instrumented with OpenTelemetry impose negligible overheads to users of such libraries that do not want to use OpenTelemetry too.
 
-### SDK Implementation
+## SDK Implementation
 
-The SDK implementation is a separate (optional) dependency. When it is plugged in it substitutes the minimal implementation that is included in the API package (exact substitution mechanism is language dependent). Here is what active components will look like in this case:
+SDK implementation is a separate (optional) dependency. When it is plugged in it substitutes the minimal implementation that is included in the API package (exact substitution mechanism is language dependent).
+
+SDK implements core functionality that is required for translating API calls into telemetry data that is ready for exporting. Here is how OpenTelemetry components look like when SDK is enabled:
 
 ![Full Operation Diagram](img/library-full.png)
 
-It is recommended to decouple common parts of the implementation from protocol-specific Telemetry Exporters. Library designers are encouraged to specify an internal Telemetry Exporter API that defines how protocol-independent and protocol-dependant parts interact. The boundary and design goals for such API will be:
+SDK defines an [Exporter interface](sdk-exporter.md). Protocol-specific exporters that are responsible for sending telemetry data to backends must implement this interface.
+
+SDK also includes optional helper exporters that can be composed for additional functionality if needed.
+
+Library designers need to define the language-specific `Exporter` interface based on [this generic specification](sdk-exporter.md). 
+
+### Protocol Exporters
+
+Telemetry backend vendors are expected to implement [Exporter interface](sdk-exporter.md). Data received via Export() function should be serialized and sent to the backend in a vendor-specific way.
 
-- Place common functionality such as queuing, batching, tagging, etc. in the protocol-independent parts of the implementation. This functionality will be applicable regardless of what Telemetry Exporter is used.
+Vendors are encouraged to keep protocol-specific exporters as simple as possible and achieve desirable additional functionality such as queuing and retrying using helpers provided by SDK.
 
-- Minimize burden of implementation for protocol-dependant telemetry exporters. The Telemetry Exporter is expected to be primarily a simple telemetry data encoder and transmitter.
+End users should be given the flexibility of making many of the decisions regarding the queuing, retrying, tagging, batching functionality that make the most sense for their application. For example, if an application's telemetry data must be delivered to a remote backend that has no guaranteed availability the end user may choose to use a persistent local queue and an `Exporter` to retry sending on failures. As opposed to that for an application that sends telemetry to a locally running Agent daemon, the end user may prefer to have a simpler exporting configuration without retrying or queueing. 
 
-- Use efficient data structures that are well suited for fast serialization to wire formats and minimize the pressure on memory managers. The latter typically requires understanding of how to optimize the rapidly-generated, short-lived telemetry data structures to make life easier for the memory manager of the specific language. General recommendation is to minimize the number of allocations and use allocation arenas where possible, thus avoiding explosion of allocation/deallocation/collection operations in the presence of high rate of telemetry data generation.
 
-### Alternate Implementations
+## Alternative Implementations
 
-The end-user application may decide to take a dependency on alternate implementation. 
+The end-user application may decide to take a dependency on alternative implementation. 
 
-The SDK provides flexibility and extensibility that may be used by many implementations. Before developing an alternative implementation, please, review extensibility points provided by OpenTelemetry.
+SDK provides flexibility and extensibility that may be used by many implementations. Before developing an alternative implementation, please, review extensibility points provided by OpenTelemetry.
 
 An example use case for alternate implementations is automated testing. A mock implementation can be plugged in during automated tests. For example it can store all generated telemetry data in memory and provide a capability to inspect this stored data. This will allow the tests to verify that the telemetry is generated correctly. Language Library authors are encouraged to provide such mock implementation.
 
-Note that mocking is also possible by using the SDK and a Mock Exporter without needed to swap out the entire SDK. 
+Note that mocking is also possible by using SDK and a Mock `Exporter` without needed to swap out the entire SDK. 
 
 The mocking approach chosen will depend on the testing goals and at which point exactly it is desirable to intercept the telemetry data path during the test.
 
 ## Concurrency and Thread-Safety
 
-See [Concurrency and Thread-Safety](concurrency.md) specification for
-guidelines on what concurrency safeties should API implementations provide
-and how they should be documented.
+See [Concurrency and Thread-Safety](concurrency.md) specification for guidelines on what concurrency safeties should API implementations provide and how they should be documented.

specification/sdk-exporter.md

@@ -0,0 +1,90 @@
+# Exporter Interface
+
+`Exporter` defines the interface that protocol-specific exporters must implement so that they can be plugged into OpenTelemetry SDK and support sending of telemetry data.
+
+The goals of the interface are:
+
+- Minimize burden of implementation for protocol-dependent telemetry exporters. The protocol exporter is expected to be primarily a simple telemetry data encoder and transmitter.
+
+- Allow implementing helpers as composable components that use the same chainable `Exporter` interface. SDK authors are encouraged to implement common functionality such as queuing, batching, tagging, etc. as helpers. This functionality will be applicable regardless of what protocol exporter is used.
+
+## Interface Definition
+
+The exporter must support two functions: **Export** and **Shutdown**. In strongly typed languages typically there will be 2 separate `Exporter` interfaces, one that accepts spans (SpanExporter) and one that accepts metrics (MetricsExporter).
+
+### Export(batch)
+
+Exports a batch of telemetry data. Protocol exporters that will implement this function are typically expected to serialize and transmit the data to the destination.
+
+Export() will never be called concurrently for the same exporter instance. Export() can be called again only after the current call returns.
+
+Export() must not block indefinitely, there must be a reasonable upper limit after which the call must time out with an error result (typically FailedRetryable).
+
+**Parameters:**
+
+batch - a batch of telemetry data. The exact data type of the batch is language specific, typically it is a list of telemetry items, e.g. for spans in Java it will be typically `Collection<ExportableSpan>` 
+
+Note that the data type for a span for illustration purposes here is written as an imaginary type ExportableSpan (similarly for metrics it would be e.g. ExportableMetrics). The actual data type must be specified by language library authors, it should be able to represent the span data that can be read by the exporter.
+
+**Returns:** ExportResult:
+
+ExportResult is one of:
+
+- Success - batch is successfully exported. For protocol exporters this typically means that the data is sent over the wire and delivered to the destination server.
+
+- FailedNoneRetryable - exporting failed. The caller must not retry exporting the same batch. The batch must be dropped. This for example can happen when the batch contains bad data and cannot be serialized.
+
+- FailedRetryable - cannot export to the destination. The caller should record the error and may retry exporting the same batch after some time. This for example can happen when the destination is unavailable, there is a network error or endpoint does not exist. 
+
+### Shutdown()
+
+Shuts down the exporter. Called when SDK is shut down. This is an opportunity for exporter to do any cleanup required.
+
+`Shutdown` should be called only once for each `Exporter` instance. After the call to `Shutdown` subsequent calls to `Export` are not allowed and should return FailedNoneRetryable error.
+
+`Shutdown` should not block indefinitely (e.g. if it attempts to flush the data and the destination is unavailable). Language library authors can decide if they want to make the shutdown timeout to be configurable.
+
+## Further Language Specialization
+
+Based on the generic interface definition layed out above library authors must define the exact interface for the particular language. 
+
+Authors are encouraged to use efficient data structures on the interface boundary that are well suited for fast serialization to wire formats by protocol exporters and minimize the pressure on memory managers. The latter typically requires understanding of how to optimize the rapidly-generated, short-lived telemetry data structures to make life easier for the memory manager of the specific language. General recommendation is to minimize the number of allocations and use allocation arenas where possible, thus avoiding explosion of allocation/deallocation/collection operations in the presence of high rate of telemetry data generation.
+
+### Examples
+
+These are examples on what the `Exporter` interface can look like in specific languages. Examples are for illustration purposes only. Language library authors are free to deviate from these provided that their design remain true to the spirit of `Exporter` concept.
+
+#### Go SpanExporter Interface
+
+```go
+type SpanExporter interface {
+    Export(batch []ExportableSpan) ExportResult
+    Shutdown()
+}
+
+type ExportResult struct {
+    Code         ExportResultCode
+    WrappedError error
+}
+
+type ExportResultCode int
+
+const (
+    Success ExportResultCode = iota
+    FailedNoneRetryable
+    FailedRetryable
+)
+```
+
+#### Java SpanExporter Interface
+
+```java
+public interface SpanExporter {
+ public enum ResultCode {
+   Success, FailedNoneRetryable, FailedRetryable
+ }
+
+ ResultCode export(Collection<ExportableSpan> batch);
+ void shutdown();
+}
+```