New component: OpenSearch Exporter [WIP]

[MitchellGale]

The purpose and use-cases of the new component

OpenSearch is used for real-time application monitoring, log analytics, website search and more. The OpenSearch exporter would convert OpenTelemetry data directly into OpenSearch allowing logs, trace, metrics to be ingested and searched using OpenSearch. Exporting OpenTelemetry data into OpenSearch will help improve observability.

Many users are currently using OpenSearch as store repository for logs and need a common way and a standard protocol to consolidate both logs and trace signals.

Example configuration for the component

exporters:
  opensearch/trace:
    keyspace: "otel"
    endpoints: [https://localhost:9200]
    traces_index: trace_index
  opensearch/log:
    endpoints: [http://localhost:9200]
    logs_index: my_log_index
    sending_queue:
      enabled: true
      num_consumers: 20
      queue_size: 1000
······
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [opensearch/log]
    traces:
      receivers: [otlp]
      exporters: [opensearch/trace]
      processors: [batch]

Telemetry data types supported

• Traces
• Logs

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am proposing to contribute this as a representative of the vendor.

Sponsor (optional)

Anthony Mirabella (Aneurysm9)

Additional context

Will start contributing with traces initially then follow up with logs.

OpenSearch has recently forked and showed the usage of OpenSearch as a backend store and exploration tools on top of the open telemetry astronomer shop demo using simple schema as the mapping physical store of the OTEL signals

https://github.com/opensearch-project/opentelemetry-demo
https://github.com/opensearch-project/opensearch-catalog/tree/main/schema/observability
https://github.com/opensearch-project/opensearch-catalog/tree/main/docs/schema

I will be directly contributing to this. I am open to becoming a member of OpenTelemetry as progress on this exporter progresses.

[YANG-DB]

• lets remove the metrics and focus on logs / traces
• also add that many users are currently using OpenSearch as store repository for logs and need a common way (otel pipeline) and a standard protocol (simple schema of Observability which is compliant with OTEL protocol ) to consolidate both logs and trace signals
• opensearch allows federations of queries from multiple sources including Prometheus and thus allowing otel users to correlate metrics, signals and traces inside the same dashboard and use the same queries

OpenSearch has recently forked and showed the usage of OpenSearch as a backend store and exploration tools on top of the open telemetry astronomer shop demo using simple schema as the mapping physical store of the OTEL signals

additional context:

• https://github.com/opensearch-project/opentelemetry-demo
• https://github.com/opensearch-project/opensearch-catalog/tree/main/schema/observability
• https://github.com/opensearch-project/opensearch-catalog/tree/main/docs/schema