OpenSearch exporter support

[thmshmm]

Is your feature request related to a problem? Please describe.
The existing elasticsearch exporter can not be used with OpenSearch as the used elasticsearch client library checks if the target backend is Elasticsearch. The support for OpenSearch should be introduced.

Describe the solution you'd like
The OpenSearch client could be integrated into the existing elasticsearch exporter and a configuration parameter could be introduced to switch clients.

Describe alternatives you've considered
An alternative would be to create a dedicated opensearch exporter and reimplement/reuse the elasticsearch exporter logic with just the client replaced.

Additional context
n/a

[r3h0]

In case anyone is wondering, here's the exact error message:

Bulk indexer error: flush: the client noticed that the server is not Elasticsearch and we do not support this unknown product

I know this because I set everything up only to find that I can't use it because of this.

I guess this is my +1 😭

Thanks for the hard work y'all are putting into OpenTelemetry. I've been following the project since the first announcement, and I'm looking forward to being a user of it someday.

[binjip978]

Hi, our storage solution is based on opendistro, so we have similar issue with elastic rest client. I already forked elasticsearchexporter https://github.com/binjip978/opentelemetry-collector-contrib/tree/opensearch-exporter/exporter/opensearchexporter and it works for our case.

But it still require proper documentation and some factoring on common code in both exporters. @jpkrohling not sure what "sponsor needed" is about, but I could provide a PR.

[jpkrohling]

From https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-new-components :

Before any code is written, open an issue providing the following information:

• Who's the sponsor for your component. A sponsor is an approver who will be in charge of being the official reviewer of the code and become a code owner for the component. For vendor-specific components, it's good to have a volunteer sponsor. If you can't find one, we'll assign one in a round-robin fashion. For non-vendor specific components, having a sponsor means that your use case has been validated.

[TomasLongo]

@r3h0 Nice! This is exactly what I was looking for. I tried to build your fork of the repo, but am not able to configure the opensearch-exporter properly. You mentioned that it works for you.

How did you configure the exporter? What key did you use in your otel-config.yaml?

Thanks in advance for your kind help. Cheers, Tomas.

[r3h0]

Not me, @TomasLongo. @binjip978's the one who built it

[TomasLongo]

Hey everyone. We successfully integrated the opensearchexporter into the otel-collector based on the fork of @binjip978.

We would be more than happy to contribute it back to the project if there is any interest.

[r3h0]

@TomasLongo there is interest, for sure. The exporter needs a sponsor, per what jpkrohling said above. Maybe you could read through the contributing docs and see if you are a good fit.

[KarstenSchnitter]

Hi everyone, there does not seem to be much progress on #10206. We really like the feature and tested the branch successfully. What can be done, to help the integration?

[StefanSa]

Hi there, @TomasLongo
thanks for this important exporter #10206.
I know that is a lot of wishes at once.
But since opensearch released the observability plugin, it would be great if the OpenSearch exporter would also support pipelines for metrics, logs and traces.

[TomasLongo]

Hey @StefanSa,

nice to see that there is interest in our contribution. We would be more than happy to provide the functionality you requested. Feel free to create an issue for the signals you want the exporter to support.

We can start working on them as soon, as #10206 got merged.

[StefanSa]

Hi @TomasLongo
done with #12323

[KarstenSchnitter]

@jpkrohling we would really like to integrate this exported into the codebase. #10206 was provided from a fork we used internally. Can you assists @TomasLongo in finding a sponsor for this issue? Maybe @dmitryax would be a possibility, since he already gave a lot of comments to #10206? If necessary, we can create another issue, as you suggested using https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-new-components. However, this will be very redundant to this very issue. Please advise, how you want us to proceed with the Opensearch exporter.

[jpkrohling]

@open-telemetry/collector-contrib-approvers: is anyone interested in sponsoring this? Perhaps someone from AWS (@Aneurysm9)?

[spakinz]

Currently, if we want to send our traces to OpenSearch and display them using the Trace Analytics plugin for OpenSearch Dashboard, we need to use OTel Collector + Data Prepper. I believe data is slightly transformed by Data Prepper in order to be displayed by the Trace Analytics plugin.

Should the new OpenSearch exporter have a configuration to allow the traces to be displayed by this plugin? I believe the ElasticSearch exporter has the mapping mode configuration option to send raw traces or to make slight transformations to the data. This would be something similar.

[dlvenable]

@spakinz , When sending trace data to OpenSearch using the OTel Collector and Data Prepper, Data Prepper does perform some transformations to make the traces display correctly in the Observability plugin. It performs two sets of changes: 1. It enriches each span with overall trace data; and 2) It creates a service map which goes into a second index.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[dbason]

#10206 looks to have stalled, but there are other people willing to work on it (we would still need a sponsor though). Would it be best to open a new PR?

[Aneurysm9]

OpenSearch accepts OTLP via DataPrepper and recommends that as an ingestion mechanism.

[dbason]

@Aneurysm9 that's true for traces and metrics but not for logs. I believe it would still be beneficial to have an Opensearch output as this then allows user to have a single agent if they are using Opensearch rather than making it an exception where you have to use DataPrepper.

[KannappanSomu]

+1

[steevex35]

Hi everyone, do we have any idea if the OpenSearch exporter will be officially added one day or not ?

[dbason]

Hi everyone, do we have any idea if the OpenSearch exporter will be officially added one day or not ?

I'm still trying to chase this up, we need to find someone to sponsor the work. I currently have a branch updated for v0.71.0. After doing that I've been wondering if it would be worthwhile having a shared objModel for elasticsearch and opensearch as these are currently identical.

[steevex35]

@dbason thanks a lot for your contribution Dan, hope someone will help you to sponsor your work.
Did you manage to build your custom collector (ocb v0.71.0) with the Opensearch exporter ?
I tried with this line but unable to fetch the module: - gomod: github.com/dbason/opentelemetry-collector-contrib/tree/opensearch-exporter/exporter/opensearchexporter v0.71.0

[dbason]

I did, you need to use a replaces section in the ocb config as follows:

exporters:
  - gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.71.0
  - gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.71.0
  - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.71.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/fileexporter v0.71.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/jaegerexporter v0.71.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/kafkaexporter v0.71.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/opencensusexporter v0.71.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter v0.71.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.71.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/zipkinexporter v0.71.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/opensearchexporter v0.71.0
replaces:
- github.com/open-telemetry/opentelemetry-collector-contrib/exporter/opensearchexporter => github.com/dbason/opentelemetry-collector-contrib/exporter/opensearchexporter v0.0.0-20230220215934-fc1091406dd0

[steevex35]

@dbason Hi Dan, Thank you very much for this, it's work perfectly with the last version of OpenSearch.

[arve0]

Hi @dbason. Thanks for your update on the OpenSearch exporter.

I'm trying it out with traces:

service:
  pipelines:
    traces:
      receivers:
      - otlp
      exporters:
      - opensearch

but opentelemetry-collector fails with:

Error: failed to build pipelines: failed to create "opensearch" exporter for data type "traces": telemetry type is not supported

I guess traces are not supported? Any idea how I should go about adding support for traces?

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.