[Exporter/OpenSearch] OpenSearch exporter setup (#23819)

Adding initial set-up for OpenSearch exporter addition. Future PRs will include adding functionality to the exporter, code coverage and e2e tests. Broken up for easier review. [New component proposal.](#23611) Start of resolution of #7905. Will come in future PR to 80+% coverage. Break-up of #23045 --------- Signed-off-by: Max Ksyunz <max.ksyunz@improving.com>
open-telemetry · Jul 21, 2023 · 675710d · 675710d
1 parent 20b3071
commit 675710d
Show file tree

Hide file tree

Showing 15 changed files with 825 additions and 5 deletions.
diff --git a/.chloggen/opensearch-exporter.yaml b/.chloggen/opensearch-exporter.yaml
@@ -0,0 +1,9 @@
+change_type: new_component
+
+component: opensearchexporter
+
+note: exports OpenTelemetry signals to [OpenSearch](https://opensearch.org/).
+
+issues: [23611]
+
+subtext:
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -69,6 +69,7 @@ exporter/logzioexporter/                                 @open-telemetry/collect
 exporter/lokiexporter/                                   @open-telemetry/collector-contrib-approvers @gramidt @gouthamve @jpkrohling @kovrus @mar4uk
 exporter/mezmoexporter/                                  @open-telemetry/collector-contrib-approvers @dashpole @billmeyer @gjanco
 exporter/opencensusexporter/                             @open-telemetry/collector-contrib-approvers @open-telemetry/collector-approvers
+exporter/opensearchexporter/                             @open-telemetry/collector-contrib-approvers @Aneurysm9 @MitchellGale @MaxKsyunz @YANG-DB
 exporter/parquetexporter/                                @open-telemetry/collector-contrib-approvers @atoulme
 exporter/prometheusexporter/                             @open-telemetry/collector-contrib-approvers @Aneurysm9
 exporter/prometheusremotewriteexporter/                  @open-telemetry/collector-contrib-approvers @Aneurysm9 @rapphil

diff --git a/.github/ISSUE_TEMPLATE/bug_report.yaml b/.github/ISSUE_TEMPLATE/bug_report.yaml
@@ -63,6 +63,7 @@ body:
       - exporter/loki
       - exporter/mezmo
       - exporter/opencensus
+      - exporter/opensearch
       - exporter/parquet
       - exporter/prometheus
       - exporter/prometheusremotewrite

diff --git a/.github/ISSUE_TEMPLATE/feature_request.yaml b/.github/ISSUE_TEMPLATE/feature_request.yaml
@@ -57,6 +57,7 @@ body:
       - exporter/loki
       - exporter/mezmo
       - exporter/opencensus
+      - exporter/opensearch
       - exporter/parquet
       - exporter/prometheus
       - exporter/prometheusremotewrite

diff --git a/.github/ISSUE_TEMPLATE/other.yaml b/.github/ISSUE_TEMPLATE/other.yaml
@@ -57,6 +57,7 @@ body:
       - exporter/loki
       - exporter/mezmo
       - exporter/opencensus
+      - exporter/opensearch
       - exporter/parquet
       - exporter/prometheus
       - exporter/prometheusremotewrite

diff --git a/.github/dependabot.yml b/.github/dependabot.yml
@@ -237,6 +237,11 @@ updates:
     schedule:
       interval: "weekly"
       day: "wednesday"
+  - package-ecosystem: "gomod"
+    directory: "/exporter/opensearchexporter"
+    schedule:
+      interval: "weekly"
+      day: "wednesday"
   - package-ecosystem: "gomod"
     directory: "/exporter/parquetexporter"
     schedule:
@@ -1097,8 +1102,3 @@ updates:
     schedule:
       interval: "weekly"
       day: "wednesday"
-  - package-ecosystem: "gomod"
-    directory: "/receiver/sqlserverreceiver"
-    schedule:
-      interval: "weekly"
-      day: "wednesday"
diff --git a/exporter/opensearchexporter/Makefile b/exporter/opensearchexporter/Makefile
@@ -0,0 +1 @@
+include ../../Makefile.Common
diff --git a/exporter/opensearchexporter/README.md b/exporter/opensearchexporter/README.md
@@ -0,0 +1,51 @@
+# OpenSearch Exporter
+
+| Status                   |           |
+| ------------------------ |-----------|
+| Stability                | [devel]   |
+| Supported pipeline types | traces    |
+| Distributions            | [contrib] |
+
+OpenSearch exporter supports sending OpenTelemetry signals as documents to [OpenSearch](https://www.opensearch.org).
+
+The documents are sent using [observability catalog](https://github.com/opensearch-project/opensearch-catalog/tree/main/schema/observability) schema.
+
+## Configuration options
+### Indexing Options
+- `dataset` (default=`default`) a user-provided label to classify source of telemetry. It is used to construct the name of the destination index or data stream.
+- `namespace` (default=`namespace`) a user-provided label to group telemetry. It is used to construct the name of the destination index or data stream.
+
+### HTTP Connection Options
+OpenSearch export supports standard (HTTP client settings](https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/confighttp#client-configuration).
+- `endpoint` (required) `<url>:<port>` of OpenSearch node to send data to.
+
+### TLS settings
+Supports standard TLS settings as part of HTTP settings. See [TLS Configuration/Client Settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md#client-configuration).
+
+### Retry Options
+- `retry_on_failure`: See [retry_on_failure](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)
+
+## Example
+
+```yaml
+extensions:
+  basicauth/client:
+  client_auth:
+    username: username
+    password: password
+
+exporters:
+  opensearch/trace:
+    endpoint: https://opensearch.example.com:9200
+    auth:
+      authenticator: basicauth/client
+# ······
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      exporters: [opensearch/trace]
+      processors: [batch]
+```
+[devel]:https://github.com/open-telemetry/opentelemetry-collector#development
+[contrib]:https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
diff --git a/exporter/opensearchexporter/config.go b/exporter/opensearchexporter/config.go
@@ -0,0 +1,31 @@
+// Copyright The OpenTelemetry Authors
+// SPDX-License-Identifier: Apache-2.0
+
+package opensearchexporter // import "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/opensearchexporter"
+
+import (
+	"errors"
+
+	"go.opentelemetry.io/collector/config/confighttp"
+	"go.opentelemetry.io/collector/exporter/exporterhelper"
+)
+
+// Config defines configuration for OpenSearch exporter.
+type Config struct {
+	confighttp.HTTPClientSettings `mapstructure:",squash"`
+	exporterhelper.RetrySettings  `mapstructure:"retry_on_failure"`
+	Namespace                     string `mapstructure:"namespace"`
+	Dataset                       string `mapstructure:"dataset"`
+}
+
+var (
+	errConfigNoEndpoint = errors.New("endpoint must be specified")
+)
+
+// Validate validates the opensearch server configuration.
+func (cfg *Config) Validate() error {
+	if len(cfg.Endpoint) == 0 {
+		return errConfigNoEndpoint
+	}
+	return nil
+}
diff --git a/exporter/opensearchexporter/config_test.go b/exporter/opensearchexporter/config_test.go
@@ -0,0 +1,84 @@
+// Copyright The OpenTelemetry Authors
+// SPDX-License-Identifier: Apache-2.0
+
+package opensearchexporter
+
+import (
+	"path/filepath"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"go.opentelemetry.io/collector/component"
+	"go.opentelemetry.io/collector/config/configauth"
+	"go.opentelemetry.io/collector/config/confighttp"
+	"go.opentelemetry.io/collector/config/configopaque"
+	"go.opentelemetry.io/collector/confmap/confmaptest"
+	"go.opentelemetry.io/collector/exporter/exporterhelper"
+)
+
+func TestLoadConfig(t *testing.T) {
+	t.Parallel()
+
+	cm, err := confmaptest.LoadConf(filepath.Join("testdata", "config.yaml"))
+	require.NoError(t, err)
+
+	defaultCfg := newDefaultConfig()
+	defaultCfg.(*Config).Endpoint = "https://opensearch.example.com:9200"
+	maxIdleConns := 100
+	idleConnTimeout := 90 * time.Second
+
+	tests := []struct {
+		id                   component.ID
+		expected             component.Config
+		configValidateAssert assert.ErrorAssertionFunc
+	}{
+		{
+			id:                   component.NewIDWithName(typeStr, ""),
+			expected:             defaultCfg,
+			configValidateAssert: assert.NoError,
+		},
+		{
+			id: component.NewIDWithName(typeStr, "trace"),
+			expected: &Config{
+				Dataset:   "ngnix",
+				Namespace: "eu",
+				HTTPClientSettings: confighttp.HTTPClientSettings{
+					Endpoint: "https://opensearch.example.com:9200",
+					Timeout:  2 * time.Minute,
+					Headers: map[string]configopaque.String{
+						"myheader": "test",
+					},
+					MaxIdleConns:    &maxIdleConns,
+					IdleConnTimeout: &idleConnTimeout,
+					Auth:            &configauth.Authentication{AuthenticatorID: component.NewID("sample_basic_auth")},
+				},
+				RetrySettings: exporterhelper.RetrySettings{
+					Enabled:             true,
+					InitialInterval:     100 * time.Millisecond,
+					MaxInterval:         30 * time.Second,
+					MaxElapsedTime:      5 * time.Minute,
+					Multiplier:          1.5,
+					RandomizationFactor: 0.5,
+				},
+			},
+			configValidateAssert: assert.NoError,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.id.String(), func(t *testing.T) {
+			factory := NewFactory()
+			cfg := factory.CreateDefaultConfig()
+
+			sub, err := cm.Sub(tt.id.String())
+			require.NoError(t, err)
+			require.NoError(t, component.UnmarshalConfig(sub, cfg))
+
+			vv := component.ValidateConfig(cfg)
+			tt.configValidateAssert(t, vv)
+			assert.Equal(t, tt.expected, cfg)
+		})
+	}
+}
diff --git a/exporter/opensearchexporter/factory.go b/exporter/opensearchexporter/factory.go
@@ -0,0 +1,51 @@
+// Copyright The OpenTelemetry Authors
+// SPDX-License-Identifier: Apache-2.0
+
+package opensearchexporter // import "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/opensearchexporter"
+
+import (
+	"context"
+
+	"go.opentelemetry.io/collector/component"
+	"go.opentelemetry.io/collector/config/confighttp"
+	"go.opentelemetry.io/collector/exporter"
+	"go.opentelemetry.io/collector/exporter/exporterhelper"
+	"go.opentelemetry.io/collector/pdata/ptrace"
+)
+
+const (
+	// The value of "type" key in configuration.
+	typeStr = "opensearch"
+	// The stability level of the exporter.
+	stability = component.StabilityLevelDevelopment
+)
+
+// NewFactory creates a factory for OpenSearch exporter.
+func NewFactory() exporter.Factory {
+	return exporter.NewFactory(
+		typeStr,
+		newDefaultConfig,
+		exporter.WithTraces(createTracesExporter, stability),
+	)
+}
+
+func newDefaultConfig() component.Config {
+	return &Config{
+		HTTPClientSettings: confighttp.NewDefaultHTTPClientSettings(),
+		Namespace:          "namespace",
+		Dataset:            "default",
+		RetrySettings:      exporterhelper.NewDefaultRetrySettings(),
+	}
+}
+
+func createTracesExporter(ctx context.Context,
+	set exporter.CreateSettings,
+	cfg component.Config) (exporter.Traces, error) {
+
+	return exporterhelper.NewTracesExporter(ctx, set, cfg, func(ctx context.Context, ld ptrace.Traces) error {
+		return nil
+	},
+		exporterhelper.WithShutdown(func(ctx context.Context) error {
+			return nil
+		}))
+}
diff --git a/exporter/opensearchexporter/go.mod b/exporter/opensearchexporter/go.mod
@@ -0,0 +1,63 @@
+module github.com/open-telemetry/opentelemetry-collector-contrib/exporter/opensearchexporter
+
+go 1.19
+
+require (
+	github.com/stretchr/testify v1.8.4
+	go.opentelemetry.io/collector/component v0.81.0
+	go.opentelemetry.io/collector/config/configauth v0.81.0
+	go.opentelemetry.io/collector/config/confighttp v0.81.0
+	go.opentelemetry.io/collector/config/configopaque v0.81.0
+	go.opentelemetry.io/collector/confmap v0.81.0
+	go.opentelemetry.io/collector/exporter v0.81.0
+	go.opentelemetry.io/collector/pdata v1.0.0-rcv0013
+)
+
+require (
+	github.com/cenkalti/backoff/v4 v4.2.1 // indirect
+	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/felixge/httpsnoop v1.0.3 // indirect
+	github.com/fsnotify/fsnotify v1.6.0 // indirect
+	github.com/go-logr/logr v1.2.4 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
+	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/golang/protobuf v1.5.3 // indirect
+	github.com/golang/snappy v0.0.4 // indirect
+	github.com/json-iterator/go v1.1.12 // indirect
+	github.com/klauspost/compress v1.16.6 // indirect
+	github.com/knadh/koanf v1.5.0 // indirect
+	github.com/knadh/koanf/v2 v2.0.1 // indirect
+	github.com/mitchellh/copystructure v1.2.0 // indirect
+	github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4 // indirect
+	github.com/mitchellh/reflectwalk v1.0.2 // indirect
+	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
+	github.com/modern-go/reflect2 v1.0.2 // indirect
+	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/rs/cors v1.9.0 // indirect
+	go.opencensus.io v0.24.0 // indirect
+	go.opentelemetry.io/collector v0.81.0 // indirect
+	go.opentelemetry.io/collector/config/configcompression v0.81.0 // indirect
+	go.opentelemetry.io/collector/config/configtelemetry v0.81.0 // indirect
+	go.opentelemetry.io/collector/config/configtls v0.81.0 // indirect
+	go.opentelemetry.io/collector/config/internal v0.81.0 // indirect
+	go.opentelemetry.io/collector/consumer v0.81.0 // indirect
+	go.opentelemetry.io/collector/extension v0.81.0 // indirect
+	go.opentelemetry.io/collector/extension/auth v0.81.0 // indirect
+	go.opentelemetry.io/collector/featuregate v1.0.0-rcv0013 // indirect
+	go.opentelemetry.io/collector/processor v0.81.0 // indirect
+	go.opentelemetry.io/collector/receiver v0.81.0 // indirect
+	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.42.0 // indirect
+	go.opentelemetry.io/otel v1.16.0 // indirect
+	go.opentelemetry.io/otel/metric v1.16.0 // indirect
+	go.opentelemetry.io/otel/trace v1.16.0 // indirect
+	go.uber.org/atomic v1.10.0 // indirect
+	go.uber.org/multierr v1.11.0 // indirect
+	go.uber.org/zap v1.24.0 // indirect
+	golang.org/x/net v0.11.0 // indirect
+	golang.org/x/sys v0.9.0 // indirect
+	golang.org/x/text v0.10.0 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc // indirect
+	google.golang.org/grpc v1.56.1 // indirect
+	google.golang.org/protobuf v1.31.0 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
+)