Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add additional index_types #3148

Closed
mishavay-aws opened this issue Aug 12, 2023 · 5 comments · Fixed by #3929
Closed

Add additional index_types #3148

mishavay-aws opened this issue Aug 12, 2023 · 5 comments · Fixed by #3929
Assignees
Labels
backlog enhancement New feature or request
Milestone

Comments

@mishavay-aws
Copy link
Contributor

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. It would be nice to have [...]

As an administrator configuring collectors that write to OpenSearch you want to configure index management policies scoped to particular index-name pattern.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Define index_type for "otel_logs_source" and "otel_metrics_source" (and others) similar to ones defined for "trace-analytics-raw" and "trace-analytics-service-map"

Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.

Without index_type defined, an index needs to be explicitly named, where it complicates creation of a scoped down policy in OpenSearch.

Additional context
Add any other context or screenshots about the feature request here.

@dlvenable
Copy link
Member

This is a good idea. We'll want to be sure to align the approach with the OpenSearch simple schema that the observability plugin is supporting.

@juergen-walter
Copy link
Contributor

@dlvenable is there a time horizon you want to tackle this feature? Or any blockers?

I could not find it on the timeline https://github.com/opensearch-project/data-prepper/projects/1

@juergen-walter
Copy link
Contributor

Solution should incorporate the changes from #3342

juergen-walter added a commit to juergen-walter/data-prepper that referenced this issue Jan 9, 2024
Signed-off-by: Jürgen Walter <juergen.walter@sap.com>
juergen-walter added a commit to juergen-walter/data-prepper that referenced this issue Jan 9, 2024
Signed-off-by: Jürgen Walter <juergen.walter@sap.com>
juergen-walter added a commit to juergen-walter/data-prepper that referenced this issue Jan 9, 2024
Signed-off-by: Jürgen Walter <juergen.walter@sap.com>
juergen-walter added a commit to juergen-walter/data-prepper that referenced this issue Jan 9, 2024
Signed-off-by: Jürgen Walter <juergen.walter@sap.com>
juergen-walter added a commit to juergen-walter/data-prepper that referenced this issue Jan 12, 2024
Signed-off-by: Jürgen Walter <juergen.walter@sap.com>
juergen-walter added a commit to juergen-walter/data-prepper that referenced this issue Jan 12, 2024
Signed-off-by: Jürgen Walter <juergen.walter@sap.com>
juergen-walter added a commit to juergen-walter/data-prepper that referenced this issue Jan 12, 2024
Signed-off-by: Jürgen Walter <juergen.walter@sap.com>
@juergen-walter
Copy link
Contributor

There is a PR #3929 for OpenTelemtry logs and metrics, reviews appreciated

@dlvenable dlvenable added this to the v2.7 milestone Mar 21, 2024
@dlvenable dlvenable modified the milestones: v2.7, v2.8 Mar 27, 2024
@dlvenable dlvenable modified the milestones: v2.8, v2.9 May 14, 2024
@dlvenable dlvenable moved this from Unplanned to In review in Data Prepper Tracking Board Jun 25, 2024
@dlvenable dlvenable modified the milestones: v2.9, v2.10 Aug 26, 2024
@dlvenable dlvenable modified the milestones: v2.10, v2.11 Oct 11, 2024
KarstenSchnitter pushed a commit that referenced this issue Oct 30, 2024
* Add ISM policies for logs and metrics

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Add index templates for logs

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Add index templates for metrics

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Add index_types for OTEL logs and metrics

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Test OpenSearch sink

works with log-analytics and metric-analytics
and index type

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Test log and metric index types

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Document log and metrics index_type usage

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Minor: Remove incorrect html tag

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Fix test by adding date_detection false

Fixes testInstantiateSinkMetricsDefaultMetricSink

Alertnative would have been to adjust the test

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Rename test

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Fix metric ism file constants

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Rename index template file for metrics

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Add assertions to tests

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Fix index patterns for logs and metrics

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Update to plugins ISM API

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Add assertion

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Revert "Update to plugins ISM API"

This reverts commit 3fd61af.

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Add fields data prepper writes

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

* Use field type data prepper writes

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

---------

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>
@linghengqian
Copy link
Contributor

This looks like a great PR. Does the OpenSearch Dashboard side have any pre-built visualization panels that correspond to this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backlog enhancement New feature or request
Projects
Development

Successfully merging a pull request may close this issue.

4 participants