Make selector string casing configurable #18854

urso · 2020-05-29T14:43:15Z

Bug

What does this PR do?

Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'.

Why is it important?

Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (#16081. #6342).
As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (#18640).
With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

configure the Elasticsearch output while configuring an index name upper case characters => Check the index has only lower case characters in Elasticsearch
configure the Kafka output with topic name having upper case characters => Check the name is not modified in Kafka.

Related issues

Closes [Winlogbeat] Output/Kafka Can’t use topics with uppercase letters anymore #18640

elasticmachine · 2020-05-29T14:43:36Z

Pinging @elastic/integrations-services (Team:Services)

elasticmachine · 2020-05-29T14:54:09Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #18854 updated]
Start Time: 2020-06-09T11:41:39.973+0000
Duration: 77 min 11 sec

Test stats 🧪

Test	Results
Failed	0
Passed	9319
Skipped	1574
Total	10893

Steps errors

Expand to view the steps failures

Name: Report to Codecov
- Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done
- Duration: 1 min 27 sec
- Start Time: 2020-06-09T12:04:48.379+0000
- log
Name: Report to Codecov
- Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done
- Duration: 1 min 27 sec
- Start Time: 2020-06-09T12:13:24.858+0000
- log
Name: Make -C generator/_templates/metricbeat test
- Description: make -C generator/_templates/metricbeat test
- Duration: 7 min 36 sec
- Start Time: 2020-06-09T12:44:11.727+0000
- log

Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'. ## Why is it important? Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342). As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640). With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs. (cherry picked from commit 28f7aca)

…19118) Make selector string casing configurable (#18854) Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'. Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (#16081. #6342). As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (#18640). With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs. (cherry picked from commit 28f7aca)

…ngs-archive * upstream/master: (119 commits) Update filebeat input docs (elastic#19110) Add ECS fields from log pipeline of PostgreSQL (elastic#19127) Init package libbeat/statestore (elastic#19117) [Ingest Manager] Retryable downloads of beats (elastic#19102) [DOCS] Add output.console to Functionbeat doc and Functionbeat reference file (elastic#18965) Add compatibility info (elastic#18929) Set ecszap version to v0.2.0 (elastic#19106) [filebeat][httpjson] Fix unit test function call (elastic#19124) [Filebeat][httpjson] Adds oauth2 support for httpjson input (elastic#18892) Allow host.* fields to be disabled in Suricata module (elastic#19107) Make selector string casing configurable (elastic#18854) Switch the GRPC communication where Agent is running the server and the beats are connecting back to Agent (elastic#18973) Disable host.* fields by default for netflow module (elastic#19087) Automatically fill zube teams on backports if available (elastic#18924) Fix crash on vsphere module (elastic#19078) [Ingest Manager] Download snapshot artifacts from snapshots repo (elastic#18685) [Ingest Manager] Basic Elastic Agent documentation (elastic#19030) Make user.id a string in system/users, in line with ECS (elastic#19019) [docs] Add 7.8 release highlights placeholder file (elastic#18493) Fix translate_sid's empty target field handling (elastic#18991) ...

Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'. Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342). As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640). With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs. (cherry picked from commit 28f7aca)

…19175)

Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'. ## Why is it important? Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342). As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640). With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs. (cherry picked from commit 28f7aca)

…19409)

Add support for configuring the string casing in the index/pipeline/key/topic 'Selector'. ## Why is it important? Elasticsearch pipeline and index names are required to be lower case only. When used with fields from events this was not always guaranteed, leading us to enforce lower case always (elastic#16081. elastic#6342). As the code is reused for Kafka topic selection, this unfortunately did lead to a Regression as some users expect strings to allow mixed case (elastic#18640). With this PR Elasticsearch related resources (e.g. index or pipeline names) are set to lowercase only, while not touching the strings in other outputs.

…rable (elastic#19175)

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 29, 2020

urso added bug libbeat Team:Services (Deprecated) Label for the former Integrations-Services team labels May 29, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 29, 2020

urso added the review label May 29, 2020

urso marked this pull request as ready for review May 29, 2020 14:51

kvch approved these changes May 29, 2020

View reviewed changes

andresrc added [zube]: Inbox [zube]: In Review and removed [zube]: Inbox labels Jun 1, 2020

urso force-pushed the selector-optional-lowercase branch from 2e5f9c3 to 804b16f Compare June 2, 2020 15:17

urso added needs_backport PR is waiting to be backported to other branches. v7.8.1 v7.9.0 labels Jun 5, 2020

urso force-pushed the selector-optional-lowercase branch from 1c9914f to d06c711 Compare June 5, 2020 13:34

urso requested a review from kvch June 9, 2020 11:37

urso added 10 commits June 9, 2020 13:40

Make selector string casing configurable

5a6df1d

Add changelog

27b89f8

Add godoc to settings

df7ef4a

add missing license header

224d359

fix es test file

f3fc524

fix logstash test

6891938

Add index naming tests to indexmgmt and fix found bugs

8a3b66c

Add tests for pipeline selector in elasticsearch output

4691667

Add kafka topic selection tests

b93f27c

Add tests for redis key selection

665e207

urso force-pushed the selector-optional-lowercase branch from 83f27eb to 665e207 Compare June 9, 2020 11:40

Add redis output changelog

a767dbf

kvch approved these changes Jun 9, 2020

View reviewed changes

urso merged commit 28f7aca into elastic:master Jun 10, 2020

urso deleted the selector-optional-lowercase branch June 10, 2020 20:27

zube bot added [zube]: Done and removed [zube]: In Review labels Jun 10, 2020

urso mentioned this pull request Jun 10, 2020

Cherry-pick #18854 to 7.x: Make selector string casing configurable #19118

Merged

4 tasks

urso removed the needs_backport PR is waiting to be backported to other branches. label Jun 10, 2020

urso added the needs_backport PR is waiting to be backported to other branches. label Jun 10, 2020

urso added the v7.7.2 label Jun 12, 2020

urso mentioned this pull request Jun 12, 2020

Cherry-pick #18854 to 7.7: Make selector string casing configurable #19175

Merged

4 tasks

urso removed the needs_backport PR is waiting to be backported to other branches. label Jun 12, 2020

urso added the needs_backport PR is waiting to be backported to other branches. label Jun 12, 2020

urso pushed a commit that referenced this pull request Jun 15, 2020

Cherry-pick #18854 to 7.7: Make selector string casing configurable (#…

aa0d320

…19175)

urso mentioned this pull request Jun 18, 2020

[DOCS] Describe known problem with kafka output #19259

Merged

andresrc removed the [zube]: Done label Jun 22, 2020

urso mentioned this pull request Jun 25, 2020

Cherry-pick #18854 to 7.8: Make selector string casing configurable #19409

Merged

4 tasks

urso removed the needs_backport PR is waiting to be backported to other branches. label Jun 25, 2020

urso pushed a commit that referenced this pull request Jun 26, 2020

Cherry-pick #18854 to 7.8: Make selector string casing configurable (#…

54016b2

…19409)

This was referenced Oct 8, 2020

Ensure ES index names are lowercase elastic/apm-server#4294

Closed

fix: ensure ES index names are lowercase elastic/apm-server#4295

Merged

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Cherry-pick elastic#18854 to 7.7: Make selector string casing configu…

26161c8

…rable (elastic#19175)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make selector string casing configurable #18854

Make selector string casing configurable #18854

urso commented May 29, 2020 •

edited by andresrc

Loading

elasticmachine commented May 29, 2020

elasticmachine commented May 29, 2020 •

edited

Loading

Build stats

Test stats 🧪

Make selector string casing configurable #18854

Make selector string casing configurable #18854

Conversation

urso commented May 29, 2020 • edited by andresrc Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

elasticmachine commented May 29, 2020

elasticmachine commented May 29, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

urso commented May 29, 2020 •

edited by andresrc

Loading

elasticmachine commented May 29, 2020 •

edited

Loading