-
Notifications
You must be signed in to change notification settings - Fork 518
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add initial support for indexing to data streams (#4409)
* idxmgmt: add support for data streams Introduce `apm-server.data_streams.enabled` config, which will be used by idxmgmt to route events to data streams based on data_stream.* fields that are expected to be in each published event. * _meta/fields.common.yml: add data_stream.* fields * model: add data_stream.{type,dataset} fields When transforming model objects into beat.Events, set the data_stream.{type,dataset} fields. We add data_stream.namespace elsewhere, using an event processor. * beater: handle apm-server.data_streams.enabled Handle the new apm-server.data_streams.enabled config: - when enabled, we add data_stream.namespace to all published events - when disabled, we remove data_stream.* fields from all published events For now we just set data_stream.namespace to "default". Later this will be based on the config received from Fleet. * processor/stream/package_tests: update tests There is a hack in here to inject data_stream.namespace in all published events, since the tests do not use the standard libbeat pipeline code. * Update approvals * Add changelog entry * datastreams: address review comments * changelogs: clarify feature is experimental * _meta: specify allowed values for data_stream.type * model: update datasets Use a common "apm." prefix, and place the service name last.
- Loading branch information
Showing
69 changed files
with
694 additions
and
131 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
// Licensed to Elasticsearch B.V. under one or more contributor | ||
// license agreements. See the NOTICE file distributed with | ||
// this work for additional information regarding copyright | ||
// ownership. Elasticsearch B.V. licenses this file to you under | ||
// the Apache License, Version 2.0 (the "License"); you may | ||
// not use this file except in compliance with the License. | ||
// You may obtain a copy of the License at | ||
// | ||
// http://www.apache.org/licenses/LICENSE-2.0 | ||
// | ||
// Unless required by applicable law or agreed to in writing, | ||
// software distributed under the License is distributed on an | ||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
// KIND, either express or implied. See the License for the | ||
// specific language governing permissions and limitations | ||
// under the License. | ||
|
||
package config | ||
|
||
// DataStreamsConfig holds data streams configuration. | ||
type DataStreamsConfig struct { | ||
Enabled bool `config:"enabled"` | ||
} | ||
|
||
func defaultDataStreamsConfig() DataStreamsConfig { | ||
return DataStreamsConfig{Enabled: false} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
// Licensed to Elasticsearch B.V. under one or more contributor | ||
// license agreements. See the NOTICE file distributed with | ||
// this work for additional information regarding copyright | ||
// ownership. Elasticsearch B.V. licenses this file to you under | ||
// the Apache License, Version 2.0 (the "License"); you may | ||
// not use this file except in compliance with the License. | ||
// You may obtain a copy of the License at | ||
// | ||
// http://www.apache.org/licenses/LICENSE-2.0 | ||
// | ||
// Unless required by applicable law or agreed to in writing, | ||
// software distributed under the License is distributed on an | ||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
// KIND, either express or implied. See the License for the | ||
// specific language governing permissions and limitations | ||
// under the License. | ||
|
||
package datastreams | ||
|
||
// Constants for data stream types. | ||
const ( | ||
LogsType = "logs" | ||
MetricsType = "metrics" | ||
TracesType = "traces" | ||
) | ||
|
||
// Cosntants for data stream event metadata fields. | ||
const ( | ||
TypeField = "data_stream.type" | ||
DatasetField = "data_stream.dataset" | ||
NamespaceField = "data_stream.namespace" | ||
) | ||
|
||
// IndexFormat holds the variable "index" format to use for the libbeat Elasticsearch output. | ||
// Each event the server publishes is expected to contain data_stream.* fields, which will be | ||
// added to the documents as well as be used for routing documents to the correct data stream. | ||
const IndexFormat = "%{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
// Licensed to Elasticsearch B.V. under one or more contributor | ||
// license agreements. See the NOTICE file distributed with | ||
// this work for additional information regarding copyright | ||
// ownership. Elasticsearch B.V. licenses this file to you under | ||
// the Apache License, Version 2.0 (the "License"); you may | ||
// not use this file except in compliance with the License. | ||
// You may obtain a copy of the License at | ||
// | ||
// http://www.apache.org/licenses/LICENSE-2.0 | ||
// | ||
// Unless required by applicable law or agreed to in writing, | ||
// software distributed under the License is distributed on an | ||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
// KIND, either express or implied. See the License for the | ||
// specific language governing permissions and limitations | ||
// under the License. | ||
|
||
package datastreams | ||
|
||
import "strings" | ||
|
||
// NormalizeServiceName translates serviceName into a string suitable | ||
// for inclusion in a data stream name. | ||
// | ||
// Concretely, this function will lowercase the string and replace any | ||
// reserved characters with "_". | ||
func NormalizeServiceName(s string) string { | ||
s = strings.ToLower(s) | ||
s = strings.Map(replaceReservedRune, s) | ||
return s | ||
} | ||
|
||
func replaceReservedRune(r rune) rune { | ||
switch r { | ||
case '\\', '/', '*', '?', '"', '<', '>', '|', ' ', ',', '#', ':': | ||
// These characters are not permitted in data stream names | ||
// by Elasticsearch. | ||
return '_' | ||
case '-': | ||
// Hyphens are used to separate the data stream type, dataset, | ||
// and namespace. | ||
return '_' | ||
} | ||
return r | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
// Licensed to Elasticsearch B.V. under one or more contributor | ||
// license agreements. See the NOTICE file distributed with | ||
// this work for additional information regarding copyright | ||
// ownership. Elasticsearch B.V. licenses this file to you under | ||
// the Apache License, Version 2.0 (the "License"); you may | ||
// not use this file except in compliance with the License. | ||
// You may obtain a copy of the License at | ||
// | ||
// http://www.apache.org/licenses/LICENSE-2.0 | ||
// | ||
// Unless required by applicable law or agreed to in writing, | ||
// software distributed under the License is distributed on an | ||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
// KIND, either express or implied. See the License for the | ||
// specific language governing permissions and limitations | ||
// under the License. | ||
|
||
package datastreams_test | ||
|
||
import ( | ||
"testing" | ||
|
||
"github.com/stretchr/testify/assert" | ||
|
||
"github.com/elastic/apm-server/datastreams" | ||
) | ||
|
||
func TestNormalizeServiceName(t *testing.T) { | ||
testNormalizeServiceName := func(expected, input string) { | ||
t.Helper() | ||
assert.Equal(t, expected, datastreams.NormalizeServiceName(input)) | ||
} | ||
testNormalizeServiceName("upper_case", "UPPER-CASE") | ||
testNormalizeServiceName("____________", "\\/*?\"<>| ,#:") | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.