Align k8s metadata configurations in Kubernetes module: add addResourceMetadata config #29133

tetianakravchenko · 2021-11-24T16:47:32Z

Signed-off-by: Tetiana Kravchenko tetiana.kravchenko@elastic.co

What does this PR do?

add configuration option for add_resource_metadata in kubernetes module.

Why is it important?

align configuration: add_resource_metadata is already available in add_kubernetes_metadata processor and the kubernetes provider

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes Align k8s metadata configurations in Kubernetes module integrations#2131

Use cases

Screenshots

metricset.name: pod:

metricset.name: container

metricset.name: state_service

Logs

mergify · 2021-11-24T16:47:55Z

This pull request does not have a backport label. Could you fix it @tetianakravchenko? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2021-11-24T17:09:52Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-11-30T14:42:58.280+0000
Duration: 146 min 46 sec
Commit: 1ab0304

Test stats 🧪

Test	Results
Failed	0
Passed	48062
Skipped	4157
Total	52219

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

ChrsMark

I left some minors but overall is looking good already. We will also need to add the setting in documentation I guess.

metricbeat/module/kubernetes/util/kubernetes.go

libbeat/autodiscover/providers/kubernetes/pod.go

ChrsMark

Looks good, we only need to ensure that we stop the watchers properly.

metricbeat/module/kubernetes/util/kubernetes.go

elasticmachine · 2021-11-30T09:55:43Z

Pinging @elastic/integrations (Team:Integrations)

MichaelKatsoulis · 2021-11-30T10:07:20Z

@tetianakravchenko you should add a Changelog entry as well

tetianakravchenko · 2021-11-30T10:07:40Z

/test

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

…er and nsWatcher to enricher Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

…nfig that leads to 'failed to parse field' error Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

…eMetadataWatchers; return resource watcher if nodeWatcher/nsWatcher failed to be created Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

tetianakravchenko · 2021-12-01T09:35:59Z

@MichaelKatsoulis please have a look 1ab0304

MichaelKatsoulis

LGTM! You can merge it!

ChrsMark · 2021-12-08T07:21:28Z

/package

…ddResourceMetadata config (#29133)" This reverts commit 95bdebf.

MichaelKatsoulis · 2022-09-21T12:48:21Z

@Mergifyio backport 7.17

mergify · 2022-09-21T12:48:54Z

Sorry but I didn't understand the command. Please consult the commands documentation 📚.

MichaelKatsoulis · 2022-09-21T12:52:20Z

@Mergifyio backport 7.17

mergify · 2022-09-21T12:53:24Z

backport 7.17

✅ Backports have been created

#33148 Align k8s metadata configurations in Kubernetes module: add addResourceMetadata config (backport #29133) has been created for branch 7.17

…ceMetadata config (#29133) * use addResourceMetadata config instead of hardcoded bool Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * revert config for local dev Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * update doc; use common function to create all watchers; add nodeWatcher and nsWatcher to enricher Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * revert ek_stack Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * add stop watchers; adjust test for dedoting labels; fix overriding config that leads to 'failed to parse field' error Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * adjust log messages; check that watcher is not nil Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * fix error message format; rename getPodMetadataWatchers -> getResourceMetadataWatchers; return resource watcher if nodeWatcher/nsWatcher failed to be created Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * add changelog Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> (cherry picked from commit 95bdebf) # Conflicts: # libbeat/autodiscover/providers/kubernetes/pod.go # libbeat/autodiscover/providers/kubernetes/pod_test.go # libbeat/common/kubernetes/metadata/metadata.go # libbeat/common/kubernetes/metadata/pod.go # libbeat/common/kubernetes/metadata/pod_test.go # libbeat/processors/add_kubernetes_metadata/indexers_test.go # metricbeat/docs/modules/kubernetes.asciidoc # metricbeat/metricbeat.reference.yml # metricbeat/module/kubernetes/_meta/config.reference.yml # metricbeat/module/kubernetes/util/kubernetes.go # x-pack/elastic-agent/pkg/composable/providers/kubernetes/pod.go # x-pack/metricbeat/metricbeat.reference.yml

* Backport #29133 to 7.17

…ddResourceMetadata config (elastic#29133)" This reverts commit ff67957.

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 24, 2021

tetianakravchenko requested a review from ChrsMark November 24, 2021 16:47

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Nov 24, 2021

ChrsMark reviewed Nov 25, 2021

View reviewed changes

metricbeat/module/kubernetes/util/kubernetes.go Outdated Show resolved Hide resolved

metricbeat/module/kubernetes/util/kubernetes.go Outdated Show resolved Hide resolved

tetianakravchenko commented Nov 25, 2021

View reviewed changes

libbeat/autodiscover/providers/kubernetes/pod.go Show resolved Hide resolved

ChrsMark reviewed Nov 25, 2021

View reviewed changes

metricbeat/module/kubernetes/util/kubernetes.go Show resolved Hide resolved

metricbeat/module/kubernetes/util/kubernetes.go Outdated Show resolved Hide resolved

metricbeat/module/kubernetes/util/kubernetes.go Outdated Show resolved Hide resolved

MichaelKatsoulis reviewed Nov 29, 2021

View reviewed changes

metricbeat/module/kubernetes/util/kubernetes.go Outdated Show resolved Hide resolved

tetianakravchenko marked this pull request as ready for review November 29, 2021 15:06

MichaelKatsoulis reviewed Nov 29, 2021

View reviewed changes

metricbeat/module/kubernetes/util/kubernetes.go Outdated Show resolved Hide resolved

tetianakravchenko changed the title ~~use addResourceMetadata config instead of hardcoded bool~~ Align k8s metadata configurations in Kubernetes module: add addResourceMetadata config Nov 29, 2021

MichaelKatsoulis added the Team:Integrations Label for the Integrations team label Nov 30, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 30, 2021

tetianakravchenko mentioned this pull request Nov 30, 2021

Refactoring creation kubernetes resource watchers #29200

Open

tetianakravchenko added 8 commits November 30, 2021 15:41

use addResourceMetadata config instead of hardcoded bool

1b7912a

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

revert config for local dev

373529b

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

update doc; use common function to create all watchers; add nodeWatch…

ba9b087

…er and nsWatcher to enricher Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

revert ek_stack

de30bff

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

add stop watchers; adjust test for dedoting labels; fix overriding co…

56c6e94

…nfig that leads to 'failed to parse field' error Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

adjust log messages; check that watcher is not nil

fdfa8e4

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

fix error message format; rename getPodMetadataWatchers -> getResourc…

524d1a9

…eMetadataWatchers; return resource watcher if nodeWatcher/nsWatcher failed to be created Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

add changelog

1ab0304

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

tetianakravchenko force-pushed the k8s-metadata-config branch from 66f1d19 to 1ab0304 Compare November 30, 2021 14:42

MichaelKatsoulis approved these changes Dec 1, 2021

View reviewed changes

tetianakravchenko merged commit 95bdebf into elastic:master Dec 1, 2021

tetianakravchenko deleted the k8s-metadata-config branch December 1, 2021 12:08

This was referenced Dec 8, 2021

Test Agent on different ns #29338

Closed

Fix k8s metadata regression #29329

Merged

tetianakravchenko restored the k8s-metadata-config branch December 8, 2021 11:27

tetianakravchenko added a commit that referenced this pull request Dec 8, 2021

Revert "Align k8s metadata configurations in Kubernetes module: add a…

a514a39

…ddResourceMetadata config (#29133)" This reverts commit 95bdebf.

tetianakravchenko mentioned this pull request Dec 8, 2021

Revert "Align k8s metadata configurations in Kubernetes module: add addResourceMetadata config" #29339

Closed

tetianakravchenko mentioned this pull request Dec 16, 2021

Add missing fields to the affected kubernetes package data_streams elastic/integrations#2351

Closed

ChrsMark mentioned this pull request Dec 20, 2021

Expose add_recourse_metadata option to k8s package elastic/integrations#2362

Closed

This was referenced Sep 19, 2022

[Metricbeat][Kubernetes] Namespace labels missing on kube-state-metrics and container metrics #33108

Closed

Add resource metadata in all possible Kubernetes module metricsets #33144

Closed

mergify bot mentioned this pull request Sep 21, 2022

Align k8s metadata configurations in Kubernetes module: add addResourceMetadata config (backport #29133) #33148

Closed

MichaelKatsoulis added a commit to MichaelKatsoulis/beats that referenced this pull request Sep 22, 2022

Backport elastic#29133 to 7.17

4fa2000

MichaelKatsoulis mentioned this pull request Sep 22, 2022

Backport #29133 to 7.17 #33165

Merged

MichaelKatsoulis added a commit that referenced this pull request Sep 29, 2022

Backport #29133 to 7.17 (#33165)

891b217

* Backport #29133 to 7.17

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Revert "Align k8s metadata configurations in Kubernetes module: add a…

8c8b071

…ddResourceMetadata config (elastic#29133)" This reverts commit ff67957.

ChrsMark mentioned this pull request May 13, 2023

Safely start k8s watchers to avoid memory leak #35452

Merged

tetianakravchenko mentioned this pull request Nov 29, 2023

Include_labels option in add_resource_metadata not working as expected elastic/elastic-agent#3636

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Align k8s metadata configurations in Kubernetes module: add addResourceMetadata config #29133

Align k8s metadata configurations in Kubernetes module: add addResourceMetadata config #29133

tetianakravchenko commented Nov 24, 2021 •

edited by MichaelKatsoulis

Loading

mergify bot commented Nov 24, 2021

elasticmachine commented Nov 24, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

ChrsMark left a comment

ChrsMark left a comment

elasticmachine commented Nov 30, 2021

MichaelKatsoulis commented Nov 30, 2021

tetianakravchenko commented Nov 30, 2021

tetianakravchenko commented Dec 1, 2021

MichaelKatsoulis left a comment

ChrsMark commented Dec 8, 2021

MichaelKatsoulis commented Sep 21, 2022

mergify bot commented Sep 21, 2022

MichaelKatsoulis commented Sep 21, 2022

mergify bot commented Sep 21, 2022

Align k8s metadata configurations in Kubernetes module: add addResourceMetadata config #29133

Align k8s metadata configurations in Kubernetes module: add addResourceMetadata config #29133

Conversation

tetianakravchenko commented Nov 24, 2021 • edited by MichaelKatsoulis Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify bot commented Nov 24, 2021

elasticmachine commented Nov 24, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

ChrsMark left a comment

Choose a reason for hiding this comment

ChrsMark left a comment

Choose a reason for hiding this comment

elasticmachine commented Nov 30, 2021

MichaelKatsoulis commented Nov 30, 2021

tetianakravchenko commented Nov 30, 2021

tetianakravchenko commented Dec 1, 2021

MichaelKatsoulis left a comment

Choose a reason for hiding this comment

ChrsMark commented Dec 8, 2021

MichaelKatsoulis commented Sep 21, 2022

mergify bot commented Sep 21, 2022

MichaelKatsoulis commented Sep 21, 2022

mergify bot commented Sep 21, 2022

✅ Backports have been created

tetianakravchenko commented Nov 24, 2021 •

edited by MichaelKatsoulis

Loading

elasticmachine commented Nov 24, 2021 •

edited by jenkins-beats-ci bot

Loading