[Metricbeat/Kibana/stats] Enforce `exclude_usage=true` #22732

afharo · 2020-11-24T11:32:59Z

What does this PR do?

Changes to the Metricbeat's kibana module: it enforces exclude_usage=true (only for the versions it's accepted).

The reason for this change is to disable the usage collection via the /api/stats endpoint, as it is no longer needed for their initial Telemetry purposes.

Why is it important?

We are working towards decoupling Monitoring and Telemetry in Kibana. At the moment, Monitoring holds in the .monitoring-kibana-* indices the Kibana's Usage details only to be able to report them when sending the telemetry reports. This data is already sent to the remote telemetry cluster via Kibana's local collectors, so there's no need for this duplication in the efforts. Thanks to these changes, Monitoring will only maintain in its indices the data they really need for its own features.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Not a Go developer myself. Did I mess anything up? 😇

How to test this PR locally

Setup Metricbeat with the kibana-xpack module enabled and notice it never asks for the /api/stats?extended=true&legacy=true&exclude_usage=false request (the last query parameter is always true).

Related issues

Use cases

Explained in the issue #22651

Screenshots

Logs

afharo · 2020-11-24T16:07:57Z

jenkins, test this please

elasticmachine · 2020-11-24T17:08:22Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: [Pull request #22732 updated]
Start Time: 2020-11-25T10:40:51.095+0000
Duration: 50 min 3 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2262
Skipped	506
Total	2768

Steps errors

Expand to view the steps failures

`Terraform Apply on x-pack/metricbeat/module/aws`

Took 0 min 14 sec . View more details on here

`Terraform Apply on x-pack/metricbeat/module/aws`

Took 0 min 15 sec . View more details on here

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	2262
Skipped	506
Total	2768

elasticmachine · 2020-11-24T17:46:46Z

Pinging @elastic/stack-monitoring (Stack monitoring)

elasticmachine · 2020-11-24T17:46:47Z

Pinging @elastic/integrations-services (Team:Services)

ycombinator · 2020-11-25T05:35:56Z

metricbeat/module/kibana/stats/stats.go

-			return err
-		}
+		m.statsHTTP.SetURI(origURI + "&exclude_usage=true")
+	}


With this change we no longer need the comment above if m.isUsageExcludable.

Well spotted! I updated it to explain what's actually happening now :)
Thank you!

ycombinator · 2020-11-25T05:38:21Z

metricbeat/module/kibana/stats/stats_test.go

-				usageLastCollectedOn: test.usageLastCollectedOn,
-				usageNextCollectOn:   test.usageNextCollectOn,
-			}
+func TestFetchNoExcludeUsage(t *testing.T) {


Thank you for adding this test. 👍

ycombinator

Just left a small comment about an outdated comment. Otherwise LGTM. Nice work!

…ana/stats/exclude_usage_removal

…true` (#22732) (#22755)

…-issues * upstream/master: (41 commits) Fix version parser regex for packaging (elastic#22581) Fix local_dynamic documentation and add providers inline doc. (elastic#22657) fix: use proper param name for e2e tests (elastic#22836) [Heartbeat] Fix exit on disabled monitor (elastic#22829) Update Golang to 1.14.12 (elastic#22790) docs: fix setup.template.overwrite typos (elastic#22804) Add docs section for ECS EC2 monitoring (elastic#22784) Fixing logic to keep list of unique cluster UUIDs (elastic#22808) Skip somewhat flaky UDP system test on Windows (elastic#22810) Fix polling node when it is not ready and monitor by hostname (elastic#22666) Skip Filebeat test_shutdown on windows 7 (elastic#22797) Make monitoring Namespace thread-safe (elastic#22640) Drop pkt_dstaddr and pkt_srcaddr when equals to "-" (elastic#22721) Add support for reading from UNIX datagram sockets (elastic#22699) Fix export dashboard command from Elastic Cloud (elastic#22746) Skip flaky winlogbeat test on Windows-7 (elastic#22754) Missing `>` (elastic#22763) (elastic#22766) Fix k8s watcher issue when node access to list nodes and ns (elastic#22714) [Metricbeat/Kibana/stats] Enforce `exclude_usage=true` (elastic#22732) Avoid sending non-numeric floats in cloud foundry integrations (elastic#22634) ...

…dows-7 * upstream/master: (41 commits) Fix version parser regex for packaging (elastic#22581) Fix local_dynamic documentation and add providers inline doc. (elastic#22657) fix: use proper param name for e2e tests (elastic#22836) [Heartbeat] Fix exit on disabled monitor (elastic#22829) Update Golang to 1.14.12 (elastic#22790) docs: fix setup.template.overwrite typos (elastic#22804) Add docs section for ECS EC2 monitoring (elastic#22784) Fixing logic to keep list of unique cluster UUIDs (elastic#22808) Skip somewhat flaky UDP system test on Windows (elastic#22810) Fix polling node when it is not ready and monitor by hostname (elastic#22666) Skip Filebeat test_shutdown on windows 7 (elastic#22797) Make monitoring Namespace thread-safe (elastic#22640) Drop pkt_dstaddr and pkt_srcaddr when equals to "-" (elastic#22721) Add support for reading from UNIX datagram sockets (elastic#22699) Fix export dashboard command from Elastic Cloud (elastic#22746) Skip flaky winlogbeat test on Windows-7 (elastic#22754) Missing `>` (elastic#22763) (elastic#22766) Fix k8s watcher issue when node access to list nodes and ns (elastic#22714) [Metricbeat/Kibana/stats] Enforce `exclude_usage=true` (elastic#22732) Avoid sending non-numeric floats in cloud foundry integrations (elastic#22634) ...

[Metricbeat/Kibana/stats] exclude_usage=true by default

06ff260

afharo added enhancement Metricbeat Metricbeat Feature:Stack Monitoring v8.0.0 Team:Services (Deprecated) Label for the former Integrations-Services team v7.11.0 labels Nov 24, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Nov 24, 2020

Add CHANGELOG

e49f7ec

afharo changed the title ~~[Metricbeat/Kibana/stats] exclude_usage=true by default~~ [Metricbeat/Kibana/stats] Enforce exclude_usage=true when accepted by the version Nov 24, 2020

afharo changed the title ~~[Metricbeat/Kibana/stats] Enforce exclude_usage=true when accepted by the version~~ [Metricbeat/Kibana/stats] Enforce exclude_usage=true Nov 24, 2020

Merge branch 'master' into metricbeat/kibana/stats/exclude_usage_removal

cc68bbb

afharo marked this pull request as ready for review November 24, 2020 17:46

afharo mentioned this pull request Nov 24, 2020

[Metricbeat/Kibana/stats] exclude_usage=true by default #22654

Closed

7 tasks

afharo requested a review from ycombinator November 24, 2020 17:48

ycombinator reviewed Nov 25, 2020

View reviewed changes

ycombinator approved these changes Nov 25, 2020

View reviewed changes

ycombinator added the needs_backport PR is waiting to be backported to other branches. label Nov 25, 2020

afharo added 2 commits November 25, 2020 10:20

Fix comment

6a17c2b

Merge branch 'master' of github.com:elastic/beats into metricbeat/kib…

40de116

…ana/stats/exclude_usage_removal

afharo merged commit 144e94b into elastic:master Nov 25, 2020

afharo deleted the metricbeat/kibana/stats/exclude_usage_removal branch November 25, 2020 12:15

afharo mentioned this pull request Nov 25, 2020

Cherry-pick to 7.x: [Metricbeat/Kibana/stats] Enforce exclude_usage=true (#22732) #22755

Merged

afharo added a commit to afharo/beats that referenced this pull request Nov 25, 2020

[Metricbeat/Kibana/stats] Enforce exclude_usage=true (elastic#22732)

15d3412

afharo added a commit that referenced this pull request Nov 26, 2020

Cherry-pick to 7.x: [Metricbeat/Kibana/stats] Enforce `exclude_usage=…

434cf8d

…true` (#22732) (#22755)

ycombinator mentioned this pull request Dec 14, 2020

Make kibana/stats metricset work for Stack Monitoring without xpack.enabled flag #21496

Merged

afharo mentioned this pull request Jan 17, 2022

[Telemetry] The caching mechanism also caches failed payloads elastic/kibana#123021

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metricbeat/Kibana/stats] Enforce `exclude_usage=true` #22732

[Metricbeat/Kibana/stats] Enforce `exclude_usage=true` #22732

afharo commented Nov 24, 2020 •

edited

Loading

afharo commented Nov 24, 2020

elasticmachine commented Nov 24, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

`Terraform Apply on x-pack/metricbeat/module/aws`

`Terraform Apply on x-pack/metricbeat/module/aws`

Test stats 🧪

elasticmachine commented Nov 24, 2020

elasticmachine commented Nov 24, 2020

ycombinator Nov 25, 2020

afharo Nov 25, 2020

ycombinator Nov 25, 2020

ycombinator left a comment

[Metricbeat/Kibana/stats] Enforce exclude_usage=true #22732

[Metricbeat/Kibana/stats] Enforce exclude_usage=true #22732

Conversation

afharo commented Nov 24, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

afharo commented Nov 24, 2020

elasticmachine commented Nov 24, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

Terraform Apply on x-pack/metricbeat/module/aws

Terraform Apply on x-pack/metricbeat/module/aws

💚 Flaky test report

Test stats 🧪

elasticmachine commented Nov 24, 2020

elasticmachine commented Nov 24, 2020

ycombinator Nov 25, 2020

Choose a reason for hiding this comment

afharo Nov 25, 2020

Choose a reason for hiding this comment

ycombinator Nov 25, 2020

Choose a reason for hiding this comment

ycombinator left a comment

Choose a reason for hiding this comment

[Metricbeat/Kibana/stats] Enforce `exclude_usage=true` #22732

[Metricbeat/Kibana/stats] Enforce `exclude_usage=true` #22732

afharo commented Nov 24, 2020 •

edited

Loading

elasticmachine commented Nov 24, 2020 •

edited by jenkins-beats-ci bot

Loading

`Terraform Apply on x-pack/metricbeat/module/aws`

`Terraform Apply on x-pack/metricbeat/module/aws`