Make kibana/stats metricset work for Stack Monitoring without xpack.enabled flag #21496

sayden · 2020-10-02T18:52:31Z

Ready for testing. Check data.json and settings_data.json for a look of how the events look like now.

elasticmachine · 2020-10-02T18:52:33Z

Pinging @elastic/stack-monitoring (Stack monitoring)

elasticmachine · 2020-10-02T18:52:33Z

Pinging @elastic/integrations-services (Team:Services)

botelastic · 2020-10-02T18:52:40Z

This pull request doesn't have a Team:<team> label.

elasticmachine · 2020-10-02T19:12:44Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #21496 updated
Start Time: 2020-12-10T18:51:53.856+0000
Duration: 56 min 15 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2265
Skipped	520
Total	2785

Steps errors

Expand to view the steps failures

`Terraform Apply on x-pack/metricbeat/module/aws`

Took 0 min 15 sec . View more details on here

`Terraform Apply on x-pack/metricbeat/module/aws`

Took 0 min 15 sec . View more details on here

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	2265
Skipped	520
Total	2785

chrisronline · 2020-10-23T18:10:31Z

I'm seeing this error when running MB:

2020-10-23T14:10:10.317-0400 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: error loading template: could not load template. Elasticsearch returned: couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."}],"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."},"status":400}. Response body: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."}],"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."},"status":400}.

elasticmachine · 2020-11-06T18:01:52Z

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	4
Passed	2194
Skipped	469
Total	2667

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

    * **Name**: `Build&Test / metricbeat-goIntegTest / TestXPackEnabled – kibana`
    * **Name**: `Build&Test / metricbeat-goIntegTest / TestFetch – status`
    * **Name**: `Build&Test / x-pack/metricbeat-build / test_migration – x-pack.metricbeat.tests.system.test_xpack_base.Test`
    * **Name**: `Build&Test / x-pack/metricbeat-build / test_template – x-pack.metricbeat.tests.system.test_xpack_base.Test`

chrisronline · 2020-11-09T18:03:28Z

I'm still getting the same error:

2020-11-09T12:59:34.256-0500 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://localhost:9200)): Connection marked as failed because the onConnect callback failed: error loading template: could not load template. Elasticsearch returned: couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."}],"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."},"status":400}. Response body: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."}],"type":"mapper_parsing_exception","reason":"Invalid [path] value [kibana.stats.usage] for field alias [kibana.stats.usage.name]: an alias must refer to an existing field in the mappings."},"status":400

…/stats_xpack_flag

…/stats_xpack_flag # Conflicts: # metricbeat/module/kibana/stats/stats.go

ycombinator · 2020-12-14T07:26:54Z

metricbeat/module/kibana/stats/_meta/data.json

                }
            },
            "snapshot": false,
-            "status": "green"
+            "status": "green",
+            "usage": {


With the recent changes in #22732, I don't think we even need to collect usage any more. WDYT @chrisronline - safe to omit usage collection in this PR here?

Yes! Please remove it

ycombinator · 2020-12-14T07:40:31Z

metricbeat/module/kibana/stats/stats.go

-	if m.XPackEnabled {
-		m.fetchSettings(r, now)
+	if err = m.fetchSettings(r); err != nil {
+		return errors.Wrap(err, "error trying to get settings data from Kibana")


So previously (before this PR), this metricset would behave as follows:

if xpack.enabled: true was set, it would produce two events in each Fetch iteration, indexed into .monitoring-kibana-*, one with type: kibana_stats and one with type: kibana_settings.

if xpack.enabled: false was set, it would produce one event in each Fetch iteration, indexed into metricbeat-*, corresponding to type: kibana_stats` (but not actually including that field).

With the changes in this PR, it seems like we would always be producing two events per Fetch iteration: one corresponding to type: kibana_stats and one corresponding to type: kibana_settings. I wonder if now it makes sense to keep the kibana/stats metricset for just the former and create a new metricset, kibana/settings, for just the latter? It would be more in line with how other Metricbeat modules work. WDYT?

Feel free to perform this split of metricsets in a follow up PR to the feature branch. For now, if you want to merge the PR with the current implementation, that's fine.

I'll do it in a different PR because I have seen that it requires that I change quite a lot of thing in the base module. I mean, I have to change them anyways but I discovered it now.

metricbeat/module/kibana/stats/_meta/settings_data.json

ycombinator

I've left a couple of comments/questions on this PR. Besides those, this PR LGTM. So I'm approving it now.

Feel free to address the comments before you merge this PR OR merge it as-is now and address the comments in the feature branch later, as you prefer.

…nabled flag (elastic#21496)

Atomic commit

9d0f7a8

sayden added Metricbeat Metricbeat Feature:Stack Monitoring Team:Services (Deprecated) Label for the former Integrations-Services team labels Oct 2, 2020

sayden self-assigned this Oct 2, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 2, 2020

Run make update

ebbbf85

sayden changed the base branch from master to feature-stack-monitoring-mb-ecs October 2, 2020 18:54

Fix typo

1c36cf5

chrisronline mentioned this pull request Oct 23, 2020

Make logstash/node metricset work for Stack Monitoring without xpack.enabled flag #21545

Closed

sayden added 4 commits November 3, 2020 13:47

Fix fields

5af71ce

Fix linting

2a46d8d

Rootfield not being defined before setting a value

a279b4d

Regenerate data.json

0bc41ac

sayden added 8 commits December 3, 2020 11:48

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/kibana…

b8cb768

…/stats_xpack_flag

Add TestData and re-generate data.json

e91c748

Update fields.go

ca4a775

Fix linting

37492eb

Fix fields.yml

82efdce

Remove duplicated field

08247b6

Remove test that isn't used anymore

c99c5bd

Fix broken test

7451eb7

sayden added 4 commits December 3, 2020 21:10

Remove new testdata function

fa5dd30

Merge branch 'feature-stack-monitoring-mb-ecs' into feature/mb/kibana…

6ec26a8

…/stats_xpack_flag # Conflicts: # metricbeat/module/kibana/stats/stats.go

Fix broken test

d28540f

Remove TODO from field description

5ebf645

sayden requested a review from ycombinator December 10, 2020 18:50

ycombinator reviewed Dec 14, 2020

View reviewed changes

metricbeat/module/kibana/stats/_meta/settings_data.json Show resolved Hide resolved

ycombinator approved these changes Dec 15, 2020

View reviewed changes

sayden merged commit 283ef2a into elastic:feature-stack-monitoring-mb-ecs Dec 16, 2020

sayden mentioned this pull request Apr 27, 2021

[Metricbeat] Remove xpack enabled flag on ES, Logstash, Beats and Kibana #24427

Merged

6 tasks

sayden deleted the feature/mb/kibana/stats_xpack_flag branch August 25, 2022 12:54

leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023

Make kibana/stats metricset work for Stack Monitoring without xpack.e…

edf8b85

…nabled flag (elastic#21496)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make kibana/stats metricset work for Stack Monitoring without xpack.enabled flag #21496

Make kibana/stats metricset work for Stack Monitoring without xpack.enabled flag #21496

sayden commented Oct 2, 2020

elasticmachine commented Oct 2, 2020

elasticmachine commented Oct 2, 2020

botelastic bot commented Oct 2, 2020

elasticmachine commented Oct 2, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

`Terraform Apply on x-pack/metricbeat/module/aws`

`Terraform Apply on x-pack/metricbeat/module/aws`

Test stats 🧪

chrisronline commented Oct 23, 2020 •

edited

Loading

elasticmachine commented Nov 6, 2020

Test stats 🧪

Genuine test errors

chrisronline commented Nov 9, 2020

ycombinator Dec 14, 2020

chrisronline Dec 15, 2020

ycombinator Dec 14, 2020

ycombinator Dec 15, 2020

sayden Dec 16, 2020

ycombinator left a comment

Make kibana/stats metricset work for Stack Monitoring without xpack.enabled flag #21496

Make kibana/stats metricset work for Stack Monitoring without xpack.enabled flag #21496

Conversation

sayden commented Oct 2, 2020

elasticmachine commented Oct 2, 2020

elasticmachine commented Oct 2, 2020

botelastic bot commented Oct 2, 2020

elasticmachine commented Oct 2, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

Terraform Apply on x-pack/metricbeat/module/aws

Terraform Apply on x-pack/metricbeat/module/aws

💚 Flaky test report

Test stats 🧪

chrisronline commented Oct 23, 2020 • edited Loading

elasticmachine commented Nov 6, 2020

🐛 Flaky test report

Test stats 🧪

Genuine test errors

chrisronline commented Nov 9, 2020

ycombinator Dec 14, 2020

Choose a reason for hiding this comment

chrisronline Dec 15, 2020

Choose a reason for hiding this comment

ycombinator Dec 14, 2020

Choose a reason for hiding this comment

ycombinator Dec 15, 2020

Choose a reason for hiding this comment

sayden Dec 16, 2020

Choose a reason for hiding this comment

ycombinator left a comment

Choose a reason for hiding this comment

elasticmachine commented Oct 2, 2020 •

edited by jenkins-beats-ci bot

Loading

`Terraform Apply on x-pack/metricbeat/module/aws`

`Terraform Apply on x-pack/metricbeat/module/aws`

chrisronline commented Oct 23, 2020 •

edited

Loading