Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include full queue metrics in the monitoring index #42439

Merged
merged 5 commits into from
Jan 29, 2025

Conversation

faec
Copy link
Contributor

@faec faec commented Jan 27, 2025

Add queue/output metrics to the Metricbeat monitoring schema so they can be included in standard Agent dashboards. This is the Beats-side half of #42093. Affected metrics are: added.{events, bytes}, consumed.{events, bytes}, removed.{events, bytes}, and filled.{events, bytes, pct}, all within libbeat.pipeline.queue, and libbeat.output.write.latency.{count, max, median, p99}.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@faec faec added bug Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team backport-8.x Automated backport to the 8.x branch with mergify labels Jan 27, 2025
@faec faec self-assigned this Jan 27, 2025
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 27, 2025
@faec faec marked this pull request as ready for review January 28, 2025 21:20
@faec faec requested review from a team as code owners January 28, 2025 21:20
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@faec faec enabled auto-merge (squash) January 29, 2025 18:06
Copy link
Member

@pickypg pickypg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some questions before approving for Stack Monitoring.

@faec
Copy link
Contributor Author

faec commented Jan 29, 2025

Note that I just added a few more fields under output.latency. These are by request of @strawgate for elastic/integrations#12488 (which is essentially a super-issue of #42093) since he is updating the relevant dashboards.

Copy link
Member

@pickypg pickypg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@faec faec merged commit 7eb2bdd into elastic:main Jan 29, 2025
142 checks passed
mergify bot pushed a commit that referenced this pull request Jan 29, 2025
Add queue metrics to the Metricbeat monitoring schema so they can be included in standard Agent dashboards. This is the Beats-side half of #42093. Affected metrics are: `added.{events, bytes}`, `consumed.{events, bytes}`, `removed.{events, bytes}`, and `filled.{events, bytes, pct}`, all within `monitoring.metrics.libbeat.pipeline.queue`.

(cherry picked from commit 7eb2bdd)
@mauri870
Copy link
Member

mauri870 commented Jan 30, 2025

This change seems to have broken some of the metricbeat tests, because of the new metrics:

E           Exception: Keys:
E           
E           ['beat.stats.libbeat.pipeline.queue.added.events', 'beat.stats.libbeat.pipeline.queue.added.bytes', 
'beat.stats.libbeat.pipeline.queue.consumed.events', 'beat.stats.libbeat.pipeline.queue.consumed.bytes', 
'beat.stats.libbeat.pipeline.queue.removed.events', 'beat.stats.libbeat.pipeline.queue.removed.bytes', 
'beat.stats.libbeat.pipeline.queue.filled.events', 'beat.stats.libbeat.pipeline.queue.filled.bytes', 
'beat.stats.libbeat.pipeline.queue.filled.pct']
E           
E           not documented in event

https://buildkite.com/elastic/beats-metricbeat/builds/13523#0194b889-3630-4849-a023-e9e352e1afd6/192-456

@strawgate
Copy link
Contributor

@mauri870 i'll make sure the integration tests are passing with my follow-on pr #42514

@jlind23 jlind23 added the backport-8.18 Automated backport to the 8.18 branch label Jan 31, 2025
@jlind23
Copy link
Collaborator

jlind23 commented Jan 31, 2025

@faec adding backport 8.18 label as the branch has been created right before this merge

@jlind23 jlind23 added the backport-8.17 Automated backport with mergify label Jan 31, 2025
mergify bot pushed a commit that referenced this pull request Jan 31, 2025
Add queue metrics to the Metricbeat monitoring schema so they can be included in standard Agent dashboards. This is the Beats-side half of #42093. Affected metrics are: `added.{events, bytes}`, `consumed.{events, bytes}`, `removed.{events, bytes}`, and `filled.{events, bytes, pct}`, all within `monitoring.metrics.libbeat.pipeline.queue`.

(cherry picked from commit 7eb2bdd)
mergify bot pushed a commit that referenced this pull request Jan 31, 2025
Add queue metrics to the Metricbeat monitoring schema so they can be included in standard Agent dashboards. This is the Beats-side half of #42093. Affected metrics are: `added.{events, bytes}`, `consumed.{events, bytes}`, `removed.{events, bytes}`, and `filled.{events, bytes, pct}`, all within `monitoring.metrics.libbeat.pipeline.queue`.

(cherry picked from commit 7eb2bdd)
jlind23 pushed a commit that referenced this pull request Jan 31, 2025
Add queue metrics to the Metricbeat monitoring schema so they can be included in standard Agent dashboards. This is the Beats-side half of #42093. Affected metrics are: `added.{events, bytes}`, `consumed.{events, bytes}`, `removed.{events, bytes}`, and `filled.{events, bytes, pct}`, all within `monitoring.metrics.libbeat.pipeline.queue`.

(cherry picked from commit 7eb2bdd)

Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
jlind23 pushed a commit that referenced this pull request Jan 31, 2025
Add queue metrics to the Metricbeat monitoring schema so they can be included in standard Agent dashboards. This is the Beats-side half of #42093. Affected metrics are: `added.{events, bytes}`, `consumed.{events, bytes}`, `removed.{events, bytes}`, and `filled.{events, bytes, pct}`, all within `monitoring.metrics.libbeat.pipeline.queue`.

(cherry picked from commit 7eb2bdd)

Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
faec added a commit that referenced this pull request Jan 31, 2025
Add queue metrics to the Metricbeat monitoring schema so they can be included in standard Agent dashboards. This is the Beats-side half of #42093. Affected metrics are: `added.{events, bytes}`, `consumed.{events, bytes}`, `removed.{events, bytes}`, and `filled.{events, bytes, pct}`, all within `monitoring.metrics.libbeat.pipeline.queue`.

(cherry picked from commit 7eb2bdd)

Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8.x Automated backport to the 8.x branch with mergify backport-8.17 Automated backport with mergify backport-8.18 Automated backport to the 8.18 branch bug Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants