Measure and expose shipper queue performance metrics

[cmacknz]

Add metrics counters and gauges for the follow set of metrics, and ensure they are exposed on the shipper's HTTP monitoring interface under /debug/vars:

• Report the following set of queue metrics:
  • The current configured maximum size of the queue:
    • Report the maximum queue size in events for the memory queue:

      elastic-agent-shipper/monitoring/queuemon.go

      Line 131 in e231e51

      MaxLevel: opt.UintWith(limit),

    • Report the maximum queue size in bytes for the disk queue.
  • The current utilization of the queue in events:
    • Report the current utilization of the queue in events for the memory queue:

      elastic-agent-shipper/monitoring/queuemon.go

      Line 130 in e231e51

      CurrentLevel: opt.UintWith(count),

    • Report the current utilization of the queue in bytes for the disk queue.
  • Implement queue metrics for the disk queue #145

The eventual goal is to expose these metrics on the agent details page (example https://$kbana/app/fleet/agents/$agentID) in fleet via the agent monitoring metrics.

[ph]

@michel-laterman Would be good to have your inputs here too

[michel-laterman]

As far as diagnostics go, we would like to be able to access recent metrics/tracing data in order to include them with our bundles.

If metrics/tracing are handled through the pipeline env then we can use a queue or buffer of some sort to capture recent data (similar to the ring-buffer)

[fearful-symmetry]

Okay, so I'm just gonna use this to track the various intersecting issues needed for queue metrics:

• merge the actual PR that defines the metrics: Define a queue metrics reporter interface  beats#31289
• Implement the "basic" queue size and max size metrics in the queue
• Implement the "oldest event" metric in the queue(s)
• Implement queue lag, once this is dealt with: Make the memory queue work with types other than publisher.Event beats#31307 And also once we get input on precisely what it should measure according to @cmacknz Tentitively replaced by metrics outlined here: Report detailed queue metrics beats#31113 (comment)
• Implement the basic metrics reporter in the shipper, including the derived metrics based on what metrics are available from the queue
• Implement occupied_retry
• implement occupied_idle

[nimarezainia]

@cmacknz @fearful-symmetry sounds like the approach here is to add these stats to the diagnostics download from the agent. Please correct me if I am wrong. Would it be possible to have stats displayed on the agent detailed page also? (i don't think at this stage we need to have dashboards for the output observability of agents).

User Persona: Anyone debugging throughput issues of an agent. (support, platform operator etc)
Use Case: Support engineer trying to tune the throughput of the agent's output, they will modify some of the queue and batching parameters. Need to then observe changes in throughput on the agent.

Some additions to the above on what we should consider:

• Details of the memory queue actually allocated
• Utilization of the memory queue
• Average batch size sent from Output queue
• Transmission rates, any stats associated with the variables that make up the output engine.

Average bytes on each write
This may indicate if maximum_batch_size needs to be increased if the average is consistently near the max configured

[cmacknz]

Removed the age of the oldest item in the queue metrics as a requirement for now. Let's start simple and just report the current size of the queue periodically and see if we need more information. Once we have the queue size over time in ES+Kibana we should be able to compute queue lag from there for example.

[cmacknz]

I am moving the output metrics to a separate issue, let's just focus on the queue metrics with this issue.

[cmacknz]

#153

[leehinman]

Tried to think of things that would be useful for the disk queue, here is what I came up with.

• Size on Disk (queue already tracks this)
• Number of Events (need to add code to track)
• Rate of Events added to queue last 1, 5, 15 minutes
• Rate of Events removed from queue last 1, 5, 15 minutes
• Date of oldest segment (once a segment is full, we shouldn't write to it again, so this should show us how tail of queue is doing and if we have at least one stuck event)

[fearful-symmetry]

Looks like it got auto-closed. Since this is a meta-issue, reopening for now...

[nimarezainia]

• Date of oldest segment (once a segment is full, we shouldn't write to it again, so this should show us how tail of queue is doing and if we have at least one stuck event)

@leehinman what happens when an event is "stuck"? does it block?

[leehinman]

@leehinman what happens when an event is "stuck"? does it block?

This only happens if you have enabled infinite retries. But in that case an event that can't be delivered will be re-tried over and over again. So it will take up space in the queue, but other events will still be delivered. If you get enough events that can't be delivered then the queue will fill up. That would block reading of new events. This is expected behavior with infinite retries.

[nimarezainia]

@leehinman what happens when an event is "stuck"? does it block?

This only happens if you have enabled infinite retries. But in that case an event that can't be delivered will be re-tried over and over again. So it will take up space in the queue, but other events will still be delivered. If you get enough events that can't be delivered then the queue will fill up. That would block reading of new events. This is expected behavior with infinite retries.

Not sure if this is an issue we need to deal with right now but Could we have a retry queue for events that have reached some retry threshold. Yank them out of the main queue and use a separate thread to retry.

[leehinman]

Not sure if this is an issue we need to deal with right now but Could we have a retry queue for events that have reached some retry threshold. Yank them out of the main queue and use a separate thread to retry.

Lots of ways to address this. I'm thinking we wait a bit to see what happens with the queues from a performance perspective and then we can work on retries & failure states.

[cmacknz]

Closing, creating a new issue with the metrics that will be in the redesigned health dashboard.