enhancement(observability): Events processed metrics for Topology

[leebenson]

Adds a EventsProcessed metrics to the Topology interface, and all implementing topology types.

Screenshot

Queries

Events processed metrics can be queried against the Topology interface, or individually on the filtered topology types, e.g:

{
  topology {
    __typename
    name
    eventsProcessed {
      eventsProcessed
    }
  }
}

Or:

{ 
  sources {
    eventsProcessed {
      timestamp
      eventsProcessed
    }
  }
}

Implementation

I've added a new GLOBAL_CONTROLLER, used to aggregate stats across the life of the API server. This can be used alongside the existing get_metrics() func, that samples over a provided interval.

[binarylogic]

@leebenson should we remove all of these as well?

https://github.com/timberio/vector/blob/9b1eada50e8679bfe4f4e97660ef8cf6e2090dae/src/internal_events/add_tags.rs#L9

I'm also curious about which other metrics we can move into the topology?

[binarylogic]

@leebenson you might want to review #4427, which I think addresses topology-level metrics.

cc @lukesteensen

[leebenson]

I'm also curious about which other metrics we can move into the topology?

As long as a metric's span is being associated with a component_name when it's emitted, then we can consume it from within GraphQL already. The resolvers just need implementing for the GraphQL schema to pull them out.

I'll be attempting this later today with 'errors' and other metrics types, so -- touch wood -- should have the necessary plumbing already.

Looks like #4427 might shift things around a bit, which I'll keep an eye on.

[Hoverbear]

Does this imply that all our metrics emit code will be interacting with blocking mutexes?

[leebenson]

No, this is actually old code. It's been removed in #4431. I initially thought it would be necessary to lock the controller for reads, since GraphQL resolvers are async and could be pulling from stats simultaneously. But the Controller is already capable of parallel reads.

[leebenson]

Closing this, as it's been merged into - and overtaken by - #4431

[lukesteensen]

As long as a metric's span is being associated with a component_name when it's emitted, then we can consume it from within GraphQL already.

This should still be exactly the same with #4427