Skip to content

Commit

Permalink
chore(observability): remove deprecated internal metrics + massive cl…
Browse files Browse the repository at this point in the history
…eanup to vector top and graphql API (#17516)

## Context

This PR is primarily, and simply, focused on one thing: removing now
deprecated internal metrics. Specifically:

- `events_in_total`
- `events_out_total`
- `processed_bytes_total`
- `processed_events_total`

These metrics have been deprecated for a while, replaced by the metrics
defined in the [Component Specification](). For example,
`component_sent_events_total` replaced `events_out_total`, and
`processed_bytes_total` is replaced by
`component_sent_bytes_total`/`component_received_bytes_total`: no more
having to remember what direction the metric refers to based on what
type of component is emitting it.

## Solution

The change to simply stop emitting these deprecated metrics is
straightforward, but most of the work in this PR centered around
updating the GraphQL API, and subsequently `vector top`, to switch from
the `processed_*` metrics to their `component_*` counterparts.

While in the area, I adjusted some of the `vector top` output to display
separated bytes in/bytes out, similar to the existing events in/events
out split. This also involved a small amount of work to adjust the
layout constraints, and so on, to accommodate for that.

Additionally, many updates were made to the component Cue files to
remove references to these now-removed metrics.

## Reviewer Notes

### Dichotomy between `processed_bytes_total` and the
`component`-prefixed replacements

Prior to this PR, we were backfilling `processed_bytes_total` in the
following way:

- for sources, alias `component_received_bytes_total` to
`processed_bytes_total`
- for sinks, alias `component_sent_bytes_total` to
`processed_bytes_total`

As such, we never emitted this metric for transforms, and
`processed_bytes_total` was an analogue for network bytes. A lot of the
work has tried to reflect that, such as in the GraphQL API, where spots
that had `processed_bytes` are now replaced with the component
type-relevant value, being either `received_bytes` or `sent_bytes`.

Closes #9314.
Closes #7346.
  • Loading branch information
tobz authored May 26, 2023
1 parent a551f33 commit 98c54ad
Show file tree
Hide file tree
Showing 121 changed files with 1,134 additions and 2,147 deletions.
22 changes: 5 additions & 17 deletions lib/vector-api-client/graphql/queries/components.graphql
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,8 @@ query ComponentsQuery($first: Int!) {
}
metrics {
__typename
processedEventsTotal {
processedEventsTotal
}
processedBytesTotal {
processedBytesTotal
receivedBytesTotal {
receivedBytesTotal
}
receivedEventsTotal {
receivedEventsTotal
Expand All @@ -37,12 +34,6 @@ query ComponentsQuery($first: Int!) {
}
metrics {
__typename
processedEventsTotal {
processedEventsTotal
}
processedBytesTotal {
processedBytesTotal
}
receivedEventsTotal {
receivedEventsTotal
}
Expand All @@ -54,15 +45,12 @@ query ComponentsQuery($first: Int!) {
... on Sink {
metrics {
__typename
processedEventsTotal {
processedEventsTotal
}
processedBytesTotal {
processedBytesTotal
}
receivedEventsTotal {
receivedEventsTotal
}
sentBytesTotal {
sentBytesTotal
}
sentEventsTotal {
sentEventsTotal
}
Expand Down
Loading

0 comments on commit 98c54ad

Please sign in to comment.