Generalize source_lag_time_seconds to component_lag_time_seconds

[binarylogic]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

• Calculate upstream lag from any component in Vector
• Calculate internal latency by subtracting source and sink lag

Attempted Solutions

No response

Proposal

We should generalize source_lag_time_seconds to component_lag_time_seconds. This will allow us to observe upstream lag from any component in the topology. As a side benefits we can also derive Vector's internal latency. I realize this is an imperfect metric, but it's a simple first step that'll eliminate these blind spots until we can implement something more sophisticated.

References

No response

Version

No response

[bruceg]

I agree this is the least effort way of generating these values. I do have some concerns.

1. Can we run into a situation where different logs coming into a component (transform or sink) could have the timestamp in differently named fields due to different source schemas? Presumably we can account for this in the schema metadata.
2. Will this become tech debt if/when we move to a more reliable time source (ie an ingest timestamp not affected by either remote clock synchronization problems or network latencies)?
3. I think we will want to distinguish these latencies based on the source component, which would require recording a source component identifier in each event. That probably requires a bit of a discussion on how best to store that (dedicated field in Metadata vs something in the dynamic metadata).
4. Given the above, how are merged events handled, both in terms of the timestamps and combining source identifiers?

[binarylogic]

1. Yes, schemas is the solution for this. Until then we will ignore events that don't use the proper timestamp field name.
2. No, it'll always be an interesting data point. Granted, it will become more of a secondary signal but still useful.
3. Sure, this is beyond the scope of this issue though. I think it's fine to have a single data point that's based off of the event's timestamp as one perspective on upstream lag.
4. Because this is happening in the topology, merging shouldn't be a concern.

To reiterate, this is just one signal/layer on the overall picture of lag and latency. I'm not looking for a single source of truth. As we implement better metrics this will become less important, but it's still useful nonetheless.

[smitthakkar96]

@jszwedko, any idea when this will be prioritised?

[jszwedko]

@jszwedko, any idea when this will be prioritised?

Unfortunately it isn't on our roadmap at the moment. We'd be happy to help support a contribution for it though.