feat: Add fingerprints array to span schema

[Mmarzex]

We need to be able to filter spans/traces by the errors/warnings that they generate. This PR Adds a repeated field on the Span schema that we can populate on ingest when we have the spans and events together.

[Danwakeem]

Looks good just had a question about the implementation details we had in mind 👍

[medikoo]

As I see in the comments, it’s only needed for ingest internals. Do we really have to add it to schema?
Other issue is that it adds a field for information that's already provided with other fields (fingerprints are already present in trace.events[]). So design-wise this addition doesn't look compelling

[Mmarzex]

@medikoo It is also used in the user facing portions of the UI. This is a field that is need by ingest and then by the user facing APIs for retrieving/filtering data.

I understand the concern but we made a decision early on to use the same fields across all three areas, so since this is needed in two of them we have to add it. It would require a change to the entire ingest pipeline to change it from sending just span typed data to sending a span & { fingerprints: [] } type, which I do not want to do. It is a slippery slope to add fields in one part that do not exist everywhere.

[Mmarzex]

@medikoo Also we don't currently expose a fingerprint property on the event schema itself, just the custom_fingerprint which is not expected to be set all the time. I think this property provides the best solution to minimize query overhead in the whole system.

[Danwakeem]

Given we need this when we pass the protobuf around at some point I am cool with adding this. We may wanna wait for @medikoo to reply before sending it though.

[medikoo]

I understand the concern but we made a decision early on to use the same fields across all three areas, so since this is needed in two of them we have to add it. It would require a change to the entire ingest pipeline to change it from sending just span typed data to sending a span & { fingerprints: [] } type, which I do not want to do. It is a slippery slope to add fields in one part that do not exist everywhere.

This signals to me that we're using same schema for different requests, where the same data is expected to be passed. in a different form.

This doesn't look as the right direction to me (ideally those requests should be defined separately). It seems we're introducing a technical debt by building a schema that is hard to reason about.

Is there really no better way to do that?

[Mmarzex]

This doesn't look as the right direction to me (ideally those requests should be defined separately). It seems we're introducing a technical debt by building a schema that is hard to reason about.

How could they be defined separately. If we pull say all events from Opensearch that have fingerprint abc123, that may return to us 1,000,000 traceIds, we can't actually then query for all traces like that. We need to be able to query the spans index in opensearch for fingerprints. This is what this achieves.

[medikoo]

How could they be defined separately. If we pull say all events from Opensearch that have fingerprint abc123, that may return to us 1,000,000 traceIds, we can't actually then query for all traces like that. We need to be able to query the spans index in opensearch for fingerprints. This is what this achieves.

@Mmarzex will moving events from TracePayload to Span improve that? Having that we would have fingerprints at span level, and across TracePayload no data will be duplicated