Define guidelines for messaging about excessive amounts of recorded data

[jtmalinowski]

What are you trying to achieve?

Specify general guidelines for behaviour of SDK libraries in cases when there's excessive/abnormal amount of telemetry data being produced (e.g. a faulty loop produces too many / too long attributes).

2 decisions to be made:

• how to inform users that the data truncation was performed (or even if we should do that at all)
• where in the specification should we put these guidelines

Additional context.

• Existing example of limiting recorded data: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#span-limits
• New incoming PR which addresses attributes globally: Add an option to limit length of values of attributes and metric values opentelemetry-specification#1130

There seem to be a couple of conflicting opinions about how to inform users about data truncation, especially in comments on open-telemetry/opentelemetry-specification#1130.

[tigrannajaryan]

@jtmalinowski since you have a related PR open do you mind if I assign this issue to you?

[jtmalinowski]

@tigrannajaryan please do

[jtmalinowski]

I'm sure @jmacd and @jkwatson had comments on this, and I may be remembering it wrong but @MrAlias I think said something too.

Comments from RUM (browser, android, ios) perspective would certainly be very useful too: @obecny (unless you have no opinion or would like to delegate this to someone else), @ivomagi, @johnbley, @alolita.

If you have no opinion on this subject, please thumbs down this comment, so I know you saw it, thanks!

[obecny]

Ideally we could inform user:

1. First error occurence - inform imemdiately
2. 2nd and so on, collect number of errors, and inform user after 20s for example with showing the error and number of times it was catched
   For above we could use our own counter metric unless there is an easier way of doing that

[alevenberg]

I went to the messaging WG today to discuss what happens when we reach link limits.

There is a default link limit for spans (128), but the GCP Pub/Sub library can publish batches of up to 1000 messages. To work around this, we're using the environment variable to set the link limit. If the user has it set to the default or a lower limit, we capture all of the links by creating "publish #" spans which don't represent new work but are only created to hold all the links. See below the example screenshot

When we discussed today, @lmolkova raised two options for solving this problem:

1. Users can increase the # of links by specifying a larger limit.
2. Sample the links and should preferably stay

Everyone seems to prefer option 1.