SDK Trace: Adding span collection limits

[toumorokoshi]

To help safeguard against a common error, adding default limits
to the number of events per span in the SDK.

Fixes #182

Changes

add default limits to the number of events per span in the SDK

• update changelog
• update spec compliance matrix

[SergeyKanzhelev]

@tigrannajaryan @bogdandrutu @Oberon00 are you satisfied with the PR update? Can you pls re-review?

[bogdandrutu]

After reading this, I am not sure if you ask for a "bytes limit" or "number of attributes", "number of events", etc limits.

I would say that the first one may sound very interesting but it comes with some performance overhead and hard to implement it right. So I am more supportive of the "max number of attributes", etc.

Independent of what is suggested here, I think the fact that is not clear has to be fixed.

[toumorokoshi]

Independent of what is suggested here, I think the fact that is not clear has to be fixed.

Good point! I've fixed to element count.

I agree that bytes has some interesting ramifications and addressed the immediate concern of stressing memory, but maybe a follow-up in another PR.

[andrewhsu]

from the issue triage meeting today with TC, if this PR is desired before GA, need 2 approvals an no disapprovals to merge, else this will be closed by end of day today @open-telemetry/technical-committee @open-telemetry/specs-approvers @SergeyKanzhelev

[jmacd]

The reason I might object to this spec change is simply that it's a lot of work to require, but it's work that was inherited from the OpenCensus libraries in many cases, e.g., I believe these limits are already implemented for OTel-Go.

[andrewhsu]

from the issue triage mtg today with TC, marking as after-ga because corresponding issue is also after-ga

[tigrannajaryan]

@open-telemetry/technical-committee should we add such a sanity limit to prevent app crashes due to instrumentation problems for the GA version?

I do not quite like the hard stance taken by this proposal. I think the spec is overstepping into what arguably can be also seen as an implementation decision.

I also do not like that we prescribe only one way to limit resource exhaustion by telemetry data. Limiting the total number of elements is not the only possible way and arguably is not necessarily completely effective either because we do not simultaneously limit the size of the element.

Some SDK's may have alternate solutions to this problem, e.g. implement limits for the total byte size rather then for the size of an individual element. By prescribing the specific behavior in the spec we prevent innovation in this area.

I would be more comfortable with this proposal if it was a recommendation.

[toumorokoshi]

Thanks for the feedback! It's great to have more input.

I also do not like that we prescribe only one way to limit resource exhaustion by telemetry data.

Do you believe this will restrict people from adding other ways? I thought the spec was always a dictation of what is a requirement for an implementation, with anything not explicitly stated up to the liberty of the implementations.

Some SDK's may have alternate solutions to this problem, e.g. implement limits for the total byte size rather then for the size of an individual element. By prescribing the specific behavior in the spec we prevent innovation in this area.

I think this is the tradeoff to be made: the con is forcing choices that are not appropriate for the implementation. the pro is standard behavior that's enables knowledge that transfers across language boundaries. We dictate several implementation details that could be argued to prevent innovation, such as dictating that the behavior of the default span processor, and it's defaults.

If the consensus (and I'd like to hear a few more chime in too) is "we don't know if this is the right behavior to dictate across implementations", I think it's fine to leave this PR be (or close).

But I would argue that this is a practical implementation detail that would prevent a significant category of human errors. And the cost of differing implementations is end user confusion.

[SergeyKanzhelev]

Some SDK's may have alternate solutions to this problem, e.g. implement limits for the total byte size rather then for the size of an individual element. By prescribing the specific behavior in the spec we prevent innovation in this area.

the proposal in current form is very toothless. 1000 is non-limit in real life. Just a statement that SDKs needs to watch out for these collections.

If anybody disagree with this PR - please mark it with "changes requested". Otherwise I will merge mid day (pacific time) tomorrow.

[tigrannajaryan]

Thanks for the feedback! It's great to have more input.

I also do not like that we prescribe only one way to limit resource exhaustion by telemetry data.

Do you believe this will restrict people from adding other ways? I thought the spec was always a dictation of what is a requirement for an implementation, with anything not explicitly stated up to the liberty of the implementations.

@toumorokoshi Not restrict, but possibly discourage. There is limited amount of time SDK authors are willing to spend on this topic. As an SDK author if I spend effort to implement a spec requirement I am less likely to go ahead and implement another additional way to limit resource exhaustion. Developer's time is a finite resource too.

Some SDK's may have alternate solutions to this problem, e.g. implement limits for the total byte size rather then for the size of an individual element. By prescribing the specific behavior in the spec we prevent innovation in this area.

I think this is the tradeoff to be made: the con is forcing choices that are not appropriate for the implementation. the pro is standard behavior that's enables knowledge that transfers across language boundaries. We dictate several implementation details that could be argued to prevent innovation, such as dictating that the behavior of the default span processor, and it's defaults.

I agree that it is beneficial to have a good standard behavior. I am not convinced the proposed standard behavior is valuable enough to be enforced across all SDKs.

If the consensus (and I'd like to hear a few more chime in too) is "we don't know if this is the right behavior to dictate across implementations", I think it's fine to leave this PR be (or close).

But I would argue that this is a practical implementation detail that would prevent a significant category of human errors. And the cost of differing implementations is end user confusion.

I would also like to hear more opinions. I am willing to be convinced.

[tigrannajaryan]

For reasons I posted in my previous comments I am putting a soft block on this to prevent premature merging, but I am open to be convinced.

[iNikem]

I would like to point to yet another aspect of the current discussion. Even if this PR is not merged, it will be allowed for SDK to implement these or similar limits. Then it will be beneficial if different languages at least used the same configuration options for these limits. E.g. the same environment variable name. Is the specification open to augment sdk-environment-variables.md with configuration for such optional behaviour?

[tigrannajaryan]

Is the specification open to augment sdk-environment-variables.md with configuration for such optional behaviour?

Yes, I think so. I would prefer configurable behavior.

[SergeyKanzhelev]

Is the specification open to augment sdk-environment-variables.md with configuration for such optional behaviour?

Yes, I think so. I would prefer configurable behavior.

Configurability was removed from this PR as we cannot formalize use cases. For instance, if somebody set the limit to 2 attributes, should it be supported? Why? If somebody sets the limit to infinity, should it be supported? No exporter will probably need those so why to allow to set it?

Configurability was important in OpenCensus as default was quite low. Like 10 or 20 as far as I remember. In OpenTelemetry I believe the general agreement that we need to start with much higher default. Which defeats the purpose of having this limits configurable on SDK side.

I see this PR in current form as a warning that something need to be done to control for unbound collections. Alternative to this PR is doing nothing - SDKs needs to be safe in general. Switching to controllable behavior would need to start with the scenarios.

[tigrannajaryan]

If somebody sets the limit to infinity, should it be supported? No exporter will probably need those so why to allow to set it?

There is a use case where collection of all telemetry is important: audit. In extreme cases it may be preferable to crash the actual operation if the telemetry that describes the operation cannot be exported. I understand that we do not necessarily care about this particular use case.

I see this PR in current form as a warning that something need to be done to control for unbound collections.

Makes sense. If it's a warning perhaps we can it a recommendation rather than hard requirement?

[toumorokoshi]

I think there's a good comprise with the optional aspect of it. Aggregating the proposal and calling out how this is addressing concerns, how about:

• make it optional
  • addresses @SergeyKanzhelev's point about having something in the spec to bring awareness to this issue.
  • enables experimentation as per @tigrannajaryan's point.
• keep the configurability, as per @iNikem and @tigrannajaryan's ask.
  • add suggested environment variables in sdk-environment-variables.md

I'm still concerned about this not being required, but I'd much prefer some awareness of the issue in the spec vs none at all. And it can be a good building block if we want to expand that into something more standardized.

[iNikem]

SDKs needs to be safe in general

I am sorry, what does it mean? SDKs cannot be safe by magic, they have to build themselves in such a way. This PR is one way to help/advice SDKs how to do that. SDKs have to implement some safety measures, this PR tries to make them similar across different languages.

Do we want to have ever language different? One has low limit, one has no limit, everyone using different configuration variables?

[tigrannajaryan]

Thank you @toumorokoshi

[tigrannajaryan]

@SergeyKanzhelev @iNikem I re-requested your review since the PR is substantially different now.

[arminru]

LGTM once @iNikem's final suggestions are applied.

Thanks for carrying this through, @toumorokoshi!

[toumorokoshi]

@arminru added!

[toumorokoshi]

@SergeyKanzhelev could you do a final review, and merge if you feel it's appropriate? I believe I've addressed all outstanding comments.