[pkg/stanza] Reconsider aggresive sampling policy for logger

[BinaryFissionGames]

Component(s)

pkg/stanza

Describe the issue you're reporting

Stanza pipelines are built with a very aggressive sampling policy.

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/2d23e4d9e0eb313da5d192e1066f444d19b8601f/pkg/stanza/pipeline/config.go#L38C2-L38C2

If I'm reading and understanding this correctly, this means that every second, 1 initial log of a particular message is allowed, then only 1/10000 logs of that particular message is allowed.

This seems overly aggressive. In fact, it actually makes the file consumer only log the first log it finds on startup here:
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/2d23e4d9e0eb313da5d192e1066f444d19b8601f/pkg/stanza/fileconsumer/file.go#L188C35-L188C35

Meaning, if you have 10 files that are picked up on startup, only one of them is logged. That poses a challenge for debugging whether a file is picked up at all.

In my opinion, we should rely on the logger already being sampled. The collector already supports sampling. It also samples by default with initial = 100, and thereafter = 100.

[github-actions]

Pinging code owners:

• pkg/stanza: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

@BinaryFissionGames I think this is a good idea. The behavior is a holdover from when stanza was an independent agent. I see no reason we should be overriding global behaviors at this point.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• pkg/stanza: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• pkg/stanza: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[codeboten]

After #29886 was merged, I noted a change in the benchmarks. It looks like a 5-10% increase in CPU usage for at least some of the benchmark tests:

[djaglowski]

I spent quite some time looking into this today and found nothing conclusive. However, there was a preexisting problem, fixed by #30301, which would may have contributed to excess logging. I can't see the full history but it appears these two tests specifically show better performance than before this change. I'm closing this issue but we feel free to reopen if there is more to discuss.