[otelcol] rfc for how to log during startup

[TylerHelmuth]

Description

This is an RFC to help us decide how we want otelcol to provide a logger before the primary logger is created. As we discuss I will update the doc. Before this is merged we should have decided on a solution and the Accepted Solution section must be updated.

Related to #10056

Link to tracking issue

This unblocks:

• Provide warning when $ is used in config for env var expansion #9162
• Log a warning when an environment variable without a value is expanded #5615

[evan-bradley]

Thanks for writing this up.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.84%. Comparing base (d4d2d9a) to head (59d8173).
Report is 14 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10066      +/-   ##
==========================================
- Coverage   91.87%   91.84%   -0.03%     
==========================================
  Files         360      361       +1     
  Lines       16725    16722       -3     
==========================================
- Hits        15366    15359       -7     
- Misses       1021     1025       +4     
  Partials      338      338              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[codeboten]

I'm in favour of the first option, buffer the logs until we either get a configured logger, or emit them over stderr on the case of the configuration failing to resolve.

One question here is what happens in the case of the configuration taking a long time to resolve (if someone is using a remote, ie s3, provider)?

[evan-bradley]

One question here is what happens in the case of the configuration taking a long time to resolve (if someone is using a remote, ie s3, provider)?

That's a good question. I think unfortunately users will just see a delay before the logs are printed on Collector startup. I think this is an acceptable tradeoff considering the Collector is usually started without a human looking at the console.

[TylerHelmuth]

I've updated #10056 to implement solution 1 and all solution requirements as an example implementation

[mx-psi]

From #10008 (comment), can we list whether preserving the following is required?

1. The timestamp
2. The EntryCaller (what line number the log originates from)
3. The 'terminal behavior' (whether to panic at panic level)

I think there is consensus about preserving (1), I think (2) would be nice, and I personally don't care much about (3) (we don't really use that at all, and we tell people not to panic at runtime)

[TylerHelmuth]

@mx-psi preserving 1 is definitely required (and called out in the rfc). I think 2 should be required as well, if we all agree I can add that. 3 I don't think needs to be preserved as panicing goes against our development principals.

[mx-psi]

@mx-psi preserving 1 is definitely required (and called out in the rfc). I think 2 should be required as well, if we all agree I can add that. 3 I don't think needs to be preserved as panicing goes against our development principals.

Oki, that works for me. A solution with a changeable zapcore.Core should work here then, and we won't need to copy any internals if we don't require (3)

[TylerHelmuth]

@codeboten @mx-psi @djaglowski @evan-bradley I've updated the RFC with an accepted solution based on the PRs discussion.