Warn about Prometheus meter registration failure

[izeye]

A meter registration failure seems to be an important event for users, but it's not easy for the users to be aware of the failure unless they register a callback for it.

It would be better to make the users easily aware of the failures by default.

See gh-5192

[shakuzen]

I don't think we should do this by default. #2068 has some history. I understand the usefulness of being able to identify these issues with a log message, but it's important to balance this with the principle that observability impact to your application should be minimal - I think this includes avoiding logging too much. It's better to lose some observability info than cause issues for the application. Logging this is far more reasonable if we're only talking about meter registration failures caused by instrumentation issues in code that users control in their application code, but it's a problem when we're talking about instrumentation issues in code that users don't control (third-party libraries). I believe that's essentially why we changed the default to doing nothing in Prometheus on meter registration failure due to the known limitation but we added a way (MeterRegistry#onMeterRegistrationFailed) to configure logging if that's what users want (possibly locally or in a test environment rather than production). That's still configurable for users without this change, but I can appreciate the point that it's hard for users to know they should configure this because it's hard to realize the problem without logging. Perhaps as a compromise, we could use a WarnThenDebugLogger. Though this will result in double logging if a user also configures logging via onMeterRegistrationFailed. @jonatan-ivanov thoughts? You mentioned a concern elsewhere about duplicate logging when a composite meter registry is used.

[izeye]

@shakuzen Thanks for the explanation!

Using a WarnThenDebugLogger seems to be a good compromise for both.

[jonatan-ivanov]

I like WarnThenDebugLogger, I would still keep the option open of making the feature opt-in (off by default). Similar to Prometheus' registry.throwExceptionOnRegistrationFailure() (maybe by adding a listener that does this instead of modifying the meterRegistrationFailed method).

Also, I think if you have a composite registry, and all the registries will report meter registration failures by default, you will get this message multiple times, e.g.: composite(prometheus, simple) will report this 3 times. With an extra config option you can control which registries are ok to report such an issue and if the feature is off by default it won't disrupt anything, then you can turn it on for the registry of your choice if you want.

What do you think?

[shakuzen]

I don't want to keep this PR in a state of limbo. I think given the above discussion, it maybe is best to not do something in MeterRegistry by default that can't be overridden due to duplicate logging with multiple registries. I know this could be a generic issue and the reasoning behind wanting to do this potentially applies to any registry - it's hard to know you have meter registration failures in the first place if nothing is logged. But since the original issue is about the known registration failure that happens with the Prometheus registry, maybe a compromise is to use a WarnThenDebugLogger in the Prometheus registry only for now. Thoughts?

[izeye]

@jonatan-ivanov @shakuzen Thanks for the feedback!

a compromise is to use a WarnThenDebugLogger in the Prometheus registry only for now.

I updated this PR in this direction.

[shakuzen]

Thanks for quickly updating the PR. I've polished things a bit in c799962