[Dynatrace] Logging verbosity: Move metadata discrepancy logging to WarnThenDebug

[pirgeo]

Many metrics have different descriptions based on their attributes (e.g., log4j2.events with tag loglevel=info has description Number of info level log events, while the same metric key with tag loglevel=trace has the description Number of trace level log events). Every time a discrepancy like that is detected, the Dynatrace exporter will log a warning. Consequently, many warnings will be logged, and continue to be logged during the runtime of the process. In order to reduce the number of warning logs produced, this PR introduces a WarnThenDebugLogger for this particular log.

While working on this, I realized that the other WarnThenDebugLoggers in the Dynatrace exporter are actually using the same instance (since they are initialized with the same class type). I fixed that by introducing thin wrappers for different WarnThenDebugLoggers. There is no change in application behaviour, just in the logging output.

[marcingrzejszczak]

Thank you for your PR. Can you please elaborate more on what is currently not working and how this fixes things? I don't fully understand things.

[pirgeo]

Absolutely! Micrometer allows multiple descriptions for one metric name. In Dynatrace, currently we only accept one description for all metrics of a metric key. Let me elaborate on my example above. Let's assume they were created like this (this is probably not true, but for the sake of the example it doesn't matter, I think):

Counter.Builder infoCounter = Counter.builder("log4j2.events")
    .tag("loglevel", "info")
    .description("Number of info level log event");
Counter.Builder traceCounter = Counter.builder("log4j2.events")
    .tag("loglevel", "trace")
    .description("Number of trace level log event");

They both have the same metric name (log4j2.events). When serializing, I would have to choose which of the descriptions I send to Dynatrace (since I can only set one of the two). Which meter is serialized first decides which description should be set - and it might be very different between exports and runs.

To circumvent this, we check for inconsistencies between descriptions before we export, and log a warning if they don't match. The problem I am trying to address with this PR is that we log a warning every time this happens, and on each export - that can be a lot of warning logs. Additionally, this is not a problem that will just right itself - it requires intervention by the application author, so continuously logging the warning does not help the person looking at the logs at all.

Hope that makes it a bit clearer - thanks for taking a look!

[shakuzen]

I don't think that's accurate. The two instances will use the same logger but the level is still controlled by the instance flag in each.

I wrote the following test:

@Test
void twoInstancesSameLoggerClass() {
    WarnThenDebugLogger first = new WarnThenDebugLogger(WarnThenDebugLoggerTest.class);
    WarnThenDebugLogger second = new WarnThenDebugLogger(WarnThenDebugLoggerTest.class);

    first.log("hi");
    second.log("hello");
}

And the log output is:

16:19:03.684 [main] WARN io.micrometer.common.util.internal.logging.WarnThenDebugLoggerTest - hi Note that subsequent logs will be logged at debug level.
16:19:03.684 [main] WARN io.micrometer.common.util.internal.logging.WarnThenDebugLoggerTest - hello Note that subsequent logs will be logged at debug level.

[pirgeo]

I think you are right, the description is off. I wanted to assert on the different WarnThenDebugLoggers, to ensure that all warning messages are triggered. Initially, I was using three separate WarnThenDebugLoggers to do that (without the subclasses). There is only one constructor for WarnThenDebugLogger and it takes a Class parameter:

micrometer/micrometer-core/src/main/java/io/micrometer/core/util/internal/logging/WarnThenDebugLogger.java

Lines 37 to 39 in 16dadc0

	public WarnThenDebugLogger(Class<?> clazz) {
	this.logger = InternalLoggerFactory.getInstance(clazz);
	}

Since there is no obvious choice of which Class type to put there, I at first put WarnThenDebugLogger.class.
In the tests, we are using the MockLoggerFactory to get these Loggers. There, I have two choices of how to retrieve the Loggers:

micrometer/micrometer-test/src/main/java/io/micrometer/core/util/internal/logging/MockLoggerFactory.java

Lines 46 to 48 in 16dadc0

	public MockLogger getLogger(Class<?> clazz) {
	return getLogger(clazz.getName());
	}

and

micrometer/micrometer-test/src/main/java/io/micrometer/core/util/internal/logging/MockLoggerFactory.java

Lines 57 to 59 in 16dadc0

	public MockLogger getLogger(String name) {
	return loggers.computeIfAbsent(name, MockLogger::new);
	}

Since there is no way of creating the WarnThenDebugLogger with a name instead of the Class object, all three used the WarnThenDebugLogger.class to create, and then the same WarnThenDebugLogger.class to retrieve them from the MockLoggerFactory. But: the MockLoggerFactory cannot distinguish between these instances, since they all have the same name. Consequently, the tests were very flaky, because it is not clear which WarnThenDebugLogger was being returned by mockLoggerFactory.getLogger(WarnThenDebugLogger.class).
In order to work around this, I created the new classes that basically set a specific "name" (by using the Class type of the subclass) to the separate WarnThenDebugLoggers. This way, there will be three separate WarnThenDebugLoggers in the MockLoggerFactory (each with a different name), and I can pull the right one to assert on.

Do you think this is the wrong approach? Alternatively, I could add API surface to the WarnThenDebugLogger to allow creating one with a specific name. Happy to discuss other alternatives as well!

[pirgeo]

Updated the description in f05701b

[shakuzen]

Thanks. Perhaps we can look at improving the usability of MockLoggerFactory or some other way to make testing this easier. /cc @jonatan-ivanov
The changes are mostly internal, though the class that will be logged is different with this change. I assume you're okay with that, and if so, I'm okay to merge the changes.

[pirgeo]

Yep, sounds good! Thank you!

[izeye]

Many metrics have different descriptions based on their attributes (e.g., log4j2.events with tag loglevel=info has description Number of info level log events, while the same metric key with tag loglevel=trace has the description Number of trace level log events).

This seems to be similar to #3201. I created #4864 to try to resolve this.

[izeye]

Since there is no way of creating the WarnThenDebugLogger with a name instead of the Class object,

I also created #4865 to try to resolve this.

[pirgeo]

Great news, thank you!