Add OTLPHandler for standard library logging module #1903

srikanthccv · 2021-06-09T23:53:11Z

Description

Part-3 of #1890. Builds on top of #1901. This adds OTLPHandler for logging module. Python logging module provides a way to send the log records to the one or more appropriate destinations. More info about handler https://docs.python.org/3/library/logging.html#handler-objects

codeboten

Code looks good, just a typo and a question about naming. Ant thoughts on whether the OTLPHandler should be part of the SDK or a separate OTLP logging package?

opentelemetry-sdk/src/opentelemetry/sdk/logs/__init__.py

codeboten · 2021-06-16T15:35:45Z

opentelemetry-sdk/src/opentelemetry/sdk/logs/__init__.py

@@ -111,6 +113,48 @@ def force_flush(self, timeout_millis: int = 30000):
        """


+class OTLPHandler(logging.Handler):


Should this be called an emitter rather than a handler? Or is this a convenience handler to allow users to add this handler to their python root handler?

Yes, this is one of the the handlers users can attach to root logger. I believe the name Handler makes more sense here.

codeboten · 2021-06-16T15:44:42Z

opentelemetry-sdk/src/opentelemetry/sdk/logs/__init__.py

@@ -121,6 +165,10 @@ def __init__(
        self._resource = resource
        self._instrumentation_info = instrumentation_info

+    @property


Good catch!

Shouldn't this be on the provider level?

Yes, this should be on provider as well.

Co-authored-by: alrex <alrex.boten@gmail.com>

srikanthccv · 2021-06-16T16:06:27Z

thoughts on whether the OTLPHandler should be part of the SDK or a separate OTLP logging package?

My initial thoughts were since this is Handler for logging module maybe we can provide this as a part of SDK and other third party libs as separate packages. No strong opinions.

srikanthccv · 2021-06-16T18:35:17Z

One other argument to provide OTLPHandler as part of the SDK so that the users doesn’t need to find yet another separate package for something common.

opentelemetry-sdk/src/opentelemetry/sdk/util/severity.py

…lemetry-python into logging-otlp-handler

lzchen · 2021-06-16T22:04:45Z

opentelemetry-sdk/src/opentelemetry/sdk/logs/severity.py

+
+
+_STD_TO_OTLP = {
+    10: SeverityNumber.DEBUG,


How come the interim levels defined like this? (11-19, 21-29, etc.)

Curious will numbers in "std" ever be anything other than 0, 10, 20, 30, 40, 50?

This follows the log data model specification for mapping source format https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#mapping-of-severitynumber. The step value from one level to another level in logging module is 10 but the it is 5 in log data model, so it is clearly many-to-one mapping for some levels. Higher the value, higher the severity. In general these intermediary levels don't come into play but logging library allows programmers to define the custom log levels if they need, so this addresses that case. For example I would define some scary b/w error and critical level like below

>>> import logging >>> SCARY = 45 >>> logging.addLevelName(SCARY, "SCARY") >>> def scary(self, message, *args, **kws): ... self._log(SCARY, message, args, **kws) ... >>> logging.Logger.scary = scary >>> logger = logging.getLogger(__name__) >>> logger.scary("Boo") Boo

lzchen · 2021-06-16T22:17:03Z

opentelemetry-sdk/src/opentelemetry/sdk/logs/__init__.py

+    def _translate(self, record: logging.LogRecord) -> LogRecord:
+        timestamp = int(record.created * 1e9)
+        span_context = get_current_span().get_span_context()
+        # TODO: attributes (or resource attributes?) from record metadata


FYI, I believe attributes can be populated from both custom and resources.

So logging.LogRecord https://docs.python.org/3/library/logging.html#logrecord-attributes has a number of attributes like funcName, filename, and processName etc.. I was wondering if we should just make these as OTLP log record attributes https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-attributes or log record resource attributes.

For logging.LogRecord attributes, it makes more sense to be part of the OTLP log record attributes.

When I was saying "custom" attributes, I meant the ability for users to add custom attributes to the LogRecord themselves (it is a feature in OpenCensus as well), but may not be defined yet in the specs.

Not any concern for now, just something to be mindful of.

logging.LogRecord has an (optional) extra attribute that holds arbitrary user-injected attributes.
This is not super well documented, but the docs can be found a couple paragraphs down from here.

From a users perspective, it looks like this:

logging.debug("msg", extra={...})

Would it be possible to map this to OTLP log record attributes? You could take the top-level attributes of extra (which would be my preference), or require them to be nested under a specific key (e.g. logging.debug("msg", extra={"oltp-attributes": {...}})).

ocelotl · 2021-06-17T20:40:19Z

opentelemetry-sdk/src/opentelemetry/sdk/logs/__init__.py

+        self._log_emitter = log_emitter or get_log_emitter(__name__)
+
+    def _translate(self, record: logging.LogRecord) -> LogRecord:
+        timestamp = int(record.created * 1e9)


Just for your information, this PEP may be relevant here.

)

* Add initial overall structure and classes for logs sdk (#1894) * Add global LogEmitterProvider and convenience function get_log_emitter (#1901) * Add OTLPHandler for standard library logging module (#1903) * Add LogProcessors implementation (#1916) * Fix typos in test_handler.py (#1953) * Add support for OTLP Log exporter (#1943) * Add support for user defined attributes in OTLPHandler (#1952) * use timeout in force_flush (#2118) * use timeout in force_flush * fix lint * Update opentelemetry-sdk/src/opentelemetry/sdk/logs/export/__init__.py Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com> * fix lint Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com> * add a ConsoleExporter for logging (#2099) Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com> * Update SDK docs and Add example with OTEL collector logging (debug) exporter (#2050) * Fix exception in severity number transformation (#2208) * Fix exception with warning message transformation * Fix lint * Fix lint * fstring * Demonstrate how to set the Resource for LogEmitterProvider (#2209) * Demonstrate how to set the Resource for LogEmitterProvider Added a Resource to the logs example to make it more complete. Previously it was using the built-in Resource. Now it adds the service.name and service.instance.id attributes. The resulting emitted log records look like this: ``` Resource labels: -> telemetry.sdk.language: STRING(python) -> telemetry.sdk.name: STRING(opentelemetry) -> telemetry.sdk.version: STRING(1.5.0) -> service.name: STRING(shoppingcart) -> service.instance.id: STRING(instance-12) InstrumentationLibraryLogs #0 InstrumentationLibrary __main__ 0.1 LogRecord #0 Timestamp: 2021-10-14 18:33:43.425820928 +0000 UTC Severity: ERROR ShortName: Body: Hyderabad, we have a major problem. Trace ID: ce1577e4a703f42d569e72593ad71888 Span ID: f8908ac4258ceff6 Flags: 1 ``` * Fix linting * Use batch processor in example (#2225) * move logs to _logs (#2240) * move logs to _logs * fix lint * move log_exporter to _log_exporter as it's still experimental (#2252) Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com> Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Co-authored-by: Leighton Chen <lechen@microsoft.com> Co-authored-by: Tigran Najaryan <4194920+tigrannajaryan@users.noreply.github.com> Co-authored-by: Owais Lone <owais@users.noreply.github.com>

)

srikanthccv added 9 commits June 10, 2021 00:42

Add global LogEmitterProvider and convenience function get_log_emitter

43c79ae

Add license

b88bb2a

Add OTLPHandler for logging module

8a071fe

Update handler

a10ccb8

fix lint

c31d36a

Update handler tests

c1d3eb6

Fix tests

0f6f3b4

Fix merge conflicts

3756f2d

Add CHANGELOG entry

287719c

srikanthccv requested review from codeboten and ocelotl June 15, 2021 01:41

srikanthccv marked this pull request as ready for review June 15, 2021 01:42

srikanthccv requested a review from a team June 15, 2021 01:42

codeboten reviewed Jun 16, 2021

View reviewed changes

Update opentelemetry-sdk/src/opentelemetry/sdk/logs/__init__.py

0dc7318

Co-authored-by: alrex <alrex.boten@gmail.com>

lzchen reviewed Jun 16, 2021

View reviewed changes

opentelemetry-sdk/src/opentelemetry/sdk/util/severity.py Outdated Show resolved Hide resolved

lzchen reviewed Jun 16, 2021

View reviewed changes

opentelemetry-sdk/src/opentelemetry/sdk/util/severity.py Outdated Show resolved Hide resolved

lzchen reviewed Jun 16, 2021

View reviewed changes

opentelemetry-sdk/src/opentelemetry/sdk/util/severity.py Outdated Show resolved Hide resolved

srikanthccv added 3 commits June 17, 2021 02:11

Review changes

ef20aee

Merge branch 'logging-otlp-handler' of github.com:lonewolf3739/opente…

ff0f491

…lemetry-python into logging-otlp-handler

Add link to log levels

3806614

lzchen reviewed Jun 16, 2021

View reviewed changes

codeboten approved these changes Jun 17, 2021

View reviewed changes

ocelotl reviewed Jun 17, 2021

View reviewed changes

ocelotl approved these changes Jun 17, 2021

View reviewed changes

lzchen approved these changes Jun 18, 2021

View reviewed changes

lzchen merged commit cdea25a into open-telemetry:logs Jun 18, 2021

srikanthccv deleted the logging-otlp-handler branch September 24, 2021 08:39

lzchen pushed a commit to lzchen/opentelemetry-python that referenced this pull request Oct 15, 2021

Add OTLPHandler for standard library logging module (open-telemetry#1903

27782e8

)

codeboten pushed a commit that referenced this pull request Oct 29, 2021

Add OTLPHandler for standard library logging module (#1903)

387a9b6

codeboten pushed a commit to codeboten/opentelemetry-python that referenced this pull request Nov 3, 2021

Add OTLPHandler for standard library logging module (open-telemetry#1903

9445d0d

)

lzchen pushed a commit that referenced this pull request Nov 4, 2021

Add OTLPHandler for standard library logging module (#1903)

44ffee5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add OTLPHandler for standard library logging module #1903

Add OTLPHandler for standard library logging module #1903

srikanthccv commented Jun 9, 2021

codeboten left a comment

codeboten Jun 16, 2021

srikanthccv Jun 16, 2021 •

edited

Loading

codeboten Jun 16, 2021

lzchen Jun 16, 2021

srikanthccv Jun 16, 2021

srikanthccv commented Jun 16, 2021 •

edited

Loading

srikanthccv commented Jun 16, 2021

lzchen Jun 16, 2021

lzchen Jun 16, 2021 •

edited

Loading

srikanthccv Jun 16, 2021

lzchen Jun 16, 2021

srikanthccv Jun 16, 2021

lzchen Jun 18, 2021

adriangb Jul 2, 2021

ocelotl Jun 17, 2021

		@@ -111,6 +113,48 @@ def force_flush(self, timeout_millis: int = 30000):
		"""


		class OTLPHandler(logging.Handler):

Add OTLPHandler for standard library logging module #1903

Add OTLPHandler for standard library logging module #1903

Conversation

srikanthccv commented Jun 9, 2021

Description

codeboten left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srikanthccv Jun 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srikanthccv commented Jun 16, 2021 • edited Loading

srikanthccv commented Jun 16, 2021

Choose a reason for hiding this comment

lzchen Jun 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srikanthccv Jun 16, 2021 •

edited

Loading

srikanthccv commented Jun 16, 2021 •

edited

Loading

lzchen Jun 16, 2021 •

edited

Loading