Add support for OpenTelemetry logging

[codeboten]

Currently there's an experimental specification for the logging signal:

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#log-and-event-record-definition.

The work for this has started in #1882

[srikanthccv]

Here is how I plan to send PRs

1. Add overall structures and classes (Add initial overall structure and classes for logs sdk #1894).
2. Add global provider (Add global log emitter provider #1895)[closed due base branch delete, will reopen].
3. Add OTLPHandler for logging module.
4. Add multilog processors and Simple, BatchLogProcessor.
5. Add Logs suport for OTLP Exporter.
6. Complete documentation & examples.

[adriangb]

Hi, thank you for working on this, this is very exciting!

I have 2 questions:

1. Would it make sense to have some sort of logging.Filterer to inject OpenTracing data into logs? The use case would be to also include this data in logs to the console or other destinations
2. For the Handlers, it looks like the Handler itself transmits the record. I thought LogRecord == Event (conceptually, not referring to any actual classes), so shouldn't OTLPHandler .emit just map to add_event in the current span? (sorry if this is a stupid question, I'm new to OT)

And I left a followup question in https://github.com/open-telemetry/opentelemetry-python/pull/1903/files#r662742003

[srikanthccv]

For the Handlers, it looks like the Handler itself transmits the record. I thought LogRecord == Event (conceptually, not referring to any actual classes), so shouldn't OTLPHandler .emit just map to add_event in the current span? (sorry if this is a stupid question, I'm new to OT)

No the LogRecord data model is different and also span events are limited in what they accept as attribute values.

Would it make sense to have some sort of logging.Filterer to inject OpenTracing data into logs? The use case would be to also include this data in logs to the console or other destinations

This https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-logging?. And to give you more context you can configure the processor to export to any place you want file or n/w dest or anything really. It is designed in such way that you can plug in your own exporter if there is a need.

[adriangb]

LogRecord data model is different and also span events are limited in what they accept as attribute values

Hmm, I guess I'm a bit confused now because the spec explicitly says that logs and events are the same thing. Is there any reading material you could point me to that might help me resolve my confusion? I want to minimize how much I hijack this thread with naïve questions.

I guess my main concern was having to configure multiple things / having multiple things writing OpenTelemetry data to the network. Would I be able to configure this handler to share a single batcher/exporter with manually emitted traces, or an instrumented framework?

This (link)

Yeah, I looked at that before. It seems like it's modifying the global log record factory. I would have made it a filter so that it could be applied selectively, but I guess the point is zero configuration, which that achieves.

[srikanthccv]

The intent is to reduce the notion of Event being seen as an entirely new concept OpenTelemetry introduced. The span Events are structured logs. It doesn't have severity, body, and doesn't allow any nested attributes. Please look at the logdata model https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#add-events and initial logging sdk spec for more info https://github.com/open-telemetry/oteps/blob/main/text/logs/0150-logging-library-sdk.md

[srikanthccv]

Initial set of tasks are completed. Anyone interested in trying this out can use pip editable install and use the collector exporter.

[lzchen]

@lonewolf3739
Can this issue be closed then?

[srikanthccv]

Yes, maybe close this after next release?

[codeboten]

What's the plan for shipping this along with the rest of the packages?

[lzchen]

@codeboten
We can probably ship an alpha version similar to what we did for metrics. https://pypi.org/project/opentelemetry-sdk/1.10a0/
Version number can probably be 1.20a0 or something.

[codeboten]

I wonder if we should ship it altogether with metrics. As in have a single experimental package, rather than a package for logging and another for metrics... any thoughts?

[lzchen]

@codeboten
I have no problem with this.

[plajjan]

I would like to use the experimental metrics support and the experimental logging support. It seems easier to me to have a single experimental version to pull in, but as long as it is possible to achieve in some way, I'm happy.

[lzchen]

@lonewolf3739

We did not ship metrics as part of the most recent release because it is not usable yet. If we plan on shipping both together, should we wait or can logs be shipped simply by itself as "experimental"?

[srikanthccv]

It can be released alone as experimental but I am fine with shipping them together as well. I remember some people on slack channel also asked for the possible release of logs sdk (along with otlp-grpc-exporter). maybe release logs now and release them combined when metrics is ready?

[overmeulen]

Hello, I'm also interested by an experimental release of the logs sdk. It would be even better if it contained both the metrics and the logs sdk but I can live with the logs sdk alone for now.

[ocelotl]

Sure, I think releasing logs separately makes more sense.

[lzchen]

@lonewolf3739
Can this issue be closed?