Logging to Application Insights Include logger name

[tkutcher]

Question

Ultimately, all I really want to be able to do is something like:

root_logger = logger.getLogger()
child_logger = logger.getLogger("some_child_module")

root_logger.info("a root log")
child_logger.info("a child log")

and in application insights be able to filter only those logs for the some_child_module logger, etc.

I am hoping to get some guidance on how to do this. It would be great if that could be supported by default somehow but for now I would just like some approach where I can filter logs for specific modules/packages, etc. I cannot find any resources on how to do this. I see #694 which seems to indicate there isn't really a straightforward way to add additional properties through custom_dimensions

[bhagyshricompany]

Hi @tkutcher Thanks for informing will discuss and update for this.

[EvanR-Dev]

Hi, following up on this:

You would likely need to instrument your logs using something such as the OpenCensus Python SDK. You could take advantage of custom_dimensions and add a "module" property to it to track the module that your logger belongs to. It can then be attached to each log and get parsed/filtered as needed. Here is an example:

import azure.functions as func
import logging
from opencensus.ext.azure.log_exporter import AzureLogHandler

app = func.FunctionApp(http_auth_level=func.AuthLevel.ANONYMOUS)

# Setup telemetry to Application Insights
logger = logging.getLogger(__name__)
logger.addHandler(AzureLogHandler())


@app.route(route="http_trigger")
def http_trigger(req: func.HttpRequest) -> func.HttpResponse:
    # The logs below show up under "traces" in Azure Monitor (Logs in App Insights)

    # Add module property to custom_dimensions
    properties = {'custom_dimensions': {'log_module1': 'module1'}}
    logger.info("Log test", extra=properties)

    # Change property name or value if desired for other logs
    properties = {'custom_dimensions': {'log_module2': 'module2'}}
    logger.info("Custom log test", extra=properties)

    return func.HttpResponse(
        "This HTTP triggered function executed successfully",
        status_code=200
    )

Log 1:

Log 2:

[tkutcher]

Thanks. Yea I am trying to avoid passing an extra for every single log as that doesn't really cover what I am looking for - I was looking more for something that could use the logger instance itself to include information on which logger made the log, not which metadata I explicitly attached to a given log.

The main use case is that I have some common packages that I want to include logs for, and I don't want the constraint on a dependent package to be to include something specific to the functions runtime.

I was looking for something more along the lines of decorating the AzureLogHandler but I couldn't quite figure out how to properly decorate it (or maybe I did, but I was failing to get any of these logs to properly show up in ApplicationInsights...).

I feel for Python applications something along these lines would be particularly useful for logging monitoring/troubleshooting - so hoping to get at least an example where this sort of code would create logs that automatically include (in the custom dimensions) the logger name:

root_logger = logger.getLogger()
external_library_logger = logger.getLogger("my_other_package")

# do something to override/intercept default logging behavior
#  Either a decorator on the AzureLogHandler, or other idea was to change the log record factory to one
#  that automatically adds custom_dimensions in the extra properties:
#
#  logging.setLogRecordFactory(custom_record_factory)

root_logger.info("a root log, specific to my azure function app")
external_library_logger.info("a child log")  # this could be in the other package, which is not az-func specific

[macieyng]

Please, don't use OpenCensus, use OpenTelemetry instead. You'll find a lot of resources in Azure Docs and OTEL. There is a thing called distro auto instrumentation that will do a lot of stuff for you. If you have any issue you can raise an issue in Azure SDK Python repo. If you want community help, join Cloud Native Foundation slack, otel-python channel.

Check those resources first. If you need any type of customization once you implement the aut instrumentation - ping me 👋

[dmillican-camillo]

Please, don't use OpenCensus, use OpenTelemetry instead. You'll find a lot of resources in Azure Docs and OTEL. There is a thing called distro auto instrumentation that will do a lot of stuff for you. If you have any issue you can raise an issue in Azure SDK Python repo. If you want community help, join Cloud Native Foundation slack, otel-python channel.

Check those resources first. If you need any type of customization once you implement the aut instrumentation - ping me 👋

Perhaps I'm looking for the wrong keywords, but I'm having a hard time finding an example of basic logging using OpenTelemetry with Applicaiton Insights. I thought I heard something about the logging feature of the python implementation of OpenTelemetry not being completed yet but that could have been a dated statement. But the person did say that tracing and metrics were implemented and logging was not yet at the time. I'd love to see a reference app with simple logging, traces, and metrics going to Application Insights.

[macieyng]

Hi @dmillican-camillo 👋 I needed time to gather things that I know and create examples. Here is a repo https://github.com/macieyng/otel-examples. Let me know if you'll find it helpful, if not open an issue and I do my best to support you.

[konstantinmiller]

You can normally inject extra information into your emitted log records by adding a LoggerAdapter to your logger. In addition, using a LoggerInjector will even allow you to inject fields computed dynamically (e.g. a user ID for the current request). Wrap this into a function that you will use across your code to create loggers.

class LogInjector(Mapping):
    static_fields = {
        'your_static_field': 'static_field',
    }
    # you can also add a list with keys for dynamic fields you want to inject
    # you need to then compute the values for them in the __getitem__ method

    def __getitem__(self, item: str) -> str:
        return self.static_fields[item]

    def __iter__(self):
        return iter(self.static_fields)

    def __len__(self):
        return len(self.static_fields)

class LoggerAdapter(logging.LoggerAdapter):
    def process(self, msg, kwargs):
        kwargs['extra'] = {**self.extra, **kwargs.get('extra', {})}  # merges with the, potentially already existing, extra argument
        return msg, kwargs

def get_logger(name):
    logger = logging.getLogger(name)
    logger = LoggerAdapter(logger, extra=LogInjector())
    return logger

However, as far as I can see it, with Azure Functions in Python, this is currently not possible due to this bug. Namely, Azure Functions overwrite the supplied extra field. And seems to be no solution for that since, meanwhile, over two years.