Enable Powertools logging for imported libraries

[FredrikMeyer]

Until recently, I used the following code for enabling Powertools logging for imported libraries:

from aws_lambda_powertools import Logger
import os
import logging as pyLogging

# https://stackoverflow.com/a/56579088/1013553
pyLogging.getLogger().setLevel(pyLogging.INFO)

import boto3

boto3.set_stream_logger()
boto3.set_stream_logger("botocore")

location_format = "%(name)s.%(funcName)s:%(lineno)d"

logger = Logger(service=os.environ["POWERTOOLS_SERVICE_NAME"], location=location_format)

# Enable JSON logging for libraries
loggers = [pyLogging.getLogger(name) for name in pyLogging.root.manager.loggerDict]
for l in loggers:
     l.addHandler(logger.registered_handler)

This worked, but resulted in duplicated logs (sometimes three times).

Is there a recommended way to make all logs JSON and with the same level?

Thanks!

[heitorlessa]

hey Fredrik, thanks for opening an issue. Besides boto, are there any other existing loggers you want to change? If you think it's helpful, we could add an additional method to make this process easier - can't think of a name now but it's doable, then you would simply have one line to do this and we'll safely do that for you. The reason you're seeing duplicates or more is that Lambda inadvertently <https://docs.aws.amazon.com/lambda/latest/dg/python-logging.html#python-logging-lib>, so when you set *pyLogging.getLogger().setLevel(pyLogging.INFO)* it allows any log messages from any logger to be propagated due to Python Logging hierarchy.

…

On Thu, 4 Nov 2021 at 11:19, Fredrik Meyer ***@***.***> wrote: Until recently, I used the following code for enabling Powertools logging for imported libraries: from aws_lambda_powertools import Loggerimport osimport logging as pyLogging # https://stackoverflow.com/a/56579088/1013553pyLogging.getLogger().setLevel(pyLogging.INFO) import boto3 boto3.set_stream_logger()boto3.set_stream_logger("botocore") location_format = "%(name)s.%(funcName)s:%(lineno)d" logger = Logger(service=os.environ["POWERTOOLS_SERVICE_NAME"], location=location_format) # Enable JSON logging for librariesloggers = [pyLogging.getLogger(name) for name in pyLogging.root.manager.loggerDict]for l in loggers: l.addHandler(logger.registered_handler) This worked, but resulted in duplicated logs (sometimes three times). Is there a recommended way to make all logs JSON and with the same level? Thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#799>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBCJSMXKXTMRMZLY26LUKJMZVANCNFSM5HLCHIYA> .

[heitorlessa]

yep, what I was thinking is having two parameters - level, deny_list - since you might want to have a different level for all of them (+noise), and you might have loggers you don't want to have it in the same format (+Log ingestion/storage cost).

If you want all imported libraries logging to also use the same JSON structure as your Powertools Logger, then it's a quick PR to get it done.

If you're in agreement, feel free to open a feature request and/or a PR- we can get it in the next release (this coming Monday by EOD the latest).

;)

[FredrikMeyer]

Thank for the very quick reply!

I'll make a feature request straight away. I'm not familiar with the logic of the log handlers, but if it's very easy I could try to do a PR as well.

So you're thinking something like this?

logger = Logger(...., level="INFO", deny_list=["dont-log-this-library", "and-not-this"])

[FredrikMeyer]

I created a feature request here now :) https://github.com/awslabs/aws-lambda-powertools-python/issues/800

[FredrikMeyer]

Btw, If I were to try to make a PR for this. How would I do it? Is it as simple as the example in the docs for enabling boto3 logs?

import boto3
boto3.set_stream_logger()
boto3.set_stream_logger('botocore')

[heitorlessa]

For this feature, it'd need to be slightly more involved. From the top of my head, it would need:

1. Add a new method in Logger to propagate its configuration to any loggers available, except the ones in the deny list
2. Add functional tests
3. Add docs explaining how to use it (we can do that)

I'm gonna add these in the feature request for completeness.

[heitorlessa]

I was thinking of a public method for this in the Logger class, so it’s more explicit on the behaviour this drives - and we can have docstrings to tell them what this does in case the name isn’t perfect for the job. Hardest part is naming :-) *As a public method:* logger = Logger() logger.clone_config_to_all_loggers(level=…, deny_list=[“logger_named_X”, “botocore”]) *As a standalone function* logger = Logger() clone_logger_handler(source_logger=logger, deny_list=[“logger_named_X”, “botocore”]) I like the latter as it’s standalone, since we’re not modifying anything at the source logger but copying its Handler & Log Level. But naming is hard haha

…

On Thu, 4 Nov 2021 at 12:52, Fredrik Meyer ***@***.***> wrote: Thank for the very quick reply! I'll make a feature request straight away. I'm not familiar with the logic of the log handlers, but if it's very easy I could try to do a PR as well. So you're thinking something like this? logger = Logger(...., level="INFO", deny_list=["dont-log-this-library", "and-not-this"]) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#799 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBBBDTPKSDNEYGBZ653UKJXWNANCNFSM5HLCHIYA> .

[FredrikMeyer]

Aha, that's better! Of those I would probably prefer number 1, it seems more "pythonic" at first glance. But both works!