Mulitple Log FIles

[sahilmtayade]

Is there a recommended way to log to multiple files? I would like to create a second logger that outputs to a new log file. I do not want the new logger to destroy or modify the existing logger that has been customized using setup_logging.

Thanks in advance.

[Dragon2fly]

Hi @sahilmtayade

Could you share more about your program setup? Are you using multiprocessing, threading, or two separate applications?

Basically, the log destination is defined by a log handler. For one application to log into different files, you add a handler for each file and set the condition for what log when to what file.

That being said, for different applications that run simultaneously, setting the logging port differently for each application is sufficient.

[sahilmtayade]

Hi @Dragon2fly

My program is single process, multi threaded. It is a Flask server that logs api requests and current state. I want to start another thread that logs to a different file for just task progress tracking.
I'm not sure how to make more handlers, since I just use the default logger, customized using setup_logging for the file path, suppression, context, and logger.setLevel.
I don't think I can use setup_logging again without affecting the default logger, which is why I opened an issue.

[sahilmtayade]

Maybe if there is some function that acts like setup_logging but returns a new logger, that would be easy to use.

[Dragon2fly]

Hi @sahilmtayade

Your case is simple, logging with threading is fully supported by default.
Python's logging requires 2 things: a handler and a logger that uses the handler.

Instead of doing the setup in code, you will do it in the log config file. The code just uses it.

---

(1) Copy the log config file (either json or yaml) and paste it into your project folder.
I'll use json file as a demonstration.

(2) Create a new handler that logs to the file (similar to error_file_handler).
In the handlers section, add something like the below.

"task_progress_handler": {
      "class": "logging.handlers.TimedRotatingFileHandler",
      "level": "DEBUG",
      "formatter": "simple",
      "filename": "logs/task_progress.txt",
      "backupCount": 15,
      "encoding": "utf8",
      "when": "midnight",
      "delay": true
    },

You could customize all the parameters as it is described in TimedRotatingFileHandler.
Note that here it still uses the simple formatter. If you need more control over the format of the output message, you could go up to the "formatters" section and create your own.

(3) Create a new logger that uses the previously added handler
Move to the loggers section in the config file, you should see a premade urllib3 logger section.
You will add something similar to that logger.

"progress_logger": {
      "level": "DEBUG",
      "handlers": ["console", "task_progress_handler"],
      "propagate": false
    }

Notice that I also add the "console" handler here. This will print the message into the console.
If you want the log message only to appear in the file, keep only "task_progress_handler".
Also, "propagate" should be kept as false to avoid the root logger logging your message again.

(4) In the setup_logging(), point it to the new config file.

from logger_tt import setup_logging

setup_logging(config_path="log_config.json")   

This assumes the log_config.json is in your project folder. Otherwise, you should use the full path to the file.

(5) Use the new logger in the code.
For all the py files that need logging, your current code with the logger may look like the below

from logging import getLogger

logger = getLogger(__name__)

You will add one line for the new logger and start using it where applicable

from logging import getLogger

logger = getLogger(__name__)                            # general logger
progress_logger = getLogger("task_progress_handler")    # progress related logger

def my_task(*arg, **kwarg):
    logger.info("My task started")
    ...
    progress_logger(f"processed item {i}th")
    ...
    logger.info("My task finished")

---

That is it. Hope it helps.