Add configurable logging filter

[br3ndonland]

Description

When applications with APIs are deployed, it is common to perform "health checks" on them. Health checks are usually performed by making HTTP requests to a designated API endpoint. These checks are made at frequent intervals, and so they can fill up the access logs with large numbers of unnecessary log records.

To avoid logging health checks, it would be helpful to have a way to filter health checks out of the logs.

Changes

This PR will add a configurable logging filter to inboard.logging_conf.

• Filters can be provided as a comma-separated string with the environment variable LOG_FILTERS, like LOG_FILTERS="/health, /heartbeat".
• The environment variable value will be converted into a set of strings.
• Each log message will then be checked for each filter in the set. If any matches are present in the log message, the logger will not log that message.

There are multiple ways to match these filters. inboard simply performs a substring membership check to see if any of the filters in the set have a match in the log message. This should work for most use cases, but the match could be overly greedy. For example, if /health is in LOG_FILTERS, it would filter out calls to /health, but it would also filter out calls to /healthy.

Another approach could be to operate on the LogRecord args, but the results would vary based on how the program supplies the log record.

• As seen in the type stubs for the logging module, the args can be either tuple[object, ...] or Mapping[str, object]. The args could be converted to a set, and compared with the set of filters using isdisjoint. This set comparison approach is potentially more precise than string matching.
• Gunicorn's access logger gunicorn.access supports several custom message fields, which it calls "atoms" in its source code, and "identifiers" in the docs. Gunicorn formats these atoms and supplies them as LogRecord args. The URL path could be logged with the %(U)s atom, and inboard could check to see if there is an exact match in the set of filters.
• Uvicorn's access logger uvicorn.access automatically supplies the URL path as its own arg, which could also work nicely with this approach.
• However, LogRecord args aren't always supplied as strings directly, so there would probably need to be isinstance checks added to check each of the args. For example, when watching files with watchgod (now renamed to watchfiles), Uvicorn logs the list of files being watched as a single arg. Lists are not hashable, so the list would need to be converted to another type (like set, or str as the logger does when formatting messages) in order to perform equality comparisons.

Substring membership checks should be adequate for this feature, and they avoid the complications of working with LogRecord args directly.

Related

• disable health check logging in access logging benoitc/gunicorn#1781
• Python 3 docs: How-To - Logging Cookbook - Using Filters to impart contextual information
• Python 3 docs: What's new in Python 3.2 - logging
• Django 4.0 docs: How-to - Logging
• Django 4.0 docs: Reference - Logging
• Django 4.0 docs: Topics - Logging
• I have reviewed the Guidelines for Contributing and the Code of Conduct.

[vercel]

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/br3ndonland/inboard/9mpRgFfvGc84eAAZuwyARSekyLjP
✅ Preview: https://inboard-git-logging-filter-br3ndonland.vercel.app

[codecov]

Codecov Report

Merging #49 (fba2fd2) into develop (80cbaf4) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##           develop       #49   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           10        10           
  Lines          266       281   +15     
=========================================
+ Hits           266       281   +15     

Flag	Coverage Δ
unit	100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
inboard/__init__.py	100.00% <100.00%> (ø)
inboard/logging_conf.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 80cbaf4...fba2fd2. Read the comment docs.

[sourcery-ai]

Sourcery Code Quality Report

❌  Merging this PR will decrease code quality in the affected files by 0.93%.

Quality metrics	Before	After	Change
Complexity	0.70 ⭐	0.83 ⭐	0.13 👎
Method Length	59.59 ⭐	59.56 ⭐	-0.03 👍
Working memory	6.86 🙂	7.05 🙂	0.19 👎
Quality	79.86% ⭐	78.93% ⭐	-0.93% 👎

Other metrics	Before	After	Change
Lines	714	833	119

Changed files	Quality Before	Quality After	Quality Change
inboard/init.py	95.67% ⭐	95.34% ⭐	-0.33% 👎
inboard/logging_conf.py	71.00% 🙂	75.15% ⭐	4.15% 👍
tests/test_logging_conf.py	83.21% ⭐	79.29% ⭐	-3.92% 👎
tests/test_start.py	79.34% ⭐	79.19% ⭐	-0.15% 👎

Here are some functions in these files that still need a tune-up:

File	Function	Complexity	Length	Working Memory	Quality	Recommendation
tests/test_start.py	TestStartServer.test_start_server_uvicorn_reload_dirs	2 ⭐	157 😞	12 😞	56.57% 🙂	Try splitting into smaller methods. Extract out complex expressions
tests/test_logging_conf.py	TestLoggingOutput.test_logging_filters	2 ⭐	134 😞	10 😞	62.70% 🙂	Try splitting into smaller methods. Extract out complex expressions
tests/test_start.py	TestStartServer.test_start_server_uvicorn_gunicorn_custom_config	0 ⭐	120 😞	9 🙂	68.44% 🙂	Try splitting into smaller methods
tests/test_start.py	TestStartServer.test_start_server_uvicorn_gunicorn	0 ⭐	100 🙂	10 😞	69.00% 🙂	Extract out complex expressions
inboard/logging_conf.py	LogFilter.set_filters	2 ⭐	32 ⭐	10 😞	77.42% ⭐	Extract out complex expressions

Legend and Explanation

The emojis denote the absolute quality of the code:

• ⭐ excellent
• 🙂 good
• 😞 poor
• ⛔ very poor

The 👍 and 👎 indicate whether the quality has improved or gotten worse with this pull request.

---

Please see our documentation here for details on how these metrics are calculated.

We are actively working on this report - lots more documentation and extra metrics to come!

Help us improve this quality report!