possible memory leak related to the Date header

[stefanoterna]

Hi all,
while I was load testing my application, and doing some tracemalloc, I noticed a continue increase in the count of references and in the allocated memory related to the following:

usr/local/lib/python3.12/email/utils.py:278: size=21.2 KiB (+72 B), count=449 (+2), average=48 B

The code at usr/local/lib/python3.12/email/utils.py:278 is in the python function format_datetime.
This is why I guess the problem is related to unreleased references to the Date header:

uvicorn/uvicorn/server.py

Line 235 in fe39100

current_date = formatdate(current_time, usegmt=True).encode()

Minimum reproducible example

So I started doing some experiments with a simple application with a single post function that just returns "ok":

• Pyhton 3.12.7
• FastAPI 0.115.0
• uvicorn 0.32.0
• running in docker

This is the code:

import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()

import asyncio
from fastapi import FastAPI, Request

import tracemalloc

def do_log(msg):
    logger.info(msg)

async def periodic_malloc(interval):
    prev_snapshot = None
    while True:
        snapshot = tracemalloc.take_snapshot()

        exclusions = ['abc', 'linecache', 'tracemalloc', 'logging']
        # take the total
        total_memory = sum(
            stat.size for stat in snapshot.statistics('lineno')
            if not any(exclude in stat.traceback[0].filename for exclude in exclusions)
        )

        if prev_snapshot:
            # compute delta
            delta_snapshot = snapshot.compare_to(prev_snapshot, 'lineno')
            delta_snapshot = [
                stat for stat in delta_snapshot
                if not any(exclude in stat.traceback[0].filename for exclude in exclusions)                
            ]
            # top 10
            delta_snapshot = delta_snapshot[:30]
        else:
            delta_snapshot = []

        prev_snapshot = snapshot

        formatted_stats = ''
        for stat in delta_snapshot:
            formatted_stats = f"{formatted_stats}\n{stat}"

        formatted_stats = f"{formatted_stats}\nTOTALMALLOC: {total_memory}\n---------------------"
        do_log(formatted_stats)

        await asyncio.sleep(interval)


app = FastAPI()

@app.post("/msg")
async def handle_post(request: Request):
    return "ok"

@app.on_event("startup")
async def on_startup():
    tracemalloc.start()
    malloc_task = asyncio.create_task(periodic_malloc(5))

Testing with randomicity of message rate and intervals

These experiments are performed with a Python 3.12.3 client doing the following in an infinite loop:

• randomly select a duration in a time range from 5 to 30 seconds
• randomly select a time_sleep from a list SLEEP_TIMES = [0.1, 0.2, 0.3, 0.05, 0.01]
• call the the /msg endpoint via requests.post (no persistent connection, no connection pool) for the selected time interval at the selected message rate, randomly selecting a payload from a list
• stop sending requests for 30 minutes with probability 1/144

Headers

headers = {'cache-control': 'no-cache', 'content-type': 'application/x-www-form-urlencoded'} , as application/x-www-form-urlencoded is a requirement I have from real-world clients.

The use of random is properly seeded for reproducibility.

Randomized experiments outcome

These are the outcomes from tracemalloc (['abc', 'linecache', 'tracemalloc', 'logging'] are excluded from tracing) :

Randomized experiments considerations

• when the client stops sending requests, the total memory allocation suddenly decrease, but the allocation from email.utils increase
• there is a sudden increase of about 10k in total memory allocation a little before 2024-10-27 04:00
• the same increase can be seen in the memory allocation from email.utils,
• so it seems like the entire increase in memory allocation is due to the unreleased now variable from email.utils

Testing without randomicity of message rate and intervals

These experiments are performed with a Python 3.12.3 client doing the following in an infinite loop:

• randomly select a payload from a list of payloads
• call the the /msg endpoint via requests.post (no persistent connection, no connection pool)
• wait for 0.1 seconds and repeat

headers = {'cache-control': 'no-cache', 'content-type': 'application/x-www-form-urlencoded'}

Not randomized experiments outcome

Not randomized experiments considerations

• when the client stops sending messages the email.utils allocation increase from about 2024-10-25 08:00 to 2024-10-25 10:00
• there is an initial increase at about 2024-10-24 200:00, both in total memory and email.utils
• it does not seem to be an increase during constant traffic rate (green and yellow regions): this is different from the previous case

Furhter considerations and research

When running the same server in Hypercorn, there is no trace of the email.utils unreleased reference

Thanks for your attention and help!