Memory tracked and collected by Memray far exceeds the resource limits set by Docker

[jalr4ever]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Hi, I am currently testing a memory usage issue. I started a Python 3.8 container using the following command and tested my program's memory usage with memray run --live:

docker run -itd \
  --name python38_limit \
  --cpus 8 \
  --memory 16g \
  --gpus all \
  python:3.8 /bin/bash

After a while, I found that Memray reported a maximum memory usage of 34G, which far exceeds the 16G memory limit of my container. Meanwhile, my task is still running.

When I use docker stats to check my container's resource usage, it shows far less than 34G:

CONTAINER ID   NAME             CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O        PIDS
5b9183ee1a7c   python38_limit   216.84%   2.305GiB / 16GiB    14.40%    11.6MB / 365kB   1.47GB / 140MB   31

Why is this happening?

Expected Behavior

Collect the correct physical memory usage of the program.

Steps To Reproduce

Using the following command to create a container on a GPU server (Nvidia V100):

docker run -itd \
  --name python38_limit \
  --cpus 8 \
  --memory 16g \
  --gpus all \
  python:3.8 /bin/bash

And then, using the code below and csv (20240821152459_100000_sample40000.csv)

from datetime import datetime

import pandas as pd
from sdv.evaluation.single_table import evaluate_quality, run_diagnostic
from sdv.metadata import SingleTableMetadata
from sdv.single_table import CTGANSynthesizer

diagnose = False
export = False
file_name = '../datasets/guests.csv'
current_time = datetime.now().strftime("%Y%m%d%H%M%S")
file_output_name = f'{file_name}_{current_time}_CTGAN_gen_synthetic.csv'

real_data = pd.read_csv(file_name)
print(real_data.head(10))

# Create metadata
metadata = SingleTableMetadata()
metadata.detect_from_dataframe(real_data)

synthesizer = CTGANSynthesizer(
    metadata,
    epochs=300,
    verbose=True
)
synthesizer.fit(real_data)
# sample
synthetic_data = synthesizer.sample(num_rows=1000)
print(synthetic_data.head())

Memray Version

1.14.0

Python Version

3.8

Operating System

Linux

Anything else?

No response