Memory leak

[superbock]

Somewhere in madmom is a memory leak. It revealed while I was working on #316 and testing different settings of the new BarTracker. My first guess was that CLPChromaProcessor eats memory, since it appears mostly when using this processor. Further investigation revealed that FramedSignal seems to be the culprit, or at least parts of it.

So far I have not been able to really spot the leak, however gc.get_objects() listed countless slices objects, all having a start and stop 2048 apart, thus I suspect that the signal_frame() function is the problem. But I can't see anything fishy inside, except the last line return signal[start:stop].

If I perform countless numpy array view operations, I can somehow replicate the behaviour:

import numpy as np
import gc
x = np.arange(100000000)  # uses ~800MB memory
for i in range(len(x) - 20):
    x[i: i+20]  # memory rapidly grows up to ~3.8GB, then stays at this level, goes down to ~3GB at the end of the loop
del x  # memory goes down to 2.3GB
gc.collect()  # remaining memory is freed

I expected that del x would free the memory, however this is not the case. Is this behaviour only local to my machine or can someone else replicate this?

[superbock]

The above code shows similar behaviour, but the "fix" doesn't change anything. So the above code simply demonstrates a similar behaviour but actually has nothing to do with it.

[superbock]

Reopening, since not all memory/reference leaks are fixed. Programs still show increasing memory utilisation, though at a much lower level.

Most programs still leak 3 or 4 slices per frame to be processed in online mode. If multiple frame sizes are used, these numbers are multiplied accordingly.

[superbock]

Looks like only Python 2.7 is leaking references, thus I am not sure if we should further investigate this issue.

[fdlm]

• Do the programs just leak the memory in online mode?
• Can you provide code that checks if a madmom program leaks in online mode? I would then check if it also happens on my machine.

[superbock]

No, this also happens in offline mode as well. Batch mode shows the same behaviour, it is just easier to monitor in online mode since longer durations can be processed more easily.

Somewhere in the code accessing the individual frames of a FramedSignal leaks references to slices. So far I have not been able to track it further down. However I am not sure if this is something we could fix easily since only Python 2.7 shows this behaviour.

#331 has some debug code to count the slice references and a fix to one leak (which only happens in online mode). Still, there are 4 references per frame (and frame size) leaking.