Memory Leaks

[melancthon]

The following code generates memory leaks and eventually crashes.

I've tried it with node v11.10.0 and v10.15.1LTS on both Windows 7 SP1 and CentOS 7.

Any ideas?

const lz4 = require("lz4")

for(let i=0;i< 10000; i++){
	let a = lz4.decode(lz4.encode("0123456789")) // memory leak!!
	if(i%100==0){
		let res = process.memoryUsage()
		console.log("mem:", res.external.toLocaleString(), res.heapTotal.toLocaleString(), res.heapUsed.toLocaleString(), res.rss.toLocaleString())
	}
}

[ankon]

Small data point here: By using the useJS option the leak seems to be considerably smaller.

const lz4 = require(".");

const opts = {useJS: true};

for (let i = 0; i < 10000; i++) {
	let a = lz4.decode(lz4.encode("0123456789", opts), opts); // memory leak!!
	if (i % 100 == 0) {
		let res = process.memoryUsage();
		console.log(
			"mem:",
			res.external.toLocaleString(),
			res.heapTotal.toLocaleString(),
			res.heapUsed.toLocaleString(),
			res.rss.toLocaleString()
		);
	}
}

[ankon]

I extended the test tool a bit as I found the initial output very scary, but after reviewing the binding code here and playing with some theories I think there is no real leak here -- rather what happens is that NodeJS/V8 figure out that running a GC isn't needed in this example.

The extended test code:

try {
	const gc = require('gc-stats')();
	gc.on('stats', stats => {
		const { gctype, pauseMS, diff: {usedHeapSize}} = stats;
		console.log(`gc ${gctype}: ${usedHeapSize}\t(${pauseMS}ms)`);
	});
} catch (err) {
	// Fine.
}

const lz4 = require('.');

const opts = {useJS: process.env.USE_JS === 'true'};
const iterations = Number(process.argv[2]) || 10000;
const forceGc = process.env.FORCE_GC === 'true';
const taskDelay = typeof process.env.TASK_DELAY !== undefined ? Number(process.env.TASK_DELAY) : undefined;

function runGc() {
	// Run with --expose-gc to make this available
	if (global.gc) {
		global.gc();
	}
}

function printStats(prefix) {
	let res = process.memoryUsage();
	console.log(
		`mem (${prefix}):\t`,
		[
			res.external.toLocaleString(),
			res.heapTotal.toLocaleString(),
			res.heapUsed.toLocaleString(),
			res.rss.toLocaleString(),
			res.arrayBuffers.toLocaleString(),
		].map(s => s.padStart(12)).join('\t')
	);
}

function delay(ms) {
	return new Promise(resolve => {
		setTimeout(resolve, ms);
	});
}

function test() {
	let encoded = lz4.encode('0123456789', opts);
	let decoded = lz4.decode(encoded, opts); // memory leak?
}

async function main(what, iterations) {
	let iteration = 1;
	do {
		console.log(`Running iteration ${iteration}`);
		runGc();
		printStats('initial');
		for (let i = 0; i < iterations; i++) {
			// Run the code-under-test
			what();

			// Do some statistics
			if (i % 100 == 0) {
				printStats('before');
				if (forceGc) {
					runGc();
					printStats('after');
				}
				if (typeof taskDelay !== 'undefined') {
					await delay(taskDelay);
				}
			}
		}
		printStats('final');
		await delay(5000);
	} while (iteration++);
}

main(test, iterations);

Some things to try:

1. Limit the memory using --max-old-space-size=5. This works both with the native bindings and the JS implementation. Anything lower and NodeJS crashes immediately for me (Linux/x86_64)
2. Run an event loop iteration every few test iterations by setting TASK_DELAY=0 (or any other integer value): This immediately shows that GCs are running when the gc-stats package is installed, and the memory statistics start behaving very differently.
3. Try to force a GC every few test iterations (using FORCE_GC=true): This doesn't change much unless there is a delay as well. Probably because the GC is triggered as a part of the event loop?
4. Just do nothing and let it run multiple iterations: Even just that shows that GCs are happening the moment the test waits for a while, and essentially things continue to run forever.

Basically: the leak here is a leak because the test loop is so tight that NodeJS/V8 simply don't run a GC.