Control number of open files in libvips' cache

[impomezia]

Just simple open this gif file http://i.imgur.com/yBEqU6A.gif by using simple code:

sharp('yBEqU6A.gif').metadata()
    .then(metadata => {
    });

It causes leak about 500 MB of memory for each open file. If 4 times to open this file, will leak about 2GB. Probably related fix https://github.com/jcupitt/libvips/pull/313

Affected current npm vesion for Windows and Ubuntu 14.04

It also may be a good idea to do sharp.format read/write, for example to disable magick.

[lovell]

Hello, please can you add sharp.cache(0) to disable libvips' cache before any other calls to sharp to see if this is a leak vs cache thing.

libvips only releases the memory for *magick opened files when the file is closed, which it won't do whilst an image is in its operation cache. In addition, I believe the memory consumed by *magick images is not included in libvips' cache memory limit.

[impomezia]

sharp.cache(0); do not make any difference.

Additional information
If open one file multiple times, it will leak only 500 MB, so memory not freed only for last open.

Simple test:

'use strict';

const sharp = require('sharp');
sharp.cache(0);

console.log(process.memoryUsage());

function test() {
  sharp('yBEqU6A.gif').metadata()
    .then(metadata => {
      console.log(process.memoryUsage());
      //console.log(`${metadata.width}x${metadata.height}`);
    });
}

for (let i = 0; i < 3; i++) {
  test();
}

setInterval(() => {}, 60000);

Result:

{ rss: 29585408, heapTotal: 12359168, heapUsed: 6666800 }
{ rss: 1845100544, heapTotal: 12359168, heapUsed: 6953936 }
{ rss: 1245327360, heapTotal: 12359168, heapUsed: 6998336 }
{ rss: 646963200, heapTotal: 12359168, heapUsed: 7008344 }

But if open different files, memory leak for each file.

[lovell]

Thanks for the quick reply and test code.

Modifying your example to also set the number of items in the cache to zero by using sharp.cache(0, 0); seems to sort this out. (The first parameter of cache refers to memory assigned/tracked by libvips, but this memory is assigned by *magick, so isn't included.)

You'd think header-only access to a GIF file would be easy, but *magick has to decompress the whole image before it can return metadata - see http://www.imagemagick.org/discourse-server/viewtopic.php?f=1&t=20017 - this animated GIF blows up to well over a gigabyte of RAM. Ouch.

Perhaps sharp's cache function should also accept a boolean that, when false, turns off all libvips caching, i.e. cache(false) implies cache(0, 0).

It might be worth modifying libvips to never cache the result of the *magick loader as it has the ability to hog so much memory.

[lovell]

@impomezia Have you been able to make any progress?

[lovell]

The new mind branch contains commit 0c910f7 to allow support for controlling the number of open files to the existing cache method.

https://github.com/lovell/sharp/blob/mind/docs/api.md#sharpcacheoptions

[lovell]

v0.13.0, with improved cache control, is now available.