Space from /dev/shm not being released when log buffer is closed

[maspholm]

Hi,

I'm having an issue with log buffer memory not being released from /dev/shm when continously creating new Publications within the same process. The log-buffer file is deleted when the publication is terminated, but the actual disk space is not released until the process is terminated. The result is an effective leak of shared memory. This happens even though there are no FileDescriptors pointing to the log buffer file held by the application (checked with heap dump).

The behaviour can be observed using some bash tools:

Check deleted files for process:
ls -l /proc/19293/fd|grep deleted
...
lrwx------. 1 jenkins jenkins 64 Mar 28 10:29 377 -> /dev/shm/aeron-jenkins/images/UDP-00000000-0-0a2e80de-4444-8B8DBAF2-E3604E6D-4.logbuffer (deleted)
lrwx------. 1 jenkins jenkins 64 Mar 28 10:29 378 -> /dev/shm/aeron-jenkins/publications/UDP-00000000-0-0a2e80de-48745-8B8DBAF3-8FA700E9-6.logbuffer (deleted)

With lsof to check disk usage (~50Mb per file in this case):
lsof|grep deleted
...
java 19293 19844 jenkins 377u REG 0,18 50332096 304617838 /dev/shm/aeron-jenkins/images/UDP-00000000-0-0a2e80de-4444-8B8DBAF2-E3604E6D-4.logbuffer (deleted)
java 19293 19844 jenkins 378u REG 0,18 50332096 304613191 /dev/shm/aeron-jenkins/publications/UDP-00000000-0-0a2e80de-48745-8B8DBAF3-8FA700E9-6.logbuffer (deleted)

Disk usage on /dev/shm before process restart:
df -vh /dev/shm
Filesystem Size Used Avail Use% Mounted on
tmpfs 16G 103M 16G 1% /dev/shm

After restart:
df -vh /dev/shm
Filesystem Size Used Avail Use% Mounted on
tmpfs 16G 6.1M 16G 1% /dev/shm

The behaviour might be OS dependent. I'm using CentOS Linux 7.3.1611.

I found a simple patch for the problem, simply deleting the file contents before calling File::delete in MappedRawLog::close:

public void close()
{
    for (final MappedByteBuffer buffer : mappedBuffers)
    {
        IoUtil.unmap(buffer);
    }

    try (FileWriter fw = new FileWriter(logFile))
    {
    }
    catch (Exception e)
    {
        //Ignore
    }

    if (!logFile.delete())
    {
        errorLog.record(new IllegalStateException(String.format("could not delete file %s", logFile)));
    }
}

Although it feels like a bit of a hack it takes away the leak of shared memory space. The file will still be there in the lsof output, but occupying no space.

Thanks

[mjpt777]

This could be that you still have a link to the inode. Are all the Aeron clients closed too?

[sthornington]

This is the same issue I have been seeing and occasionally mentioning in the gitter.

[maspholm]

Thanks for quick reply. Indeed all Aeron clients are closed and GCed. (Also, from what I can see AeronClient::delete does nothing?). I even searched through all FileDescriptor objects on the heap and none of them were pointing to any of the log buffer files.

[mjpt777]

Fixed with 5532206