Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak if decompressed stream is not explicitly closed #27

Closed
robsmith11 opened this issue Feb 14, 2024 · 0 comments · Fixed by #43 or #45
Closed

Memory leak if decompressed stream is not explicitly closed #27

robsmith11 opened this issue Feb 14, 2024 · 0 comments · Fixed by #43 or #45

Comments

@robsmith11
Copy link

I was iterating over thousands of bzip2 files with a function like this:

function read_first_line(filename)
    open(filename) do f
        f2 = CodecBzip2.Bzip2DecompressorStream(f)
        r = first(eachline(f2))
        # close(f2)
    end
end

The original form (with close(f2) commented out) would leak over 100 GB memory and begin swapping. Calling GC.gc() had no effect on the memory usage. But when using close(f2), the julia process memory usage remains under 1 GB.

So I'm guessing some memory isn't getting cleared up properly. I would expect that the f2 decompressed stream should be cleared from memory at some point after it goes out of scope, at least before the Julia process runs out of system memory or when GC.gc() is run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant