Memory leak in fs2.compress.deflate

[lukestephenson]

There is a memory leak in fs2 when using fs2.compress.deflate with a large Stream. Similar to #987, it results in a long chain of FreeC instances.

The code to reproduce is:

package benchmarks

import java.io.File
import java.util.zip.Deflater

import cats.effect.IO
import fs2.compress.deflate

object CompressTest {
  val DefaultBufferSize = 1
  val src = "/Users/luke.stephenson/projects/listings/results4.txt"
  val target = "/Users/luke.stephenson/projects/listings/results3.txt"

  val bufferSize: Int = 32
  val level: Int = Deflater.DEFAULT_COMPRESSION

  def main(args: Array[String]): Unit = {
    val file = new File(src)

    val start = System.currentTimeMillis()

    def process = fs2.io.file.readAll[IO](file.toPath, DefaultBufferSize)
        .through(deflate(
          level = level,
          nowrap = true,
          bufferSize = bufferSize
        ))
      .run

    process.unsafeRunSync()

    val end = System.currentTimeMillis()

    println(s"took ${end - start}ms")
  }

}

Be sure to run this against the latest version of fs2 containing PR #992. With the source being a very large file, the heap dump shows:

This is probably the root cause of a memory leak in the http4s GZip implementation. http4s/http4s#1561

[lukestephenson]

@mpilquist I'd really love to know how you are tracking down these memory leaks.

[SystemFw]

@lukestephenson There's a writeup in #964

[mpilquist]

This leak is caused by _deflate_stream and _deflate_step being defined corecursively where _deflate_stream calls _deflate_step out of tail position (it adds an extra flatMap on the result). I haven't tried fixing yet so if someone beats me to it, that's great. :)