Random read_entry "decompression error" exception with zlib 1.2.9+

[einars]

Since the notorious zlib update, I've been having hard-to-pin-down somewhat reproducible issues when processing large zip files (700 mb) and using a lot of memory while processing. I can reproduce them all the time with my core+camlzip processor on of my huge file that I am not able to share publicly, but been unable to extract to a smaller, reproducible case (even adding a debug output may make the issue go away).

The problem manifests itself as suddenly being unable to unpack zip file — the Zlib.Error "decompression error" on random files gets thrown, whereas the zip file is itself perfectly fine.

What I've been able to pinpoint so far is that it started occuring with a zlib commit madler/zlib@b516b4b — Mark Adler added some sanity checks,

The exception gets thrown in camlzip_inflateEnd, as zlib returns error,

the reason for exception is that inflateStateCheck checks the stream structure, and inside there's a "state" substructure that has a reverse pointer to the stream. The new check verifies if these streams are actually equal (madler/zlib@b516b4b#diff-327188edf18799ffbb5a51cc69f797e8R113) — and suddenly, they are not anymore.

Here's my zlib debug info,

# let lines = Zip.read_entry z entry |> String.split ~on:'\n' in ...
inflatestatecheck failed
strm   0x7f7c7cd8f7b0
state  0x2a35070
state->strm 0x7f7c887b5100
state->mode 16203 (distext)
Uncaught exception:

Zip.Error("weather.zip", "wlask.min", "decompression error")

Called from file "src/exn.ml", line 90, characters 6-10

I suppose that probably the garbage collector or something sometimes moves things around and the structure turns invalid, or something — — — any ideas?

(Up-to-date 64-bit archlinux, ocaml 4.04.0 and all via opam)

[xavierleroy]

You are correct that the z_stream is allocated in the OCaml heap and can therefore be moved by the GC. I had no idea this could be a problem at all.

[rlepigre]

I'm having a similar problem with zlib 1.2.11 while opening a fairly small PNG image (117Ko) using imagelib. The problem is solved by the code of the pull request #2