Create zip by concatenating pre-compressed files

[craigds]

Our websites offer generated downloads of user-filtered data. Data is generated by workers in parallel while the user is waiting, and the user clicks a link to download it once the workers have completed processing.

At present the workers can generate files in parallel, but the final task of creating a zip is non-parallelisable. We'd like to fix that so the users can start downloading data faster.

To stream zips to the user, we'd want:

• workers generating DEFLATE-compressed files rather than non-compressed files. ie. all files produced would be run through gzip
• mod_zip (or similar project I haven't found yet) would stream those files into a coherent zip, without re-compressing.
• correct zip headers

My understanding is that mod_zip can't currently create zips from pre-compressed files. Is that correct? What is your feeling about how difficult it would be to implement?

I also gather from #23 that mod_zip generates non-compressed zip files, so it can produce correct filesize headers. Is that still correct? This change would obviously also fix that.

[martin-sucha]

@craigds Based on your description above, you might be interested in zipserve Go library that I made just this weekend. It supports deflated (or any other zip compression method, since the library does not handle the actual compression itself) file data.

[craigds]

That looks close to perfect, thanks for the link :)