slow TiffWriter performance

[ebremer]

with 6.3.0 (I think it's the same with earlier versions), TiffWriter performance is slow. I'm writing individual tiles out, with LZW compression. The amount of data is small, hardly and IO activity. The final file is perhaps 200MB in size but it takes nearly an hour. Now, my computer is fast 8 cores, 3800X 8-cores, 64GB RAM and a Corsair MP600 M.2 disk. I've notice this issue for some time but wanted to ask about it, knowing a little bit more about TIF file structures now, I'm not entirely sure why the tile TiffWriter performance is so slow.

[sbesson]

@ebremer there have been several public reports about conversion performance. Over the last months, we have started reviewing various bottlenecks in the writing API for some specific scenarios. Ultimately some large API changes will probably be necessary to ensure TiffWriter is I/O bound in most cases. Until then, two options might be worth trying out:

• TiffSaver: make tile writing faster #3465 will be available in the upcoming Bio-Formats 6.3.1 and aims at improving the writing of tiled TIFF files
• https://forum.image.sc/t/converting-whole-slide-images-to-ome-tiff-a-new-workflow/32110 introduces a set of publicly available tools developed specifically for the conversion of large WSI data from proprietary formats into OME-TIFF in high-performance computing environments.

Interested to hear your feedback and whether either of these options makes a difference in your use case.

[tischi]

Hi @sbesson and @ebremer,
My experience is that this still is very slow. See also here: https://forum.image.sc/t/bioformats-saving-with-compression-is-too-slow/20522/11
But it looks like @dgault adopted the issue?

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/bioformats-saving-with-compression-is-too-slow/20522/13

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/pyramidometiffwriter-really-bad-performance-when-closing-the-writer/95294/9