FileNotFoundError When Running HoVer-Net on WSI

[tjbencomo]

• TIA Toolbox version: 1.5.1
• Python version: 3.10.14
• Operating System: Ubuntu 18.04.6 LTS (Bionic Beaver)

Description

I tried running HoVer-Net withNucleusInstanceSegmentor on a WSI image, but after running for 12 hours with a GPU and showing progress I encounter a FileNotFoundError when it tries to save the output file.

While the code was running, I see that the output directory I specified (/hpc/temp/setty_m/tbencomo/morphology/tiatoolbox/test-dir2) was created and exists, but when it crashes and I check the files the directory no longer exists - somehow the output directory is getting deleted. I think this is causing the FileNotFoundError error because the output directory does not exist.

The example code and dataset from the documentation works without issue. My WSI file is significantly larger (18MB for your example data sample_wsi.svs vs my file which is 1.2GB). I am running a RTX 2080ti.

What I Did

import logging
import warnings
if logging.getLogger().hasHandlers():
    logging.getLogger().handlers.clear()
import cv2
import joblib
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
from tiatoolbox import logger
from tiatoolbox.models.engine.nucleus_instance_segmentor import NucleusInstanceSegmentor
from tiatoolbox.utils.misc import download_data, imread

from tiatoolbox.utils.visualization import overlay_prediction_contours
from tiatoolbox.wsicore.wsireader import WSIReader

warnings.filterwarnings("ignore")
mpl.rcParams["figure.dpi"] = 300  # for high resolution figure in notebook
mpl.rcParams["figure.facecolor"] = "white"  # To make sure text is visible in dark mode
plt.rcParams.update({"font.size": 5})

inst_segmentor = NucleusInstanceSegmentor(
    pretrained_model="hovernet_fast-pannuke",
    num_loader_workers=0,
    num_postproc_workers=0,
    batch_size=8,
    auto_generate_mask=True,
    verbose=False,
)

wsi_file2 = '/fh/fast/setty_m/nghiem_setty_collaboration/incyte-xenium/he-images/XE015/0022273.svs'

# WSI prediction
# if ON_GPU=False, this part will take more than a couple of hours to process.
wsi_output = inst_segmentor.predict(
    [wsi_file2],
    masks=None,
    save_dir="/hpc/temp/setty_m/tbencomo/morphology/tiatoolbox/test-dir2/",
    mode="wsi",
    on_gpu=True,
    crash_on_exception=True,
)

Here is the log:

|2024-08-28|11:26:41.625| [WARNING] Read: Scale > 1.This means that the desired resolution is higher than the WSI baseline (maximum encoded res
olution). Interpolation of read regions may occur.
^MProcess Batch:   0%|                                      | 0/1 [00:00<?, ?it/s]^MProcess Batch: 100%|##############################| 1/1 [00
:00<00:00,  1.74it/s]^MProcess Batch: 100%|##############################| 1/1 [00:00<00:00,  1.73it/s]
^MProcess Batch:   0%|                                      | 0/1 [00:00<?, ?it/s]^MProcess Batch: 100%|##############################| 1/1 [00
:00<00:00,  6.86it/s]^MProcess Batch: 100%|##############################| 1/1 [00:00<00:00,  6.85it/s]
^MProcess Batch:   0%|                                      | 0/1 [00:00<?, ?it/s]^MProcess Batch: 100%|##############################| 1/1 [00
:00<00:00, 11.81it/s]
^MProcess Batch:   0%|                                      | 0/1 [00:00<?, ?it/s]^MProcess Batch: 100%|##############################| 1/1 [00
:00<00:00, 11.41it/s]
^MProcess Batch:   0%|                                      | 0/1 [00:00<?, ?it/s]^MProcess Batch: 100%|##############################| 1/1 [00
:00<00:00, 10.75it/s]
^MProcess Batch:   0%|                                      | 0/1 [00:00<?, ?it/s]^MProcess Batch: 100%|##############################| 1/1 [00
:00<00:00,  8.86it/s]^MProcess Batch: 100%|##############################| 1/1 [00:00<00:00,  8.84it/s]
^MProcess Batch:   0%|                                      | 0/2 [00:00<?, ?it/s]^MProcess Batch:  50%|###############               | 1/2 [00
:00<00:00,  1.84it/s]^MProcess Batch: 100%|##############################| 2/2 [00:00<00:00,  2.13it/s]^MProcess Batch: 100%|##################
############| 2/2 [00:00<00:00,  2.08it/s]
^MProcess Batch:   0%|                                      | 0/1 [00:00<?, ?it/s]^MProcess Batch: 100%|##############################| 1/1 [00
:00<00:00,  2.38it/s]^MProcess Batch: 100%|##############################| 1/1 [00:00<00:00,  2.38it/s]
^MProcess Batch:   0%|                                      | 0/1 [00:00<?, ?it/s]^MProcess Batch: 100%|##############################| 1/1 [00
:00<00:00, 10.78it/s]
^MProcess Batch:   0%|                                      | 0/1 [00:00<?, ?it/s]^MProcess Batch: 100%|##############################| 1/1 [00
:00<00:00,  4.66it/s]^MProcess Batch: 100%|##############################| 1/1 [00:00<00:00,  4.65it/s]
^MProcess Batch:   0%|                                      | 0/2 [00:00<?, ?it/s]^MProcess Batch:  50%|###############               | 1/2 [00
:00<00:00,  2.43it/s]^MProcess Batch: 100%|##############################| 2/2 [00:00<00:00,  3.29it/s]^MProcess Batch: 100%|##################
############| 2/2 [00:00<00:00,  3.12it/s]
^MProcess Batch:   0%|                                      | 0/1 [00:00<?, ?it/s]^MProcess Batch: 100%|##############################| 1/1 [00
:00<00:00,  8.36it/s]^MProcess Batch: 100%|##############################| 1/1 [00:00<00:00,  8.34it/s]
^MProcess Batch:   0%|                                      | 0/1 [00:00<?, ?it/s]^MProcess Batch: 100%|##############################| 1/1 [00
:00<00:00, 11.47it/s]
^MProcess Batch:   0%|                                      | 0/1 [00:00<?, ?it/s]^MProcess Batch: 100%|##############################| 1/1 [00
:00<00:00, 11.96it/s]
^MProcess Batch:   0%|                                      | 0/1 [00:00<?, ?it/s]^MProcess Batch: 100%|##############################| 1/1 [00
:00<00:00, 16.63it/s]
^MProcess Batch:   0%|                                      | 0/1 [00:00<?, ?it/s]^MProcess Batch: 100%|##############################| 1/1 [00
:00<00:00, 11.22it/s]
^MProcess Batch:   0%|                                      | 0/1 [00:00<?, ?it/s]^MProcess Batch: 100%|##############################| 1/1 [00
:00<00:00, 11.88it/s]
[More batch process output]
Traceback (most recent call last):
  File "/hpc/temp/setty_m/tbencomo/morphology/tiatoolbox/test_wsi.py", line 68, in <module>
    wsi_output = inst_segmentor.predict(
  File "/home/tbencomo/micromamba/envs/tiatoolbox/lib/python3.10/site-packages/tiatoolbox/models/engine/semantic_segmentor.py", line 1399, in predict
    self._predict_wsi_handle_exception(
  File "/home/tbencomo/micromamba/envs/tiatoolbox/lib/python3.10/site-packages/tiatoolbox/models/engine/semantic_segmentor.py", line 1249, in _predict_wsi_handle_exception
    raise err  # noqa: TRY201
  File "/home/tbencomo/micromamba/envs/tiatoolbox/lib/python3.10/site-packages/tiatoolbox/models/engine/semantic_segmentor.py", line 1225, in _predict_wsi_handle_exception
    self._predict_one_wsi(wsi_idx, ioconfig, str(wsi_save_path), mode)
  File "/home/tbencomo/micromamba/envs/tiatoolbox/lib/python3.10/site-packages/tiatoolbox/models/engine/nucleus_instance_segmentor.py", line 754, in _predict_one_wsi
    joblib.dump(self._wsi_inst_info, f"{save_path}.dat")
  File "/home/tbencomo/micromamba/envs/tiatoolbox/lib/python3.10/site-packages/joblib/numpy_pickle.py", line 552, in dump
    with open(filename, 'wb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/hpc/temp/setty_m/tbencomo/morphology/tiatoolbox/test-dir2/0.dat'

[shaneahmed]

Would it be possible to share a sample file?

[tjbencomo]

Yeah definitely - do you have any email I can share it with via GoogleDrive?

[shaneahmed]

You can share it with tia@dcs.warwick.ac.uk.

[shaneahmed]

Please reopen this issue if you still have this problem.