Meet "RuntimeError: dictionary changed size during iteration " in extract_composition_features() function

[xiachenrui]

• TIA Toolbox version: 1.4.1
• Python version: 3.11
• Operating System: Ubuntu 22.04

Description

I try to run tutorial at https://tia-toolbox.readthedocs.io/en/latest/_notebooks/jnb/inference-pipelines/slide-graph.html#cell-composition-extraction (I use my wsi file rather than your example file), but meet following error. By the way I can run nucleus instance segmentor tutorial successfully.

What I Did

################ func from tutorial ############
def get_cell_compositions(
    wsi_path: str,
    mask_path: str,
    inst_pred_path: str,
    save_dir: str,
    num_types: int = 6,
    patch_input_shape: tuple[int] = (512, 512),
    stride_shape: tuple[int] = (512, 512),
    resolution: float = 0.25,
    units: str = "mpp",
) -> None:
    r"""
    Estimates cellular composition.
    """
    reader = WSIReader.open(wsi_path)
    inst_pred = joblib.load(inst_pred_path)
    # Convert to {key: int, value: dict}
    inst_pred = {i: v for i, (_, v) in enumerate(inst_pred.items())}

    inst_boxes = [v["box"] for v in inst_pred.values()]
    inst_boxes = np.array(inst_boxes)

    geometries = [shapely_box(*bounds) for bounds in inst_boxes]
    spatial_indexer = STRtree(geometries)

    # * Generate patch coordinates (in xy format)
    wsi_shape = reader.slide_dimensions(resolution=resolution, units=units)

    (patch_inputs, _) = PatchExtractor.get_coordinates(
        image_shape=wsi_shape,
        patch_input_shape=patch_input_shape,
        patch_output_shape=patch_input_shape,
        stride_shape=stride_shape,
    )

    # Filter out coords which dont lie in mask
    selected_coord_indices = PatchExtractor.filter_coordinates(
        WSIReader.open(mask_path),
        patch_inputs,
        wsi_shape=wsi_shape,
        min_mask_ratio=0.5,
    )
    patch_inputs = patch_inputs[selected_coord_indices]

    bounds_compositions = []
    for bounds in patch_inputs:
        bounds_ = shapely_box(*bounds)
        indices = [
            geo
            for geo in spatial_indexer.query(bounds_)
            if bounds_.contains(geometries[geo])
        ]
        insts = [inst_pred[v]["type"] for v in indices]
        uids, freqs = np.unique(insts, return_counts=True)
        # A bound may not contain all types, hence, to sync
        # the array and placement across all types, we create
        # a holder then fill the count within.
        holder = np.zeros(num_types, dtype=np.int16)
        holder[uids.astype(int)] = freqs
        bounds_compositions.append(holder)
    bounds_compositions = np.array(bounds_compositions)

    base_name = Path(wsi_path).stem
    # Output in the same saving protocol for construct graph
    np.save(f"{save_dir}/{base_name}.position.npy", patch_inputs)
    np.save(f"{save_dir}/{base_name}.features.npy", bounds_compositions)


def extract_composition_features(
    wsi_paths: list[str],
    save_dir: str,
    preproc_func: Callable = None,
    msk_paths: list[str] = None,
) -> list:
    r"""
    """
    inst_segmentor = NucleusInstanceSegmentor(
        pretrained_model="hovernet_fast-pannuke",
        batch_size=16,
        num_postproc_workers=4,
        num_loader_workers=4,
        auto_generate_mask=True,
    )
    # bigger tile shape for postprocessing performance
    inst_segmentor.ioconfig.tile_shape = (4000, 4000)
    # Injecting customized preprocessing functions,
    # check the document or sample codes below for API
    inst_segmentor.model.preproc_func = preproc_func

    if Path(save_dir).is_dir():
        shutil.rmtree(save_dir)
    output_map_list = inst_segmentor.predict(
        wsi_paths,
        msk_paths,
        mode="wsi",
        on_gpu=torch.cuda.is_available(),
        crash_on_exception=True,
        save_dir=save_dir,
    )
    # Rename output files of toolbox
    output_paths = []
    for input_path, output_path in output_map_list:
        input_name = Path(input_path).stem

        output_parent_dir = Path(output_path).parent

        src_path = Path(f"{output_path}.dat")
        new_path = Path(f"{output_parent_dir}/{input_name}.dat")
        src_path.rename(new_path)
        output_paths.append(new_path)

    # TODO(TBC): Parallelize this if possible  # noqa: TD003, FIX002
    for idx, path in enumerate(output_paths):
        get_cell_compositions(wsi_paths[idx], msk_paths[idx], path, save_dir)
    return output_paths


################## run ###################################
wsi_paths = "./experiments/CMU-1.tif"
save_dir= 'tmp/test'
meta = extract_composition_features(wsi_paths, save_dir)

Error message

  File "conda/lib/python3.11/site-packages/tiatoolbox/models/engine/semantic_segmentor.py", line 1328, in predict                                                           │·························
    self._predict_wsi_handle_exception(                                                                                                                                                                      │·························
  File "/conda/lib/python3.11/site-packages/tiatoolbox/models/engine/semantic_segmentor.py", line 1180, in _predict_wsi_handle_exception                                     │·························
    raise err                                                                                                                                                                                                │·························
  File "conda/lib/python3.11/site-packages/tiatoolbox/models/engine/semantic_segmentor.py", line 1156, in _predict_wsi_handle_exception                                     │·························
    self._predict_one_wsi(wsi_idx, ioconfig, str(wsi_save_path), mode)                                                                                                                                       │·························
  File "conda/lib/python3.11/site-packages/tiatoolbox/models/engine/nucleus_instance_segmentor.py", line 737, in _predict_one_wsi                                           │·························
    self._merge_post_process_results()                                                                                                                                                                       │·························
  File "conda/lib/python3.11/site-packages/tiatoolbox/models/engine/nucleus_instance_segmentor.py", line 784, in _merge_post_process_results                                │·························
    raise future.exception()                                                                                                                                                                                 │·························
  File "conda/lib/python3.11/multiprocessing/queues.py", line 244, in _feed                                                                                                 │·························
    obj = _ForkingPickler.dumps(obj)                                                                                                                                                                         │·························
          ^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                         │·························
  File "conda/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps                                                                                               │·························
    cls(buf, protocol).dump(obj) 

[shaneahmed]

It seems to be similar to #605 @adamshephard please can you have a look at this?

[adamshephard]

Hi @xiachenrui,
Thanks for reaching out with this. I have thus far been unable to replicate this issue. Did this error arise on colab or on your local machine? Can you possibly send the full error log so that I can determine where the issue is coming from? I can't see where the dictionary changing size error arises in the current error message you have posted. Thanks for your help.
Adam

[shaneahmed]

We were not able to reproduce this issue. Closing this issue. Please feel free to reopen the issue if required.