mist_train failing on last fold: IndexError: single positional indexer is out-of-bounds

[fuentesdt]

kirby$ pwd
/rsrch3/ip/dtfuentes/github/oncopigdata
kirby$ /rsrch1/ip/dtfuentes/minicondamist/bin/mist_train --data /rsrch3/ip/dtfuentes/github/oncopigdata/misttrain/dataset.json --numpy /rsrch3/ip/dtfuentes/github/oncopigdata/misttrainnumpy --results /rsrch3/ip/dtfuentes/github/oncopigdata/misttrainresults --gpus 1 --amp --pocket --epochs 100
.
.
.
.

Fold 4: Epoch 98/100 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8/8 • 0:00:06 • loss: 0.1460
Validating ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/2 • 0:00:10 • val_loss: 0.2040
Validation loss did NOT improve from 0.1937

Fold 4: Epoch 99/100 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8/8 • 0:00:06 • loss: 0.1346
Validating ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/2 • 0:00:10 • val_loss: 0.1886
Validation loss IMPROVED from 0.1937 to 0.1886

Fold 4: Epoch 100/100 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8/8 • 0:00:06 • loss: 0.1422
Validating ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/2 • 0:00:10 • val_loss: 0.2077
Validation loss did NOT improve from 0.1886

Testing on fold ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4/4 • 0:00:55

Evaluating ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/36 • 0:00:16
Traceback (most recent call last):
File "/rsrch1/ip/dtfuentes/minicondamist/bin/mist_train", line 33, in
sys.exit(load_entry_point('mist-medical', 'console_scripts', 'mist_train')())
File "/rsrch1/ip/dtfuentes/github/MIST/mist/scripts/train_entrypoint.py", line 28, in train_entry
main(args)
File "/rsrch1/ip/dtfuentes/github/MIST/mist/main.py", line 62, in main
evaluate(args.data,
File "/rsrch1/ip/dtfuentes/github/MIST/mist/evaluate_preds/evaluate.py", line 135, in evaluate
truth = paths.loc[paths['id'].astype(str) == patient_id].iloc[0]['mask']
File "/rsrch1/ip/dtfuentes/minicondamist/lib/python3.8/site-packages/pandas/core/indexing.py", line 1103, in getitem
return self._getitem_axis(maybe_callable, axis=axis)
File "/rsrch1/ip/dtfuentes/minicondamist/lib/python3.8/site-packages/pandas/core/indexing.py", line 1656, in _getitem_axis
self._validate_integer(key, axis)
File "/rsrch1/ip/dtfuentes/minicondamist/lib/python3.8/site-packages/pandas/core/indexing.py", line 1589, in _validate_integer
raise IndexError("single positional indexer is out-of-bounds")
IndexError: single positional indexer is out-of-bounds

[aecelaya]

Looks like you're using an earlier version. Try upgrading to v0.4.15. It has more error handling for the evaluation and prediction.

Also, try running ls -a in your predictions folder. The error is telling you that it can't find a certain file. Usually, that means that there might be a hidden file that made its way into the predictions folder, like a .DS_Store of .ipynb_checkpoints. Either way, the latest version can handle it and will let you know which files fail with prediction or evaluation.