Volume-based QC check after segmentation

[af-a]

Description

This adds the existing QC check from quick_qc.py in run_prediction.py to evaluate segmentations, particularly to halt processing of cases that fail the test.

If a single subject is processed, the inference script exits with an error if the segmentation fails the QC check. This is propagated to run_fastsurfer.sh, which will also fail before running recon-surf (unless surf_only).

If multiple subjects are processed, only a warning message is logged for every failing subject, in addition to a count of failures at the end.

The list of failed subjects is maintained. This can be used to write a qc.txt for every subject, indicating that this case failed this QC check (and possibly others to be added in the future), or alternatively a list of all failed cases.

An open question is where qc.txt could be located.

(@m-reuter)

[m-reuter]

I discussed with @LeHenschel that probably we should create directories for the outputs and write the logfile there.

[af-a]

We could:

1. If batch processing (a Freesurfer directory structure is assumed), add $SUBJECT_DIR/scripts/qc.txt.
2. If single processing (a Freesurfer directory structure is not guaranteed), create $SEG_PATH/scripts/qc.txt OR always output a pre-defined directory structure (for e.g., Freesurfer's) in run_prediction.py and save to a pre-defined location ($SUBJECT_DIR/scripts/qc.txt).

In ether case, the sub-directory could be something other than scripts/.

[LeHenschel]

If I understood it correctly, running single input stops the process anyway, and there is no output written, if the qc-check passes, right? Do we need a separate log-file in this case? We could add the failure message output to the log-file that is anyway generated if the user specifies it or print it to stdout.

For multi-subject, we anyhow have the scripts directory and can either add a log-file there or again add the output to deep-seg.log (because the segmentation failed = related to the network/input data)?

The list of failed subjects is maintained. This can be used to write a qc.txt for every subject, indicating that this case failed this QC check (and possibly others to be added in the future), or alternatively a list of all failed cases.

For this I would honestly create a log-file in a user specified location and just return the list with all failed subjects. This seems more useful than having a qc.txt in each failed case which just contains "failed"? Unless there is some more subject-specific information explaining what failed.

[LeHenschel]

Implementation looks ok to me. Do you have test cases somewhere to run the code on?

i think the multi-processing might benefit from writing the failed cases to a list directly instead of just appending it to a list. This will be lost in case the run_prediction.py script fails on a subject (for other reasons than the volume check).

[af-a]

If I understood it correctly, running single input stops the process anyway, and there is no output written, if the qc-check passes, right?

At the moment, for single input, if the seg fails the QC check, the process is not stopped before saving outputs; it just returns an error code once it's done and logs a warning message.

For this I would honestly create a log-file in a user specified location and just return the list with all failed subjects. This seems more useful than having a qc.txt in each failed case which just contains "failed"? Unless there is some more subject-specific information explaining what failed.

I think the idea is that qc.txt will show failed/success for a list of future qc checks.

Implementation looks ok to me. Do you have test cases somewhere to run the code on?

There are some test cases in /groups/ag-reuter/projects/ahmed/test_set_la5c_adni.

i think the multi-processing might benefit from writing the failed cases to a list directly instead of just appending it to a list.

Noted. It would be better to write directly to file for a subject-list-style qc.txt.

---

Ok, so the current suggestion is:

1. for single processing: no log file.
2. for batch processing: i) if we want to always create a file listing QC results from now on, a log file in each $SUBJECT/scripts/; ii) if we don't want that, a single log file just listing failed subjects (location specified through an argument).

[af-a]

Implemented as described above, with a single, optional failed subject-list file in the batch processing case.