Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Total segmentation volume is too small. Segmentation may be corrupted. #256

Closed
araikes opened this issue Feb 15, 2023 · 20 comments
Closed
Labels
question Further information is requested

Comments

@araikes
Copy link

araikes commented Feb 15, 2023

Question/Support Request

I have a number of individuals for whom I received the error message in the subject line. I'm trying to debug what went awry. When I overlay aseg.auto_noCCseg on orig.mgz there doesn't appear to be any obvious mis-segmentation (see example below). Is there a concise way for me to debug where these errors are coming from?

Noteworthy (maybe): I am running this on individuals with Alzheimer's. I wouldn't think this would affect the total segmentation, but perhaps.

Screenshots

image

Environment

  • FastSurfer Version: 2.0.1
  • OS: Official Singularity image

Execution

singularity pull docker://deepmi/fastsurfer:2.0.1
singularity exec --nv --cleanenv -B /xdisk/adamraikes/xxx/nifti:/input -B /xdisk/adamraikes/xxx/derivatives/fastsurfer-2.0.1:/output -B /groups/adamraikes/license.txt:/license.txt /groups/adamraikes/singularity_images/fastsurfer-2.0.1.sif /fastsurfer/run_fastsurfer.sh --fs_license /license.txt --t1 /input/sub-13452019/ses-10/anat/sub-13452019_ses-10_T1w.nii.gz --sid 13452019_ses-10 --sd /output --parallel

@araikes araikes added the question Further information is requested label Feb 15, 2023
@m-reuter
Copy link
Member

Thanks for reporting. Indeed the segmentation looks ok ( on my phone). What volume number are you getting? Maybe our threshold is too conservative.

@araikes
Copy link
Author

araikes commented Feb 15, 2023

This is the stdout from my SLURM job in which this individual ran. I checked deep_seg.log but it doesn't have the segmentation volume.

python3.8 /fastsurfer/FastSurferCNN/run_prediction.py --t1 /input/sub-13452019/ses-10/anat/sub-13452019_ses-10_T1w.nii.gz --aparc_aseg_segfile /output/13452019_ses-10/mri/aparc.DKTatlas+aseg.deep.mgz --conformed_name /output/13452019_ses-10/mri/orig.mgz --sid 13452019_ses-10 --seg_log /output/13452019_ses-10/scripts/deep-seg.log --vox_size min --batch_size 1 --viewagg_device auto --device auto
[INFO: run_prediction.py:  341]: Checking or downloading default checkpoints ...
[INFO: run_prediction.py:  158]: Output will be stored in: /output
[INFO: misc.py:  159]: Using device: cpu
[INFO: run_prediction.py:  121]: Running view aggregation on cpu
[INFO: inference.py:   95]: Loading checkpoint /fastsurfer/FastSurferCNN/checkpoints/aparc_vinn_coronal_v2.0.0.pkl
[INFO: inference.py:   95]: Loading checkpoint /fastsurfer/FastSurferCNN/checkpoints/aparc_vinn_sagittal_v2.0.0.pkl
[INFO: inference.py:   95]: Loading checkpoint /fastsurfer/FastSurferCNN/checkpoints/aparc_vinn_axial_v2.0.0.pkl
[INFO: run_prediction.py:  359]: Analyzing single subject /input/sub-13452019/ses-10/anat/sub-13452019_ses-10_T1w.nii.gz
[INFO: run_prediction.py:  247]: Successfully saved image as /output/13452019_ses-10/mri/orig/001.mgz
[INFO: run_prediction.py:  176]: Conforming image
Input:    min: 0.0  max: 166513.76358032227
rescale:  min: 0.0  max: 89084.8635154724  scale: 0.002862439138784909
Output:   min: 0.0  max: 255.0
[INFO: run_prediction.py:  247]: Successfully saved image as /output/13452019_ses-10/mri/orig.mgz
[INFO: run_prediction.py:  210]: Run coronal prediction
[INFO: dataset.py:   55]: Loading Coronal with input voxelsize (1.0, 1.0)
[INFO: dataset.py:   64]: Successfully loaded Image from /input/sub-13452019/ses-10/anat/sub-13452019_ses-10_T1w.nii.gz
100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 256/256 [02:03<00:00,  2.08batch/s]
[INFO: inference.py:  198]: Inference on 256 batches for coronal successful
[INFO: inference.py:  217]: Coronal inference on /input/sub-13452019/ses-10/anat/sub-13452019_ses-10_T1w.nii.gz finished in 123.0854 seconds
[INFO: run_prediction.py:  210]: Run sagittal prediction
[INFO: dataset.py:   46]: Loading Sagittal with input voxelsize (1.0, 1.0)
[INFO: dataset.py:   64]: Successfully loaded Image from /input/sub-13452019/ses-10/anat/sub-13452019_ses-10_T1w.nii.gz
100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 256/256 [01:58<00:00,  2.17batch/s]
[INFO: inference.py:  198]: Inference on 256 batches for sagittal successful
[INFO: inference.py:  217]: Sagittal inference on /input/sub-13452019/ses-10/anat/sub-13452019_ses-10_T1w.nii.gz finished in 118.1760 seconds
[INFO: run_prediction.py:  210]: Run axial prediction
[INFO: dataset.py:   51]: Loading Axial with input voxelsize (1.0, 1.0)
[INFO: dataset.py:   64]: Successfully loaded Image from /input/sub-13452019/ses-10/anat/sub-13452019_ses-10_T1w.nii.gz
100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 256/256 [01:57<00:00,  2.18batch/s]
[INFO: inference.py:  198]: Inference on 256 batches for axial successful
[INFO: inference.py:  217]: Axial inference on /input/sub-13452019/ses-10/anat/sub-13452019_ses-10_T1w.nii.gz finished in 117.5716 seconds
[INFO: run_prediction.py:  247]: Successfully saved image as /output/13452019_ses-10/mri/aparc.DKTatlas+aseg.deep.mgz
[INFO: run_prediction.py:  383]: Creating brainmask based on segmentation...
Creating dilated mask ...
Frontal region special treatment:  12766
  Found 1 connected component(s)!
[INFO: run_prediction.py:  247]: Successfully saved image as /output/13452019_ses-10/mri/mask.mgz
[INFO: run_prediction.py:  389]: Creating aseg based on segmentation...
Reducing to aseg ...
FlipWM: rh 0 and lh 0 flipped.
[INFO: run_prediction.py:  247]: Successfully saved image as /output/13452019_ses-10/mri/aseg.auto_noCCseg.mgz
[INFO: run_prediction.py:  397]: Running volume-based QC check on segmentation...
Checking total volume ...
Voxel size in mm3: 1.0
Total segmentation volume in liter: 0.77
[WARNING: run_prediction.py:  400]: Total segmentation volume is too small. Segmentation may be corrupted.
[ERROR: run_prediction.py:  415]: Single subject failed the volume-based QC check.
ERROR: Segmentation failed QC checks.

@m-reuter
Copy link
Member

Thanks, the volume is 0.77 liter which is also our threshold , in our tests we did not encounter such small volumes except if the segmentation was broken. In your case the segmentation looks good, so the threshold can be decreased.

def check_volume(aparc_aseg_segfile, voxvol, thres=0.77):

I reduced it in dev to 0.75 , so one option is to checkout dev and build a docker/singularity image from that (note that there are some other commits in dev and it has not been thoroughly tested), or create a new image from your stable image after entering the image and manually changing that value in quick_qc.py . I wish we had a flag to ignore those errors, but currently we don't.

@araikes
Copy link
Author

araikes commented Feb 15, 2023

Thanks for the quick edit. If my call is exactly that and that's all I'm planning at the moment, is there anything in the dev branch that would be substantively different from 2.0.1 or would I need to re-run all of my participants?

@dkuegler
Copy link
Member

I think you should just be able to run --surf_only (just add that flag to your call) to create the surfaces. (Nothing of note really happens between the error that you encountered and the surface generation).

no dev or modifications needed, just use stable / your existing image.

@araikes
Copy link
Author

araikes commented Feb 15, 2023

@dkuegler,

I'll give it a shot and see if it work. Thanks.

@araikes
Copy link
Author

araikes commented Feb 15, 2023

Using --surf_only does not bypass the segmentation QC check.

@araikes
Copy link
Author

araikes commented Feb 15, 2023

@m-reuter,

Just another use case in this dataset. This individual has a total volume of 0.72.

image

@m-reuter
Copy link
Member

Are these adults? I guess we need to go even lower or switch off the test and come up with something different.

@m-reuter
Copy link
Member

Just switched from ERROR to a WARNING in dev. You can checkout stable, copy quick_qc.py over from dev and re-build the containers. We may also push a hot fix release at some point.

@m-reuter
Copy link
Member

I created a stable patch release v2.0.2 that gives a WARNING and continues. We will probably update docker images today or tomorrow.

In the future we need to:

  • find a better way to check for segmentation errors (maybe check cortical GM volume in relation to total volume)
  • quick_qc.py should not be called in multiple places (currently run_prediction and recon-surf)
  • users should be able to ignore errors via a flag and restart processing from that location
  • run_prediction is very flexible and quick_qc does not make sense when other segmentations are performed (it is designed only for full brain analysis). So it should be removed there and only run if the full brain module is run.

@araikes
Copy link
Author

araikes commented Feb 16, 2023

Thanks @m-reuter. I appreciate the quick fix.

@araikes
Copy link
Author

araikes commented Feb 24, 2023

@m-reuter,

I pulled v. 2.0.4's Docker image and I still get an error and termination of the segmentation when the volume is <= 0.75.

@araikes
Copy link
Author

araikes commented Feb 24, 2023

It's here:

if len(s_dirs) == 1:
if qc_failed_subject_count:
LOGGER.error("Single subject failed the volume-based QC check.")
sys.exit(1)

@m-reuter
Copy link
Member

Thanks for testing. I had fixed quick_qc to print a warning and not exit with error, but it turns out quick_qc is not actually called. Only a function from that file is called and here it is still taken as an error. This means we need to do another hot fix. Sorry for that.

@m-reuter
Copy link
Member

By the way, you might be able to workaround it this time by doing the --surf_only , as that should bypass the first check.

Another question: the smallest ratio you had is .72 ?
And what age range are these cases from?

@araikes
Copy link
Author

araikes commented Feb 24, 2023

It's a patient population with Alzheimer's. All aged late 70s to late 90s.

@araikes
Copy link
Author

araikes commented Feb 24, 2023

Also, in the short term, locally I simply changed that sys.exit(1) to sys.exit(0) so it prints the warning and mounted that file in as run_prediction.py and it seems to be working without issue.

@dkuegler
Copy link
Member

@araikes It makes sense this would work for you.

One question, when you reran the subject with --surf_only, you said this did not work. It seems to me you reran on a clean subject folder. Can you confirm, you did NOT rerun on the "result" of the last (failed) run?

@m-reuter
Copy link
Member

should be solved as we now only generate warnings and reduced the threshold further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants