eager

Introduction

This example is used to demonstrate 3D-Unet int8 accuracy by tuning with Neural Compressor on PyTorch FBGEMM path.

The 3D-Unet source code comes from mlperf, commit SHA is b7e8f0da170a421161410d18e5d2a05d75d6bccf. nnUnet commit SHA is b38c69b345b2f60cd0d053039669e8f988b0c0af. User could diff them with this example to know which changes are made to integrate with Neural Compressor.

The model is performing BraTS 2019 brain tumor segmentation task.

Prerequisites

note

PyTorch 1.6.0 and above version has bug on layernorm int8 implementation, which causes 3D-Unet int8 model get ~0.0 accuracy. so this example takes PyTorch 1.5.0 as requirements.

  # install PyTorch 1.5.0+cpu
  pip install torch==1.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

  # download BraTS 2019 from https://www.med.upenn.edu/cbica/brats2019/data.html
  export DOWNLOAD_DATA_DIR=<path/to/MICCAI_BraTS_2019_Data_Training> # point to location of downloaded BraTS 2019 Training dataset.

  # install dependency required by data preprocessing script
  cd ./nnUnet
  python setup.py install
  cd ..

  # download pytorch model
  make download_pytorch_model

  # generate preprocessed data
  make preprocess_data

  # create postprocess dir
  make mkdir_postprocessed_data

  # generate calibration preprocessed data
  python preprocess.py --preprocessed_data_dir=./build/calib_preprocess/ --validation_fold_file=./brats_cal_images_list.txt

  # install mlperf loadgen required by tuning script
  git clone https://github.com/mlcommons/inference.git --recursive
  cd inference
  git checkout b7e8f0da170a421161410d18e5d2a05d75d6bccf
  cd loadgen
  pip install absl-py
  python setup.py install
  cd ..

running cmd

  make run_pytorch_NC_tuning
  
  or

  python run.py --model_dir=build/result/nnUNet/3d_fullres/Task043_BraTS2019/nnUNetTrainerV2__nnUNetPlansv2.mlperf.1 --backend=pytorch --accuracy --preprocessed_data_dir=build/preprocessed_data/ --mlperf_conf=./mlperf.conf --tune

Model Baseline

model	framework	accuracy	dataset	model link	model source	precision	notes
3D-Unet	PyTorch	mean = 0.85300 (whole tumor = 0.9141, tumor core = 0.8679, enhancing tumor = 0.7770)	Fold 1 of BraTS 2019 Training Dataset	from zenodo	Trained in PyTorch using codes fromnnUnet on Fold 0, Fold 2, Fold 3, and Fold 4 of BraTS 2019 Training Dataset.	fp32