Based on Deep Ensemble, we use truth discovery within the probability simplex to find the truth vector, which is better than the naive mean ensemble, w.r.t uncertainty. Further, we use geometric optimization to ensure the accuracy is preserved provably, as shown as follows:
See "requirements.txt" for installation. Then download our logits from Google Drive and store them in "./pitfalldee/megacache". Note that due to the limited space in my cloud storage, only a small portion of logits are shared; however, you are free to use your own logits, and see pytorch-ensembles for a complete list of models/datasets used in our paper.
Run jupyter notebook "UN_all_cifar.ipynb" to reproduce the following results:
and
Modify "logits_file" to switch between different datasets/models.
If you find the software useful please consider citing:
@inproceedings{ma2020truth,
title={Improving Uncertainty Calibration of Deep Neural Networks via Truth Discovery and Geometric Optimization},
author={Ma, Chunwei and Huang, Ziyun and Xian, Jiayi and Gao, Mingchen and Xu, Jinhui},
booktitle={Uncertainty in Artificial Intelligence (UAI)},
pages={75-85},
year={2021},
organization={PMLR}
}
Acknowledgments
In this project we use (parts of) the official implementations of the following works:
We thank the respective authors for open-sourcing their methods, which makes this work possible.
If you have any problem please contact me.