Approximate PR/ROC AUC Metics On Large Data During Training

[EricZimmermann]

Hello,

During my work, I've encountered issues when aiming to compute PR AUC on large MRI volumes during training without impacting train time. To account for this, I've written some code to approximate these curves quickly on the gpu as a Ignite Metric. I assume I can estimate the upper and lower bounds of my models output and use these estimates to discretize the integral over predefined thresholds.

Is this code worth contributing?

[wyli]

would be nice to have a pull request for discussion. reminds me of this code from @yiheng-wang-nv https://github.com/Project-MONAI/tutorials/blob/77382cdb36fd9b99a693e2588a769fbc4f69cb06/kaggle/RANZCR/4th_place_solution/utils.py#L104-L116

[EricZimmermann]

Sorry for the delayed response, I've been quite busy and making a PR will take some time.
I wanted to highlight that what I'd be contributing would be an approximate metric that really only shines for very large data (mostly brains) and would be suited for people wanting to remove memory bottlenecks when doing segmentation and optimizing their model across a variety of operating points. Would it be possible to post or send a code snippet before going through the whole PR process? Thank you so much for your time!

[vfdev-5]

@EricZimmermann sure, no worries about the time. Yes, we can start llke that or maybe just send a draft PR to iterate over later.

[EricZimmermann]

I can make a public colab and share the link if that would be okay

[vfdev-5]

Sounds good 👍

[EricZimmermann]

Here it is!
Let me know what you think! It seems to have been useful in our lab thus far.

https://colab.research.google.com/drive/17Xo80swTxcaZsZOKS08GI-LHYHkg5kud?usp=sharing