-
Notifications
You must be signed in to change notification settings - Fork 106
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
A bit confused about the Gram
implementation
#173
Comments
It's indeed confusing. Like you said we are getting >50% AUROC for GRAM in all our experiments. Applying the negation seems the correct thing to do, but will lead to <50% AUROC. I will try to do some investigation when I'm available. |
Hi, I worked on the gram matrix method and I seem to have fixed the implementation here The results on CIFAR10 obtained with the current gram-matrix implementation are as follows:
With the corrected implementation, I was able to get:
When I used the same checkpoints with the code referred to by @SauceCat , I was able to get marginally higher results for SVHN but did not test other datasets. The new code is not fully polished but seems to be working as expected. I also did not run experiments on datasets other CIFAR10 as InD. Thank you for the OpenOOD benchmark and considering the gram matrix method for inclusion in the benchmark! |
@chandramouli-sastry Thanks for sharing the results, and glad to see the much improved numbers with the updated implementation. Would you mind opening a pull request for this? Meanwhile we will update the gram matrix results in both the paper and the leaderboard. |
Thank you! I just created a pull request for your review. |
https://github.com/Jingkang50/OpenOOD/blob/main/openood/postprocessors/gram_postprocessor.py#L115
I wonder why the
dev
is used asconf
directly? Isn't the larger the deviations the more likely the sample would be an OOD?I checked the original implementation here: https://github.com/VectorInstitute/gram-ood-detection/blob/master/ResNet_Cifar10.ipynb
I found it actually using the negative of deviations when calculating the metrics.
But the metric looks ok, so I am quite confused. Am I missing sth?
The text was updated successfully, but these errors were encountered: