-
Notifications
You must be signed in to change notification settings - Fork 106
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CombOOD results for OpenOOD Leadboard #235
Comments
@zjysteven Hi Jingyang, Please help check this request. If you need some csv files (with easy-paste values) from the authors, we can leave the comments here. |
Hi @rmagesh148 thank you for reaching out. For the leaderboard, would you mind sharing us direct csv files containing your results for easy integration? You can find examples here #232, #207 (comment), #193 (comment). Also feel free to open a PR if you want to integrate your method into OpenOOD. |
Hi @zjysteven : Please check this link for the results. https://drive.google.com/drive/folders/1Ec0Iz7-i87RS36uHar6j7otDpKBr113I?usp=drive_link , let me know if you have any questions. Thanks! |
Hi @rmagesh148 it seems that the results are obtained on v1.0 benchmark while the leaderboard assumes that all methods are compared using v1.5 updated benchmark. For example for ImageNet we have refrained from using MNIST as OOD dataset (see more changes at https://github.com/Jingkang50/OpenOOD/wiki/OpenOOD-v1.5-change-log). Is it possible to rerun your methods with v1.5 benchmark? I imagine this wouldn't require much extra efforts (since CombOOD is post-hoc, right?), and you can find example of easily doing this in this colab notebook. |
@zjysteven : okay, sure! I will rerun for Imagenet and will let you know the results quickly but can you please update the results for other datasets ? |
There are also some slight changes to the CIFAR benchmarks too in v1.5, so it would be best to rerun on CIFAR10/100 :) |
okay, will be back with the results soon! Thank you for the prompt response. |
Hello I am Saiful, co -author of the CombOOD paper.
|
Hi Saiful @MdSaifulIslamSajol,
|
Hello, we extracted the 200 classes from imagnet-r dataset (https://github.com/hendrycks/imagenet-r). Then used only those 200 classes to call the imagnet-200 images like you did here . However, we are getting some discrepancy while checking the test accuracy with your given checkpoint. The test accuracy is very low. We would like to verify our version of train_imagenet200.txt file matches with your train_imagenet200.txt file. Can you please provide us your [train_imagenet200.txt ] (
|
Hi @MdSaifulIslamSajol if you download our |
Hi @zjysteven : Could you please find the results in the drive and let me know if you need anything else from us? Thank you! |
Hey Hi!
We would like to submit our CombOOD results to the OpenOOD Leaderboard.
Here's the link to my paper (https://epubs.siam.org/doi/abs/10.1137/1.9781611978032.74),
and the GitHub repository (https://github.com/rmagesh148/combood/tree/main).
Please let me know if you want me to open a PR for the same. Thank you!
The text was updated successfully, but these errors were encountered: