Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Discrepancy between AUC score and ROC plot #165

Closed
SherlockMones opened this issue Jul 27, 2022 · 5 comments
Closed

Discrepancy between AUC score and ROC plot #165

SherlockMones opened this issue Jul 27, 2022 · 5 comments
Assignees
@SherlockMones
Labels
bug Something isn't working

Comments

@SherlockMones
Copy link
Collaborator

Testing landmark classification, that results in bad AUC scores (between 0.4 and 0.6), although ROC shows impeccable classification. F1 also amazing scores (~ 0.9). See screeenshots for details

image

image
@SherlockMones SherlockMones added the bug Something isn't working label Jul 27, 2022
@muellerdo
Copy link
Member

https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html

muellerdo added a commit that referenced this issue Jul 27, 2022
@muellerdo
fix(Evaluation): fix incorrect AUC computation - closes #165
976369a
@muellerdo
Copy link
Member

Before:
plot performance barplot

After:
plot performance barplot

@muellerdo
Copy link
Member

Issue was incorrect probability access if no threshold is provided (automatic). For some logic error I thought I had to pick the maximum softmax of all classes which is obviously incorrect...

Fixed it! 🎉

@muellerdo
Copy link
Member

PR started

@muellerdo muellerdo mentioned this issue Jul 27, 2022
Merged
muellerdo added a commit that referenced this issue Jul 27, 2022
@muellerdo
Merge pull request #168 from frankkramer-lab/fix_eval_auc
72dbbdf 
fix(Evaluation): fix incorrect AUC computation - closes #165
@muellerdo
Copy link
Member

Merged in dev branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@SherlockMones SherlockMones

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@muellerdo @SherlockMones