[Metrics] Add multiclass auroc #4236

ddrevicky · 2020-10-19T16:12:30Z

What does this PR do?

Implements functional multiclass AUROC.

Notes on the code:

Had to pass reorder=False for auc because tests against sklearn kept showing different values and debugging showed that the difference was actually coming out of our auc using torch.argsort which is unstable. A short colab notebook documenting this. I submitted a separate issue #4237 for that.

Before submitting

Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?
Did you verify new and existing tests pass locally with your changes?
If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

ddrevicky · 2020-10-19T16:36:22Z

pytorch_lightning/metrics/functional/classification.py

+    @multiclass_auc_decorator(reorder=False)
+    def _multiclass_auroc(pred, target, sample_weight, num_classes):
+        return multiclass_roc(pred, target, sample_weight, num_classes)
+
+    class_aurocs = _multiclass_auroc(pred=pred, target=target,
+                                     sample_weight=sample_weight,
+                                     num_classes=num_classes)
+    return torch.mean(class_aurocs)


I've implemented this using the multiclass_auc_decorator similarly as is done for auroc but have to say that as not such an experienced Pythonist I was scratching my head for a good while trying to figure out what the multiclass_auc_decorator was doing. It's possible that for other people reading the code it might also take unnecessary amount of time. Would the following be more readable? No decorator is needed either outside or within the multiclass_auroc function. Just my humble opinion, which do you guys prefer? :)

class_rocs = multiclass_roc(pred=pred, target=target, sample_weight=sample_weight, num_classes=num_classes) class_aurocs = [] for fpr, tpr, _ in class_rocs: class_aurocs.append(auc(fpr, tpr, reorder=False)) return torch.mean(torch.stack(class_aurocs))

codecov · 2020-10-19T16:38:27Z

Codecov Report

Merging #4236 into master will increase coverage by 3%.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master   #4236    +/-   ##
=======================================
+ Coverage      90%     93%    +3%     
=======================================
  Files         113     113            
  Lines        8232    8191    -41     
=======================================
+ Hits         7387    7612   +225     
+ Misses        845     579   -266

CHANGELOG.md

tests/metrics/functional/test_classification.py

teddykoker

LGTM

Borda

lgtm

pytorch_lightning/metrics/functional/classification.py

tchaton

Great PR ! Some extra tests on matrix size would be great !

tchaton · 2020-10-20T13:12:32Z

pytorch_lightning/metrics/functional/classification.py

+            "Multiclass AUROC metric expects the target scores to be"
+            " probabilities, i.e. they should sum up to 1.0 over classes")
+
+    if torch.unique(target).size(0) != pred.size(1):


Shouldn't it be torch.unique(target).size(0) <= pred.size(1)

target = torch.tensor([0, 0, 0, 0]) >>> torch.unique(target).size(0) 1

Or we could have a get_num_classes utils too.

Well the metric is undefined when torch.unique(target).size(0) < pred.size(1), that's why there is a strict equals. The way this implementation works (based on sklearn's version) is, that it uses a one-vs-rest strategy of computing the AUROC. For n classes it computes n binary AUROCs, for each class in turn, it is considered a positive class and all the other classes negative. Then it averages those.

E.g., for n=3 and target=[0, 1, 1, 2], for class 0 we binarize the target to make 0 the positive class: [1, 0, 0, 0] and compute the AUC of ROC of that.

If a target label is not present in the target, e.g. n=3 and target=[0, 1, 1, 1] then for the absent class 2 the binarized target would look like [0, 0, 0, 0] (all negative) and ROC cannot be computed (would raise an error). Consequently, the whole multiclass AUROC is undefined in that case.

As for get_num_classes there already is such a util, but it does something different than we need here. It doesn't look at dimensions of the predictions, just at the max value in both and deduced num classes from that (which when I think about it now, could fail silently for example when n_cls=5 but target=[0,1,2,3].

Make sense !

tchaton · 2020-10-20T13:13:09Z

pytorch_lightning/metrics/functional/classification.py

+        >>> target = torch.tensor([0, 1, 3, 2])
+        >>> multiclass_auroc(pred, target)   # doctest: +NORMALIZE_WHITESPACE
+        tensor(0.6667)
+    """


Also, we should check pred.size(0) == target.size(0)

I can add that of course but this is not a check that is done in any other metric implementation. So if it's done here it should probably be done everywhere. If that's desired, I could add a helper to classification.py

def check_batch_dims(pred, target): if not pred.size(0) == target.size(0): raise ValueError(f"Batch size for prediction ({pred.size(0)}) and target ({target.size(0)}) must be equal.")

Would that work? Then this helper could be used in each metric instead of copy pasting the if clause and the exception.

As we are slowly unifying the functional and class based interface, we are doing more checks for shape, so this will come in a future PR :]

tchaton · 2020-10-28T12:21:43Z

pytorch_lightning/metrics/functional/classification.py

+            "Multiclass AUROC metric expects the target scores to be"
+            " probabilities, i.e. they should sum up to 1.0 over classes")
+
+    if torch.unique(target).size(0) != pred.size(1):


Make sense !

SkafteNicki · 2020-10-30T11:46:04Z

CHANGELOG.md

@@ -106,6 +106,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).

 - Added trace functionality to the function `to_torchscript` ([#4142](https://github.com/PyTorchLightning/pytorch-lightning/pull/4142))

+- Added multiclass AUROC metric ([#4236](https://github.com/PyTorchLightning/pytorch-lightning/pull/4236))


@ddrevicky could you move this to the unreleased section?

Should be okay now.

* Add functional multiclass AUROC metric * Add multiclass_auroc tests * fixup! Add functional multiclass AUROC metric * fixup! fixup! Add functional multiclass AUROC metric * Add multiclass_auroc doc reference * Update CHANGELOG * formatting * Shorter error message regex match in tests * Set num classes as pytest parameter * formatting * Update CHANGELOG Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com> Co-authored-by: Nicki Skafte <skaftenicki@gmail.com> (cherry picked from commit 38bb4e2)

ddrevicky added 5 commits October 19, 2020 18:07

Add functional multiclass AUROC metric

983b795

Add multiclass_auroc tests

62137f4

fixup! Add functional multiclass AUROC metric

b3947e0

fixup! fixup! Add functional multiclass AUROC metric

70d4d0d

Add multiclass_auroc doc reference

c36cd9b

mergify bot requested a review from a team October 19, 2020 16:12

Update CHANGELOG

3bf01c7

ddrevicky commented Oct 19, 2020

View reviewed changes

ddrevicky changed the title ~~Feature/3304 multiclass auroc~~ [Metrics] Add multiclass auroc Oct 19, 2020

Borda added feature Is an improvement or enhancement Metrics labels Oct 19, 2020

Borda added this to the 1.0.x milestone Oct 19, 2020

Borda reviewed Oct 19, 2020

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

tests/metrics/functional/test_classification.py Outdated Show resolved Hide resolved

tests/metrics/functional/test_classification.py Outdated Show resolved Hide resolved

mergify bot requested a review from a team October 19, 2020 18:56

Borda requested review from justusschock, ananyahjha93, SkafteNicki and teddykoker and removed request for a team October 19, 2020 18:56

mergify bot requested a review from a team October 19, 2020 18:57

teddykoker approved these changes Oct 19, 2020

View reviewed changes

formatting

6bfca06

edenlightning modified the milestones: 1.0.3, 1.1 Oct 19, 2020

ddrevicky added 2 commits October 20, 2020 09:45

Shorter error message regex match in tests

1ee1457

Set num classes as pytest parameter

e3076d5

justusschock approved these changes Oct 20, 2020

View reviewed changes

mergify bot requested a review from a team October 20, 2020 08:07

Borda approved these changes Oct 20, 2020

View reviewed changes

pytorch_lightning/metrics/functional/classification.py Outdated Show resolved Hide resolved

pytorch_lightning/metrics/functional/classification.py Outdated Show resolved Hide resolved

pytorch_lightning/metrics/functional/classification.py Outdated Show resolved Hide resolved

formatting

19c4b73

Borda added the ready PRs ready to be merged label Oct 20, 2020

tchaton requested changes Oct 20, 2020

View reviewed changes

NumesSanguis mentioned this pull request Oct 23, 2020

error when using auroc #4293

Closed

ddrevicky requested a review from tchaton October 23, 2020 08:55

tchaton approved these changes Oct 28, 2020

View reviewed changes

Merge branch 'master' into feature/3304-multiclass-auroc

60c3c98

SkafteNicki requested review from awaelchli, nateraw, SeanNaren and williamFalcon as code owners October 28, 2020 12:30

SkafteNicki added 3 commits October 28, 2020 16:43

Merge branch 'master' into feature/3304-multiclass-auroc

30ccb62

Merge branch 'master' into feature/3304-multiclass-auroc

7f82b3b

Merge branch 'master' into feature/3304-multiclass-auroc

f681b24

rohitgr7 self-requested a review October 30, 2020 11:38

SkafteNicki reviewed Oct 30, 2020

View reviewed changes

ddrevicky and others added 2 commits October 30, 2020 18:18

Update CHANGELOG

ee4078d

Merge branch 'master' into feature/3304-multiclass-auroc

8b1b7f1

edenlightning added the hacktoberfest-accepted label Oct 30, 2020

SkafteNicki merged commit 38bb4e2 into Lightning-AI:master Oct 30, 2020

NumesSanguis mentioned this pull request Nov 4, 2020

[Metrics] Confusion matrix class interface #4348

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metrics] Add multiclass auroc #4236

[Metrics] Add multiclass auroc #4236

ddrevicky commented Oct 19, 2020 •

edited

Loading

ddrevicky Oct 19, 2020

codecov bot commented Oct 19, 2020 •

edited

Loading

teddykoker left a comment

Borda left a comment

tchaton left a comment

tchaton Oct 20, 2020

tchaton Oct 20, 2020

ddrevicky Oct 20, 2020 •

edited

Loading

tchaton Oct 28, 2020

tchaton Oct 20, 2020

ddrevicky Oct 20, 2020

SkafteNicki Oct 25, 2020

tchaton Oct 28, 2020

SkafteNicki Oct 30, 2020

ddrevicky Oct 30, 2020 •

edited

Loading

		@@ -106,6 +106,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).

		- Added trace functionality to the function `to_torchscript` ([#4142](https://github.com/PyTorchLightning/pytorch-lightning/pull/4142))

		- Added multiclass AUROC metric ([#4236](https://github.com/PyTorchLightning/pytorch-lightning/pull/4236))

[Metrics] Add multiclass auroc #4236

[Metrics] Add multiclass auroc #4236

Conversation

ddrevicky commented Oct 19, 2020 • edited Loading

What does this PR do?

Before submitting

PR review

Did you have fun?

Choose a reason for hiding this comment

codecov bot commented Oct 19, 2020 • edited Loading

Codecov Report

teddykoker left a comment

Choose a reason for hiding this comment

Borda left a comment

Choose a reason for hiding this comment

tchaton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddrevicky Oct 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddrevicky Oct 30, 2020 • edited Loading

Choose a reason for hiding this comment

ddrevicky commented Oct 19, 2020 •

edited

Loading

codecov bot commented Oct 19, 2020 •

edited

Loading

ddrevicky Oct 20, 2020 •

edited

Loading

ddrevicky Oct 30, 2020 •

edited

Loading