Sigmoid Cross-Entropy Loss: ignore selected targets by ignore_label
#4986
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sig-ce learns to ignore by zeroing out the loss/diff at targets equal to the configured `ignore_label`. n.b. as of now the loss/diff are not properly normalized when there are ignored targets. sig-ce loss should adopt the same normalization options as softmax loss.
shelhamer
force-pushed
the
sigce-ignore
branch
4 times, most recently
from
f4d7208 to
4001295
Compare
sig-ce loss handles all the same normalizations as the softmax loss; refer to BVLC#3296 for more detail. this preserves the default normalization for sig-ce loss: batch size.
shelhamer
force-pushed
the
sigce-ignore
branch
from
4001295 to
3d62e3c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sigmoid cross-entropy loss learns to ignore by zeroing out the loss/diff at targets equal to
the configured
ignore_label.ignore_labelNote that sig-ce loss and softmax loss have different defaults for normalization: batch size (sig-ce) and valid (softmax).
TODO: the common normalization and ignore logic should be pulled up into
LossLayer, and layers should die loudly by default instead of silently ignoring these options.