Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CachedGISTEmbedLoss #2592

Merged
merged 11 commits into from
Apr 16, 2024
+329 −11
Merged

Add CachedGISTEmbedLoss #2592

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
7aa827c
Add CachedGISTEmbedLoss to __init__ to allow for easier import
JacksonCakes Apr 13, 2024
5e05045
Add intiial implementation
JacksonCakes Apr 13, 2024
98b6ecf
Add initial implementation
JacksonCakes Apr 13, 2024
f4d7f38
Add docstring
JacksonCakes Apr 13, 2024
263d21c
Remove unused imports
JacksonCakes Apr 13, 2024
5c054da
Sync with main repo
JacksonCakes Apr 13, 2024
3208e61
Update guided similarity computation in mini-batch
JacksonCakes Apr 15, 2024
3215c06
Fix guiding mask by adding offset
JacksonCakes Apr 15, 2024
6ccf388
Write docstring on multiple lines
tomaarsen Apr 16, 2024
6a6455d
Add CachedGIST to loss API
tomaarsen Apr 16, 2024
fccc92a
Add CachedGIST to Loss Overview
tomaarsen Apr 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/package_reference/losses.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,11 @@ This allows our network to be fine-tuned to recognize the similarity of sentence
.. autoclass:: sentence_transformers.losses.GISTEmbedLoss
```

## CachedGISTEmbedLoss
```eval_rst
.. autoclass:: sentence_transformers.losses.CachedGISTEmbedLoss
```

## MSELoss
```eval_rst
.. autoclass:: sentence_transformers.losses.MSELoss
Expand Down
22 changes: 11 additions & 11 deletions docs/training/loss_overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,17 @@ Loss functions play a critical role in the performance of your fine-tuned model.

**Note**: you can often convert one training data format into another, allowing more loss functions to be viable for your scenario. For example, `(sentence_A, sentence_B) pairs` with `class` labels can be converted into `(anchor, positive, negative) triplets` by sampling sentences with the same or different classes.

| Texts | Labels | Appropriate Loss Functions |
|-----------------------------------------------|--------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `single sentences` | `class` | <a href="../package_reference/losses.html#batchalltripletloss">`BatchAllTripletLoss`</a><br><a href="../package_reference/losses.html#batchhardsoftmargintripletloss">`BatchHardSoftMarginTripletLoss`</a><br><a href="../package_reference/losses.html#batchhardtripletloss">`BatchHardTripletLoss`</a><br><a href="../package_reference/losses.html#batchsemihardtripletloss">`BatchSemiHardTripletLoss`</a> |
| `single sentences` | `none` | <a href="../package_reference/losses.html#contrastivetensionloss">`ContrastiveTensionLoss`</a><br><a href="../package_reference/losses.html#denoisingautoencoderloss">`DenoisingAutoEncoderLoss`</a> |
| `(anchor, anchor) pairs` | `none` | <a href="../package_reference/losses.html#contrastivetensionlossinbatchnegatives">`ContrastiveTensionLossInBatchNegatives`</a> |
| `(damaged_sentence, original_sentence) pairs` | `none` | <a href="../package_reference/losses.html#denoisingautoencoderloss">`DenoisingAutoEncoderLoss`</a> |
| `(sentence_A, sentence_B) pairs` | `class` | <a href="../package_reference/losses.html#softmaxloss">`SoftmaxLoss`</a> |
| `(anchor, positive) pairs` | `none` | <a href="../package_reference/losses.html#cachedmultiplenegativesrankingloss">`CachedMultipleNegativesRankingLoss`</a><br><a href="../package_reference/losses.html#multiplenegativesrankingloss">`MultipleNegativesRankingLoss`</a><br><a href="../package_reference/losses.html#multiplenegativessymmetricrankingloss">`MultipleNegativesSymmetricRankingLoss`</a><br><a href="../package_reference/losses.html#megabatchmarginloss">`MegaBatchMarginLoss`</a><br><a href="../package_reference/losses.html#gistembedloss">`GISTEmbedLoss`</a> |
| `(anchor, positive/negative) pairs` | `1 if positive, 0 if negative` | <a href="../package_reference/losses.html#contrastiveloss">`ContrastiveLoss`</a><br><a href="../package_reference/losses.html#onlinecontrastiveloss">`OnlineContrastiveLoss`</a> |
| `(sentence_A, sentence_B) pairs` | `float similarity score` | <a href="../package_reference/losses.html#cosentloss">`CoSENTLoss`</a><br><a href="../package_reference/losses.html#angleloss">`AnglELoss`</a><br><a href="../package_reference/losses.html#cosinesimilarityloss">`CosineSimilarityLoss`</a> |
| `(anchor, positive, negative) triplets` | `none` | <a href="../package_reference/losses.html#cachedmultiplenegativesrankingloss">`CachedMultipleNegativesRankingLoss`</a><br><a href="../package_reference/losses.html#multiplenegativesrankingloss">`MultipleNegativesRankingLoss`</a><br><a href="../package_reference/losses.html#tripletloss">`TripletLoss`</a><br><a href="../package_reference/losses.html#gistembedloss">`GISTEmbedLoss`</a> |
| Texts | Labels | Appropriate Loss Functions |
|-----------------------------------------------|--------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `single sentences` | `class` | <a href="../package_reference/losses.html#batchalltripletloss">`BatchAllTripletLoss`</a><br><a href="../package_reference/losses.html#batchhardsoftmargintripletloss">`BatchHardSoftMarginTripletLoss`</a><br><a href="../package_reference/losses.html#batchhardtripletloss">`BatchHardTripletLoss`</a><br><a href="../package_reference/losses.html#batchsemihardtripletloss">`BatchSemiHardTripletLoss`</a> |
| `single sentences` | `none` | <a href="../package_reference/losses.html#contrastivetensionloss">`ContrastiveTensionLoss`</a><br><a href="../package_reference/losses.html#denoisingautoencoderloss">`DenoisingAutoEncoderLoss`</a> |
| `(anchor, anchor) pairs` | `none` | <a href="../package_reference/losses.html#contrastivetensionlossinbatchnegatives">`ContrastiveTensionLossInBatchNegatives`</a> |
| `(damaged_sentence, original_sentence) pairs` | `none` | <a href="../package_reference/losses.html#denoisingautoencoderloss">`DenoisingAutoEncoderLoss`</a> |
| `(sentence_A, sentence_B) pairs` | `class` | <a href="../package_reference/losses.html#softmaxloss">`SoftmaxLoss`</a> |
| `(anchor, positive) pairs` | `none` | <a href="../package_reference/losses.html#cachedmultiplenegativesrankingloss">`CachedMultipleNegativesRankingLoss`</a><br><a href="../package_reference/losses.html#multiplenegativesrankingloss">`MultipleNegativesRankingLoss`</a><br><a href="../package_reference/losses.html#multiplenegativessymmetricrankingloss">`MultipleNegativesSymmetricRankingLoss`</a><br><a href="../package_reference/losses.html#megabatchmarginloss">`MegaBatchMarginLoss`</a><br><a href="../package_reference/losses.html#cachedgistembedloss">`CachedGISTEmbedLoss`</a><br><a href="../package_reference/losses.html#gistembedloss">`GISTEmbedLoss`</a> |
| `(anchor, positive/negative) pairs` | `1 if positive, 0 if negative` | <a href="../package_reference/losses.html#contrastiveloss">`ContrastiveLoss`</a><br><a href="../package_reference/losses.html#onlinecontrastiveloss">`OnlineContrastiveLoss`</a> |
| `(sentence_A, sentence_B) pairs` | `float similarity score` | <a href="../package_reference/losses.html#cosentloss">`CoSENTLoss`</a><br><a href="../package_reference/losses.html#angleloss">`AnglELoss`</a><br><a href="../package_reference/losses.html#cosinesimilarityloss">`CosineSimilarityLoss`</a> |
| `(anchor, positive, negative) triplets` | `none` | <a href="../package_reference/losses.html#cachedmultiplenegativesrankingloss">`CachedMultipleNegativesRankingLoss`</a><br><a href="../package_reference/losses.html#multiplenegativesrankingloss">`MultipleNegativesRankingLoss`</a><br><a href="../package_reference/losses.html#tripletloss">`TripletLoss`</a><br><a href="../package_reference/losses.html#cachedgistembedloss">`CachedGISTEmbedLoss`</a><br><a href="../package_reference/losses.html#gistembedloss">`GISTEmbedLoss`</a> |

## Loss modifiers

Expand Down
Loading
Loading