Support lazy, recursive sentence splitting #7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
danieldk
force-pushed
the
greedy-recursive-splitting
branch
2 times, most recently
from
e66053a
to
e918001
Compare
We use sentence splitting in the biaffine parser to keep the O(n^2) biaffine attention model tractable. However, since the sentence splitter makes errors, the parser may not have the correct head available. This change adds another splitting strategy. The goal of this strategy is to use the highest-probability splits to partition a doc until each partition is smaller than or equal to a maximum length. This reduces the number of attachment errors as a result of incorrect sentence splits, while providing an upper bound on complexity. The algorithm works as follows: * If the length |d| > max_length: - Find the highest-probability split in d according to senter. - Split d into d_1 and d_2 using the highest probability split. - Recursively apply this algorithm to d_1 and d_2. * Otherwise: do nothing
We use a back-off when the first token is the best splitting point, to avoid an infinite recursion. The back-off was simply to use the second token, refine this to choose the second-most probable splitting point.
We now use the spaCy parser scorer.
Now that Thinc doesn't set the Tensor type globally anymore, we have to make sure that Tensors are placed on the correct device.
Before this change, we'd use the senter pipe directly. However, this did not work with the transformer model without modifications (because it clears tensors after backprop). By using the functionality proposed in explosion/spaCy#11002 we can use the activations that are stored by the senter pipe in `Doc`.
This measure was removed.
Also make evaluation targets depend on the corpus they use.
Suggested by @kadarakos
danieldk
force-pushed
the
greedy-recursive-splitting
branch
from
13304a3
to
7e8b69c
Compare
|
Please don't review this PR, I am doing some force pushes here, mostly to get this building against spaCy v4. I'll close this PR and open up a new one when it's ready for review. |
kadarakos
reviewed
Feb 27, 2023
| for doc in docs: | ||
| activations = doc.activations.get(senter.name, None) | ||
| if activations is None: | ||
| raise ValueError("Greedy splitting requires `senter` with `save_activations` enabled.\n" |
There was a problem hiding this comment.
Suggested change
| raise ValueError("Greedy splitting requires `senter` with `save_activations` enabled.\n" | |
| raise ValueError("Lazy splitting requires `senter` with `save_activations` enabled.\n" |
There was a problem hiding this comment.
I'll fix that, but we really shouldn't use this as a reviewing PR, it already had force pushes and stuff, so I'll close it.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We use sentence splitting in the biaffine parser to keep the O(n^2) biaffine attention model tractable. However, since the sentence splitter makes errors, the parser may not have the correct head available.
This change adds another splitting strategy as the preferred splitting. The goal of this strategy is to split up a
Docinto pieces that are as large as possible given a maximum n_max. This reduces the number of attachment errors as a result of incorrect sentence splits, while providing an upper bound on complexity (O(n_max^2)).The algorithm works as follows:
Note: draft, requires functionality from PR explosion/spaCy#11002, which targets spaCy v4.