* initial coref_er pipe

* matcher more flexible

* base coref component without actual model

* initial setup of coref_er.score

* rename to include_label

* preliminary score_clusters method

* apply scoring in coref component

* IO fix

* return None loss for now

* rename to CoreferenceResolver

* some preliminary unit tests

* use registry as callable

This includes the coref code that was being tested separately, modified
to work in spaCy. It hasn't been tested yet and presumably still needs
fixes.

In particular, the evaluation code is currently omitted. It's unclear at
the moment whether we want to use a complex scorer similar to the
official one, or a simpler scorer using more modern evaluation methods.

Ended up not making a difference, but oh well.

When sentences are not available, just treat the whole doc as one
sentence. A reasonable general fallback, but important due to the init
call, where upstream components aren't run.

Training seems to actually run now!

This makes their scope tighter and more contained, and has the nice side
effect that fewer things need to be passed around for backprop.

The loss was being returned as a single element array, which caused
training to die when it attempted to turn it into JSON.

This is closer to the traditional evaluation method. That uses an
average of three scores, this is just using the bcubed metric for now
(nothing special about bcubed, just picked one).

The scoring implementation comes from the coval project. It relies on
scipy, which is one issue, and is rather involved, which is another.

Besides being comparable with traditional evaluations, this scoring is
relatively fast.

The intent of this was that it would be a component pipeline that used
entities as input, but that's now covered by the get_mentions function
as a pipeline arg.

`make_clean_doc` is not needed and was removed.

`logsumexp` may be needed if I misunderstood the loss calculation, so I
left it in for now with a note.

At a few points in the code it's normal to get a "2d" array where each
row is a single entry. Calling squeeze will make that a proper 1d
array... unless it's just one entry, in which case it turns into a 0d
scalar. That's not what we want; flatten() provides the desired
behavior.

Shouldn't change correctness but is a little clearer

The call here was creating a float64 array, which was turning many
downstream scores into float64s. Later on these values were assigned to
a float32 array in backprop, and numerical underflow caused things to go
to zero.

That's almost certainly not the only reason things go to zero, but it is
incorrect.

I think this change is correct, but intuition doesn't really help
here...

This should maybe go elsewhere?

This doesn't change the math but makes the transposes slightly easier to
understand (maybe?).

In practice this is only ever used with axis=1, so it wasn't causing
issues, even though it was wrong.

Comment wondered if there should be some subtraction to avoid double
counting, but it probably doesn't matter because the diagonal is 0.

I think this was technically incorrect but harmless. The reason the code
here is different than the reference in coref-hoi is that the indices
there are such that they get +1 at the end of processing, while the code
here handles indices directly.

Coref docs are kind of long, and using 10 samples on a smallish GPU can
cause OOMs.

The calculation of this in the coref-hoi code is hard to follow. Based
on comments and variable names it sounds like it's using the doc length,
but it might actually be the number of mentions? Number of mentions
should be much larger and seems more correct, but might want to revisit
this.

This rewrites the loss to not use the Thinc crossentropy code at all.
The main difference here is that the negative predictions are being
masked out (= marginalized over), but negative gradient is still being
reflected.

I'm still not sure this is exactly right but models seem to train
reliably now.

Not necessary for convergence, but in coref-hoi this seems to add a few
f1 points.

Note that there are two width-related features in coref-hoi. This is a
"prior" that is added to mention scores. The other width related feature
is appended to the span embedding representation for other layers to
reference.

This pulls out references to needed bits so that other parts (the larger
embeddings) can be freed before backprop.

The tuplify code here was added to Thinc proper and that's been
released, so no need to have it here any more.

This was the slowest part of the code, and using scatter_add here
probably reduces the runtime by 50%.

This is now cleaner and significantly faster. There's still some messy
parts in the code (particularly variable names), will get to that later.

Some of the lengths and backprop weren't right.

Also various cleanup.

Accidentally deleted it

There was an off-by-one error in how mentions are generated that would
affect mentions at the end of a sentence. This was pretty nasty.

Was using batch absolute indices to manage mentions, but extract_spans
expects doc-relative ones.

This test was failing not because the thing it was testing wasn't
working, but because of the way span equality works. Span equality
relies on doc equality, and doc equality is object identity, so spans
from different docs will never be equal.

This calculates scores as an average of three metrics. As noted in the
code, these metrics all have issues, but we want to use them to match up
with prior work.

This should be replaced with some simpler default scoring and the scorer
here should be moved to an external project to be passed in just for
generating the traditional scores.

Also formatting.

This continue should be a break. The current form doesn't cause errors
but using a break will be a bit faster.

This generall means fewer spans are considered, which makes individual
steps in training faster but can make training take longer to find the
good spans.

In the reference implementations, there's usually a function to build a
ffnn of arbitrary depth, consisting of a stack of Linear >> Relu >>
Dropout. In practice the depth is always 1 in coref-hoi, but in earlier
iterations of the model, which are more similar to our model here (since
we aren't using attention or even necessarily BERT), using a small depth
like 2 was common. This hard-codes a stack of 2.

In brief tests this allows similar performance to the unstacked version
with much smaller embedding sizes.

The depth of the stack could be made into a hyperparameter.

Scoring code was just using one metric, not all three of interest.

This brings coref up to date, in particular giving access to 3.2
features.

…ound doc.spans)

This absolutely does not work. First step here is getting over most of
the code in roughly the files we want it in. After the code has been
pulled over it can be restructured to match spaCy and cleaned up.

Replaced with model arguments

Accidentally omitted before

The span predictor component is initialized but not used at all now.
Plan is to work on it after the word level clustering part is trainable
end-to-end.

Evaluate does not work - predict hasn't been updated

Evaluation needs fixing, and code still needs cleanup.

This doesn't work as a component because it needs to modify gold data,
so instead it's a conversion script (in another repo).

Moved everything into coref_util.py, deleted wl-specific file.

This should be moved into its own file, but for now just stubbing out
the methods.

The way fake batching works is that the pipeline component calls the
model repeatedly in a loop internally. It feels like this should break
something, but it worked in testing.

Another issue is that this changes the signature of some of the pipeline
functions, though I don't think that's an issue.

Tested with batch size of 2, so more testing is needed, but this is a
start.