💫 Lemmatizer should apply rules on OOV words

[honnibal]

@juanmirocks points out in #327 that the lemmatizer fails on OOV words:

>>> nlp.vocab.morphology.lemmatizer(u'endosomes', 'noun', morphology={'number': 'plur'})set([u'endosomes'])
>>> nlp.vocab.morphology.lemmatizer(u'chromosomes', 'noun', morphology={'number': 'plur'})
set([u'chromosome'])

Suggested patch to lemmatizer.py

    oov_forms = []
    for old, new in rules:
        if string.endswith(old):
            form = string[:len(string) - len(old)] + new
            if form in index or not form.isalpha():
                forms.append(form)
            else:
                oov_forms.append(form)
    if not forms:
        forms.extend(oov_forms)

Your Environment

• Operating System:
• Python Version Used:
• spaCy Version Used:
• Environment Information:

[juanmirocks]

@honnibal thanks for looking into this -- Know I understand the problem better.

Looking forward to a fix as I think it will greatly improve the performance of my methods.

Kindly let me know if I can help with the patch.

[ines]

@juanmirocks If you have time to add the patch to lemmatizer.py and make a pull request, that would be great 👍

The only thing that's important is to make sure it's properly tested. In this case, I think it'd be fine to just add a regression test for this issue, using the above examples.

[juanmirocks]

Thanks Ines. I will gladly help and try. Is that patch code all what is necessary as per decided for Hannibal? I will dive in

…

On Mon, 30 Jan 2017 at 14:48, Ines Montani ***@***.***> wrote: @juanmirocks <https://github.com/juanmirocks> If you have time to add the patch to lemmatizer.py and make a pull request, that would be great 👍 The only thing that's important is to make sure it's properly tested. In this case, I think it'd be fine to just add a regression test <https://github.com/explosion/spaCy/blob/master/CONTRIBUTING.md#fixing-bugs> for this issue, using the above examples. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#781 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGQH0KVj1NvUEf7qdGmE9e2roqlMaOAks5rXeoZgaJpZM4LwgLo> .

[juanmirocks]

Other examples that I will use for debugging and testing:

• "colocalizes" is lemmatized the same, "colocalizes"

  • PP2A colocalizes with shugoshin at centromeres and is required for centromeric protection .

[juanmirocks]

Pull request created

[honnibal]

Merged!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.