Normalized form of "am" is "a.m."

[patperry]

It appears that for the default English model, the normalized form of "am" is "a.m.". This seems reasonable, but then shouldn't the normalized form of "I'm" be ["i", "a.m."]?

Would it be better to make the normalized form of "am" and "a.m." be "am"?

How to reproduce the behaviour

from spacy.lang.en import English
nlp = English()
print([tok.norm_ for tok in nlp("I'm fine. I am fine")])
#> ['i', 'am', 'fine', '.', 'i', 'a.m.', 'fine']

Your Environment

• Operating System: Linux version 4.9.93-linuxkit-aufs
• Python Version Used: 3.7.0
• spaCy Version Used: 2.0.12
• Environment Information:

[honnibal]

I think this is probably a bad norm exception, because it really refers to "am" as in the time. The more common case is the verb to be --- so we should remove this.

[gavrieltal]

Is this supposed to be the relevant code, in tokenizer_exceptions.py, around line 290?

for h in range(1, 12 + 1):                                                                      
    for period in ["a.m.", "am"]:                                                               
        _exc["%d%s" % (h, period)] = [                                                          
            {ORTH: "%d" % h},                                                                   
            {ORTH: period, LEMMA: "a.m.", NORM: "a.m."}]                                        
    for period in ["p.m.", "pm"]:                                                               
        _exc["%d%s" % (h, period)] = [                                                          
            {ORTH: "%d" % h},                                                                   
            {ORTH: period, LEMMA: "p.m.", NORM: "p.m."}]

And, if so, why would one get that the norm of token "am" in "I am fine" is "a.m." when there was no number beforehand such as in "3am"?

[ines]

And, if so, why would one get that the norm of token "am" in "I am fine" is "a.m." when there was no number beforehand such as in "3am"?

You're correct, this doesn't make sense and the above exceptions really only target tokens like "1am" or "10p.m".

I've been trying to find where this norm is set and it's pretty mysterious... 🤔 I haven't been able to figure it out yet.

[patperry]

Also this is troubling:

import spacy
from spacy.lang.en import English
nlp = English()
print([tok.norm_ for tok in nlp('The normalized form of "a" is "gonna".')])
#> ['the', 'normalized', 'form', 'of', '"', 'gonna', '"', 'is', '"', 'going', 'to', '"', '.']

[ines]

@patperry

print([tok.norm_ for tok in nlp('The normalized form of "a" is "gonna".')])

Thanks for the example – this might explain a lot. Looks like this is actually an issue with the caching of the token attributes. There was a similar bug a while ago, but I'm pretty sure we fixed that. But it might still be related.

[honnibal]

Fix should now be up on develop: #3029

Thanks for your help reporting this.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.