Problem using doc array() and bytes() functions together

[clippered]

How to reproduce the behaviour

import spacy
from spacy.attrs import ENT_IOB, ENT_TYPE
from spacy.tokens import Doc


nlp = spacy.load('en_core_web_sm')
doc = nlp('This is 10%.')
print(doc[2].orth_, doc[2].pos_, doc[2].tag_, doc[2].ent_type_)  # displays "10 NUM CD PERCENT"

# removing '10%' entity
header = [ENT_IOB, ENT_TYPE]
ent_array = doc.to_array(header)
idx = 2
ent_array[idx, 0] = 0
ent_array[idx, 1] = 0
doc.from_array(header, ent_array)

print(doc[2].orth_, doc[2].pos_, doc[2].tag_, doc[2].ent_type_)  # displays "10 NUM CD "

# serializing then deserializing
bytes = doc.to_bytes()
doc2 = Doc(nlp.vocab).from_bytes(bytes)
print(doc2[2].orth_, doc2[2].pos_, doc2[2].tag_, doc2[2].ent_type_)  # displays "10   "

Basically, my use case would be to change some of the tagged entities while keeping other entities. From the small example above, it seems the problem occurs when calling the serialization/deserialization part after entity removal part. When you run it, it shows:

10 NUM CD PERCENT
10 NUM CD 
10

However, commenting out the entity removal part (the call to to_array up to from_array) of the code, the serialization works fine and shows:

10 NUM CD PERCENT
10 NUM CD PERCENT
10 NUM CD PERCENT

Environment

• Operating System: OSX 10.14
• Python Version Used: Python 3.6.6
• spaCy Version Used: spaCy 2.0.18
• Environment Information:

[clippered]

Apologies if I might be missing some information.
In the example above, I want to keep the POS/TAG attributes after serialization. Something with an output of:

10 NUM CD PERCENT
10 NUM CD 
10 NUM CD 

[honnibal]

Thanks for the example, this definitely looks suspicious. Would you mind making a pull request with your example as an xfail-ed test? It would be in the spacy/tests/regression directory.

[honnibal]

It looks to me like the problem might be with the is_tagged attribute. If this is set to False, the POS tags are not serialized, leading to the mismatch we're seeing here.

[honnibal]

Yep, if we do doc.from_array([ENT_IOB, ENT_TYPE]), it was seeing that no TAG is set, and then clobbering setting doc.is_tagged to false. This led us to not serialize the tag data.

Fixed now 🎉

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.