Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

size confusion when loading custom vectors #544

Closed
michaelcapizzi opened this issue Oct 20, 2016 · 9 comments
Closed

size confusion when loading custom vectors #544

michaelcapizzi opened this issue Oct 20, 2016 · 9 comments
Labels
bug Bugs and behaviour differing from documentation

Comments

@michaelcapizzi
Copy link

This is a follow up to this issue which still persists.

I am not confident that spacy is housing my vectors after loading.

First of all, I have created a bin of my vectors using vocab.write_binary_vectors(). They are 200-dimensions, but after successfully loading them into my existing instance of English(), they still appear to be 300-dimensions.

>>> nlp = English(vectors=lambda vocab: vocab.load_vectors_from_bin_loc("/path/to/my/binary/vectors/w2v.bin"))
>>> nlp.vocab.__getitem__("this").vector.shape
(300,)

The weirdest thing, though, is that these vectors are not the "original" vectors loaded by spacy (GloVe 200-dimensions):

>>> nlp2 = English()
>>> nlp2.vocab.__getitem__("this").vector.shape
(300,)
>>> nlp.vocab.__getitem__("this").vector == nlp2.vocab.__getitem__("this").vector
array([False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False], dtype=bool)

So this vector is different from the "original", preloaded vector for "this", but it's still 300 dimensions.

The same thing happens if I use vocab.load_vectors() instead of vocab.load_vectors_from_bin_loc():

>>> nlp3 = English(vectors=lambda vocab: vocab.load_vectors("/path/to/my/vectors/in/text/format/w2v.txt"))
>>> nlp3.vocab.__getitem__("this").vector.shape
(300,)
>>> nlp3.vocab.__getitem__("this").vector == nlp.vocab.__getitem__("this").vector
array([False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False], dtype=bool)

At least, however, they are the same as the vectors that were loaded from the bin file:

>>> nlp3.vocab.__getitem__("this").vector == nlp2.vocab.__getitem__("this").vector
array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True], dtype=bool)
@honnibal
Copy link
Member

I need to update the docs — this was indeed broken at the 1.0 release (actually the GloVe loading was also broken). Can you try again with v1.0.5, using the keyword add_vectors instead of vectors?

@michaelcapizzi
Copy link
Author

michaelcapizzi commented Oct 20, 2016 •
edited
Loading

Sure thing. But, I'm sorry, I'm a bit unclear. I'm not sure what command to use and where add_vectors comes in.

So I tried to load vectors from a text file in 1.0.5 using vocab.add_vectors(path/to/vectors.txt):

>>> from spacy.en import English
>>> nlp = English()
>>> this_vector = nlp.vocab.__getitem__("this").vector
>>> this_vector.shape
(300,)
>>> nlp2 = English(vectors=lambda vocab: vocab.add_vectors("/Users/mcapizzi/Github/nlp-pipeline/jupyter_notebooks/data/sample_w2v.txt"))
>>> this_vector_2 = nlp2.vocab.__getitem__("this").vector
>>> this_vector_2.shape
(300,)
>>> this_vector == this_vector_2
array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True], dtype=bool)

So now it appears that the vectors did not get loaded in as the vector for "this" didn't change.

Then I tried vocab.add_vectors(path/to/binary/vectors.bin):

>>> nlp3 = English(vectors=lambda vocab: vocab.add_vectors("/Users/mcapizzi/Github/nlp-pipeline/jupyter_notebooks/data/sample_w2v.bin"))
>>> this_vector_3 = nlp3.vocab.__getitem__("this").vector
>>> this_vector_3.shape
(300,)
>>> this_vector == this_vector_3
array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True], dtype=bool)

Same result.

If I misunderstood what you wanted me to try, please clarify and I'll happily test it out.

@honnibal
Copy link
Member

Sorry, I meant like this:

nlp2 = English(add_vectors=lambda vocab: vocab.load_vectors("/Users/mcapizzi/Github/nlp-pipeline/jupyter_notebooks/data/sample_w2v.txt"))

@honnibal honnibal added the bug Bugs and behaviour differing from documentation label Oct 21, 2016
@michaelcapizzi
Copy link
Author

Got it.

Unfortunately, a new error (again, I'm in 1.0.5:

>>> nlp = English(add_vectors=lambda vocab: vocab.load_vectors("/Users/mcapizzi/Github/nlp-pipeline/jupyter_notebooks/data/sample_w2v.txt")
... )
>>> nlp.vocab.__getitem__("this").vector.shape
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "spacy/lexeme.pyx", line 105, in spacy.lexeme.Lexeme.vector.__get__ (spacy/lexeme.cpp:4614)
ValueError: Word vectors set to length 0. This may be because the data is not installed. If you haven't already, run
python -m spacy.en.download all
to install the data.

And it does the same thing when trying to load a bin file:

>>> nlp = English(add_vectors=lambda vocab: vocab.load_vectors("/Users/mcapizzi/Github/nlp-pipeline/jupyter_notebooks/data/sample_w2v.bin"))
>>> nlp.vocab.__getitem__("this").vector.shape
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "spacy/lexeme.pyx", line 105, in spacy.lexeme.Lexeme.vector.__get__ (spacy/lexeme.cpp:4614)
ValueError: Word vectors set to length 0. This may be because the data is not installed. If you haven't already, run
python -m spacy.en.download all
to install the data.

@honnibal
Copy link
Member

Hmm. Thanks for your patience.

honnibal added a commit that referenced this issue Oct 21, 2016
@honnibal
Add test for vector resizing, re Issue #544
e7af75e
@honnibal
Copy link
Member

honnibal commented Oct 21, 2016 •
edited
Loading

I really should've slowed down and tested this more carefully — I'm trying to do too many things at once.

I've added a method vocab.resize_vectors(new_size), to support the workflow where you want to assign the new vectors on lexeme.vector = new_vector. I think I fixed the vector loading too.

If you want to try this out, you can do:

pip install cython==0.23
pip install cymem thinc preshed
pip install https://github.com/explosion/spaCy/archive/master.zip

@honnibal
Copy link
Member

This should be fixed in 1.1.0. Please reopen if it's not!

@michaelcapizzi
Copy link
Author

michaelcapizzi commented Nov 8, 2016 •
edited
Loading

Thanks @honnibal . It indeed works in 1.1.0.

One clarification for anyone who may be having trouble: the argument to vocab.load_vectors() must be a buffer not a path to a file:

f = open("path/to/vectors.txt", "r")
p.vocab.load_vectors(f)
f.close()

This is clear in the source code and documentation, but could be easily overlooked.

@gurunathparasaram gurunathparasaram mentioned this issue Sep 6, 2017
Closed
@danielhers danielhers mentioned this issue Nov 8, 2017
Closed
@lock
Copy link

lock bot commented May 8, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked as resolved and limited conversation to collaborators May 8, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Bugs and behaviour differing from documentation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@honnibal @michaelcapizzi