OverflowError: Python int too large to convert to C long

[luopuya]

word, weights = parts[0], map(REAL, parts[1:])
# ...
result.syn0[line_no] = weights

map return an iterator instead of a list in Python 3
seems to wrap in list call: list(map(REAL, parts[1:]))

[piskvorky]

Yes, it is wrapped in list call. This was fixed in this commit: 2562d00

Or is the issue something else? I'm not sure I understood your report.

[luopuya]

Oh, I use the master branch.
I got another mistake with it.

At line 521 in reset_weights

def reset_weights(self):
    # ...
    random.seed(uint32(self.hashfxn(self.index2word[i] + str(self.seed))))

hash (built-in function) in 64 bit Python return an 64 bit integer, and rasie OverflowError: Python int too large to convert to C long

[luopuya]

[In doc2vec]
Is it sentence (len < 5) would not produce an vector?
Thanks for your reply

[gojomo]

re: "OverflowError: Python int too large to convert to C long"

Are you still getting that? What's your OS & python version?

re: "sentence (len < 5)"

See #325 & #346 – known problem with min_count affecting short docs in Doc2Vec; fix is on its way.

[piskvorky]

@luopuya is this relevant? Do you still get the "OverflowError: Python int too large to convert to C long" in reset_weights?

[luopuya]

I fix it by setting my hashfxn in construct function

def myhashfxn(obj):
return hash(obj) % (2 ** 32)

Word2Vec(hashfxn=myhashfxn)

[gojomo]

I'm looking at related 32-/64-bit issues, and would like to understand what's happening here... but I can only trigger a ValueError (not an OverflowError) when passing large values into random.seed(), and further, the call to uint32() should (and in my testing does) reliably mask the value into the acceptable range. (Essentially, it's already doing what myhashfxn() does.)

Is there a chance that you were getting the error from some older code (that doesn't match the "line 521" excerpt you quoted)? Or maybe you're using some older Numpy version of numpy.uint32() that would raise the OverflowError?

If your hashfxn-workaround still seems necessary, it'd be good to know the versions of Python, gensim, and Numpy in play.

[luopuya]

Sorry for late reply. It is old version
Anaconda use conda to manage packages, conda's latest gensim version is 0.10.3.

[eyaler]

I am getting this error as well on:
python 3.4.3
anaconda 2.3.0 (64-bit)
numpy 1.9.2
gensim 0.10.3
code from: rare-technologies.com/deep-learning-with-word2vec-and-gensim/
the hashfxn workaround works

[eyaler]

see also discussion here: https://www.kaggle.com/c/word2vec-nlp-tutorial/forums/t/11197/gensim-word2vec-cython-on-windows/59956

[GraphGrailAi]

@eyaler I confirm the same error:
Winpython 3.4.3.2 64 bit
numpy 1.10b
scipy 0.15
gensim 0.12.1

[gojomo]

@GraphGrail – are you sure it's gensim 0.12.1 that's being run when you get the error, and not some older version? (Can you share the exact error message, with reported line numbers?)

(If this is happening, then maybe the Windows Python + numpy that people are using, that claims to be 64-bit, may not really be 64-bit? Another user having a similar problem suggested there is no true numpy support for 64-bit Windows... and even though I'd usually suggest 64-bit Python, they reported fewer problems, on Windows, after they switched to 32 bit.)

[tmylk]

@GraphGrail Could you create a PR with a test that triggers this error? We now have continuous Win 32 and 64 builds.
Otherwise the issue will be closed in 7 days

[maribethrc]

I am still getting the same error as @luopuya with the following versions:
Genism 3.4.0
Python 3.6
Numpy 1.15.4

@luopuya where did you put your hashfxn? somewhere inside word2vec.py?

[gojomo]

@maribethrc What OS, and is there a chance you're running a 32-bit Python, or an inadvertently mixed set of 32/64-bit versions of Python and 3rd-party libraries? (How did you install Python, and how did you install required libraries like Numpy?)