Series.unique converts uint64 to int64 (with overflow)

[dnspies]

>>> import pandas as pd
>>> import numpy as np
>>> s = pd.Series([1,2,2**63], dtype=np.dtype('uint64'))
>>> s
0                      1
1                      2
2    9223372036854775808
dtype: uint64
>>> s.unique()
array([                   1,                    2, -9223372036854775808])

Series.unique should preserve whatever type the values are. But instead it changes from uint64 to int64.

INSTALLED VERSIONS

...
python: 3.4.3.final.0
python-bits: 64
...

...
pandas: 0.17.1
numpy: 1.10.4
...

[jreback]

this is a sympton of #4471 in general uint64 support is not great. PR's are welcome to fix that!

[gfyoung]

@jreback : IIUC, your statement is incorrect. The bug traces back as follows:

When you call Series.unique(), it calls the super-class implementation here. That method then calls unique1d() here. The erroneous check is here, which catches np.uint64 but then incorrectly converts them to int64.

The correct patch I think is to change the check to np.signedinteger and then have a separate one for np.unsignedinteger. However, I cannot find a proper implementation of a uint64 hashtable. In addition, things seem to get insanely complicated as we move to hashtable.pyx (and related helper .pxi.in files) khash.pxd, and khash.h, which setup all of the hashtable functionality.

The easy way out but less desirable way I think is to modify this branch here and convert the resulting array back to the original dtype.

Thoughts?

[jreback]

@gfyoung I was pointing to a 'generic' issue with uint64. it doesn't work in lots of places!

[gfyoung]

@jreback : Ah, fair enough. But thoughts about this?

[jreback]

However, I cannot find a proper implementation of a uint64 hashtable. In addition, things seem to get insanely complicated as we move to hashtable.pyx (and related helper .pxi.in files) khash.pxd, and khash.h, which setup all of the hashtable functionality.

hmm

[gfyoung]

@jreback : If someone knows how to navigate through that swamp of hashtable code, then by all means, patch it! 😄 But yes, it's just a matter of adding another hashtable class.

[jreback]

you can use python hashing, no?

In [9]: hash(np.uint64(2**63+1))
Out[9]: 5

In [10]: hash(np.int64(2**63-1))
Out[10]: 3

[gfyoung]

@jreback : Perhaps, but then what's going on with this entire hashtable class? Why was it implemented that way if Python hashing is feasible?

[jreback]

prob not that hard to write some klib type stuff in cython

[jreback]

klib is >> faster than python for the types of things we do (and doesn't have as much overhead for pyobjects)

[jreback]

or just add to the klib stuff (might be easier)

[gfyoung]

@jreback : I think adding to the klib is the best solution, but I don't really understand how all of it works.

[jreback]

me neither :<

[jreback]

its just template stuff in a header only library

[jreback]

you can prob copy int64 and just do a replace. (maybe)

[gfyoung]

@jreback : True that file is, but I have no idea how it gets populated.