BUG: Prevent uint64 overflow in Series.unique #14915

gfyoung · 2016-12-19T05:43:48Z

Introduces a UInt64HashTable class to hash uint64 elements and prevent overflow in functions like Series.unique.

codecov-io · 2016-12-19T06:57:28Z

Current coverage is 84.65% (diff: 100%)

Merging #14915 into master will increase coverage by <.01%

@@             master     #14915   diff @@
==========================================
  Files           144        144          
  Lines         51016      51020     +4   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43184      43189     +5   
+ Misses         7832       7831     -1   
  Partials          0          0

Powered by Codecov. Last update 3ccb501...380c580

jreback · 2016-12-19T10:22:29Z

pandas/tests/test_base.py

+                                      np.arange(len(xs), dtype=np.int64))
+
+    def test_get_unique(self):
+        s = pd.Series([1, 2, 2**63, 2**63], dtype=np.uint64)


i think needs a bit more testing
most test are in algos for things like this

"Needs a bit more testing" is vague. What sort of cases are you thinking of? Secondly, the affected function has no relation to algos. The code path is completely different.

However, you did catch a bug in pd.unique, which does travel through algorithms.py and can now benefit from my hashtable. 😄

not on the hashtable itself. ideally you should add uint64 to lots of places that we test int64 (construction type things). IOW, to actually exercise this code. I know this is vague. happy to merge this and to do a followup with more tests (and potentially breaking things). later.

Okay, well that's a larger issue beyond this PR. However, let's not merge until I patch pd.unique, which suffers from a similar issue as Series.unique.

jreback · 2016-12-19T11:32:56Z

you will need to rebase after #14919. I moved a couple of things, and added a HashTable testing section.

Though what I would really like is to systematically add unit64 to as many places as we test other dtypes and see what breaks / falls out (can do in another PR). Then create issues for incompat places.

jreback · 2016-12-19T20:19:49Z

what was the bug (from before I moved unique)?

gfyoung · 2016-12-19T20:21:00Z

@jreback : The diff explains it all. We were smashing all integer types together into one hashtable, which does not work if you have uint.

jreback · 2016-12-19T20:27:16Z

ok, lgtm. ping on green.

gfyoung · 2016-12-19T20:42:41Z

@jreback : Sounds good. I changed my mind about changing pd.unique in this PR. core.algorithms is riddled with uint64 bugs that need to be checked. I'll tackle in a follow-up.

jreback · 2016-12-19T20:55:28Z

@jreback : Sounds good. I changed my mind about changing pd.unique in this PR. core.algorithms is riddled with uint64 bugs that need to be checked. I'll tackle in a follow-up.

totally fine. we can make a master issue of these if that is helpful.

Introduces a UInt64HashTable class to hash uint64 elements and prevent overflow in functions like Series.unique. Closes pandas-devgh-14721.

gfyoung · 2016-12-20T09:16:23Z

@jreback : Everything is green, so ready to merge if there are no other concerns.

jreback · 2016-12-20T13:46:51Z

thanks!

as I said, ideally have an issue which greatly expands test coverage for uint64 (and just mark the tests that are failing (e.g. assert that they are failing or comment out)).

Then can come back around and fix.

Uses UInt64HashTable to patch a uint64 overflow bug in pd.unique analogous to that seen in Series.unique (patched in pandas-devgh-14915).

1) duplicated() Updates documentation to describe the "values" parameter in the signature, adds tests for uint64, and refactors to use duplicated_uint64. 2) mode() Updates documentation to describe the "values" parameter in the signature, adds tests for uint64, and reactors to use mode_uint64. 3) unique() Uses UInt64HashTable to patch a uint64 overflow bug analogous to that seen in Series.unique (patched in pandas-devgh-14915). 4) Types API Introduces "is_signed_integer_dtype" and "is_unsigned _integer_dtype" to the public API. Used in refactoring/ patching of 1-3.

Adds `uint64` ranking functions to `algos.pyx` to allow for proper ranking with `uint64`. Also introduces partial patch for `factorize()` by adding `uint64` hashtables and vectors for usage. However, this patch is only partial because the larger bug of non- support for `uint64` in `Index` has not been fixed (**UPDATE**: tackled in #14937): ~~~python >>> from pandas import Index, np >>> Index(np.array([2**63], dtype=np.uint64)) Int64Index([-9223372036854775808], dtype='int64') ~~~ Also patches a bug in `UInt64HashTable` from #14915 that had an erroneous null condition that was caught during testing and was hence removed. Author: gfyoung <gfyoung17@gmail.com> Closes #14935 from gfyoung/core-algorithms-uint64-two and squashes the following commits: 2598cea [gfyoung] BUG: Patch rank() uint64 behavior

Introduces a `UInt64HashTable` class to hash `uint64` elements and prevent overflow in functions like `Series.unique`. Closes pandas-dev#14721. Author: gfyoung <gfyoung17@gmail.com> Closes pandas-dev#14915 from gfyoung/uint64-hashtable-patch and squashes the following commits: 380c580 [gfyoung] BUG: Prevent uint64 overflow in Series.unique

Adds `uint64` ranking functions to `algos.pyx` to allow for proper ranking with `uint64`. Also introduces partial patch for `factorize()` by adding `uint64` hashtables and vectors for usage. However, this patch is only partial because the larger bug of non- support for `uint64` in `Index` has not been fixed (**UPDATE**: tackled in pandas-dev#14937): ~~~python >>> from pandas import Index, np >>> Index(np.array([2**63], dtype=np.uint64)) Int64Index([-9223372036854775808], dtype='int64') ~~~ Also patches a bug in `UInt64HashTable` from pandas-dev#14915 that had an erroneous null condition that was caught during testing and was hence removed. Author: gfyoung <gfyoung17@gmail.com> Closes pandas-dev#14935 from gfyoung/core-algorithms-uint64-two and squashes the following commits: 2598cea [gfyoung] BUG: Patch rank() uint64 behavior

sinhrks added Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff Bug labels Dec 19, 2016

jreback requested changes Dec 19, 2016

View reviewed changes

gfyoung force-pushed the uint64-hashtable-patch branch from 3e21ad6 to 03e926e Compare December 19, 2016 19:20

gfyoung closed this Dec 19, 2016

gfyoung deleted the uint64-hashtable-patch branch December 19, 2016 19:22

gfyoung restored the uint64-hashtable-patch branch December 19, 2016 19:22

gfyoung reopened this Dec 19, 2016

jreback approved these changes Dec 19, 2016

View reviewed changes

jreback added this to the 0.20.0 milestone Dec 19, 2016

gfyoung force-pushed the uint64-hashtable-patch branch from 03e926e to 1583235 Compare December 20, 2016 00:15

gfyoung closed this Dec 20, 2016

gfyoung force-pushed the uint64-hashtable-patch branch from 1583235 to 8e630b6 Compare December 20, 2016 00:15

BUG: Prevent uint64 overflow in Series.unique

380c580

Introduces a UInt64HashTable class to hash uint64 elements and prevent overflow in functions like Series.unique. Closes pandas-devgh-14721.

gfyoung reopened this Dec 20, 2016

jreback closed this in b35c689 Dec 20, 2016

gfyoung deleted the uint64-hashtable-patch branch December 20, 2016 14:39

gfyoung added a commit to forking-repos/pandas that referenced this pull request Dec 20, 2016

BUG: Use UInt64HashTable in pd.unique

98ab416

Uses UInt64HashTable to patch a uint64 overflow bug in pd.unique analogous to that seen in Series.unique (patched in pandas-devgh-14915).

gfyoung added a commit to forking-repos/pandas that referenced this pull request Dec 20, 2016

BUG: Use UInt64HashTable in pd.unique

1528347

Uses UInt64HashTable to patch a uint64 overflow bug in pd.unique analogous to that seen in Series.unique (patched in pandas-devgh-14915).

gfyoung added a commit to forking-repos/pandas that referenced this pull request Dec 20, 2016

BUG: Use UInt64HashTable in pd.unique

84f3ee1

Uses UInt64HashTable to patch a uint64 overflow bug in pd.unique analogous to that seen in Series.unique (patched in pandas-devgh-14915).

gfyoung added a commit to forking-repos/pandas that referenced this pull request Dec 20, 2016

BUG: Use UInt64HashTable in pd.unique

d9b7c84

Uses UInt64HashTable to patch a uint64 overflow bug in pd.unique analogous to that seen in Series.unique (patched in pandas-devgh-14915).

gfyoung mentioned this pull request Dec 21, 2016

BUG: Patch rank() uint64 behavior #14935

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Prevent uint64 overflow in Series.unique #14915

BUG: Prevent uint64 overflow in Series.unique #14915

gfyoung commented Dec 19, 2016

codecov-io commented Dec 19, 2016 •

edited

Loading

jreback Dec 19, 2016

gfyoung Dec 19, 2016

jreback Dec 19, 2016

gfyoung Dec 19, 2016

jreback commented Dec 19, 2016

jreback commented Dec 19, 2016

gfyoung commented Dec 19, 2016 •

edited

Loading

jreback commented Dec 19, 2016

gfyoung commented Dec 19, 2016 •

edited

Loading

jreback commented Dec 19, 2016

gfyoung commented Dec 20, 2016

jreback commented Dec 20, 2016

BUG: Prevent uint64 overflow in Series.unique #14915

BUG: Prevent uint64 overflow in Series.unique #14915

Conversation

gfyoung commented Dec 19, 2016

codecov-io commented Dec 19, 2016 • edited Loading

Current coverage is 84.65% (diff: 100%)

jreback Dec 19, 2016

Choose a reason for hiding this comment

gfyoung Dec 19, 2016

Choose a reason for hiding this comment

jreback Dec 19, 2016

Choose a reason for hiding this comment

gfyoung Dec 19, 2016

Choose a reason for hiding this comment

jreback commented Dec 19, 2016

jreback commented Dec 19, 2016

gfyoung commented Dec 19, 2016 • edited Loading

jreback commented Dec 19, 2016

gfyoung commented Dec 19, 2016 • edited Loading

jreback commented Dec 19, 2016

gfyoung commented Dec 20, 2016

jreback commented Dec 20, 2016

codecov-io commented Dec 19, 2016 •

edited

Loading

gfyoung commented Dec 19, 2016 •

edited

Loading

gfyoung commented Dec 19, 2016 •

edited

Loading