API: Series[bool][key] = np.nan -> cast to object #38709

jbrockmendel · 2020-12-27T00:16:16Z

closes API: setting np.nan into Series[bool] #38692
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

sits on top of #38688

…g-block-setitem

jbrockmendel · 2021-01-27T01:57:59Z

yikes, the existing behavior is even worse than i thought:

ser = pd.Series([True, False])
ser2 = ser.copy()

ser.iloc[1] = np.nan

ser2.iloc[1] = np.float64("NAN")

>>> ser
0    1.0
1    NaN
dtype: float64

>>> ser2
0    True
1     NaN
dtype: object

This fixes that; ill add a test. gentle ping @jreback this is probably the toughest of the outstanding setitem-related PRs

jreback

I am ok with this change as long as we have a sub-section in api breaking.

jreback · 2021-01-27T14:20:05Z

pandas/core/array_algos/putmask.py

@@ -106,8 +106,11 @@ def putmask_smart(values: np.ndarray, mask: np.ndarray, new) -> np.ndarray:
        # preserves dtype if possible
        return _putmask_preserve(values, new, mask)

-    # change the dtype if needed
-    dtype, _ = maybe_promote(new.dtype)
+    if values.dtype == bool and new.dtype.kind == "f":


hmm, shouldn't maybe_promote handle this? (`maybe_promote(new.dtype, values.dtype) ?

wouldnt find_common_type make more sense?

yah this is nicer, updated

…g-block-setitem

jbrockmendel · 2021-01-27T17:40:48Z

have a sub-section in api breaking.

added + green

jreback · 2021-01-28T03:30:00Z

thanks, very nice

jbrockmendel · 2021-01-28T03:52:07Z

way cool. between this and #39163 ill be un-blocked on unifying a bunch of casting behavior i think

simonjayhawkins · 2021-06-29T13:42:00Z

The appears to be a performance regression here https://pandas.pydata.org/speed/pandas/#frame_methods.MaskBool.time_frame_mask_bools

There has been some improvement since.

(pandas-dev) simon@T3630:~/pandas/asv_bench (master)$ asv compare v1.2.5 1.3.x |grep time_frame_mask_bools
+     5.62±0.09ms       30.8±0.5ms     5.49  frame_methods.MaskBool.time_frame_mask_bools

profile

(pandas-dev) simon@T3630:~/pandas/asv_bench (master)$ asv profile frame_methods.MaskBool.time_frame_mask_bools 3217b436c754bdf62081408f0417ff2061702e75|head -n 30
· Discovering benchmarks
·· Uninstalling from conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
·· Installing 3217b436 <v1.3.0rc0~1332> into conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
· Profile data does not already exist. Running profiler now.
·· Benchmarking conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
··· Running (frame_methods.MaskBool.time_frame_mask_bools--).
··· frame_methods.MaskBool.time_frame_mask_bools            42.4±0.5ms

Tue Jun 29 14:36:40 2021    /tmp/tmpoghq5rs1

         41910 function calls (41895 primitive calls) in 0.051 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.051    0.051 {built-in method builtins.exec}
        1    0.000    0.000    0.051    0.051 /home/simon/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py:540(method_caller)
        1    0.001    0.001    0.051    0.051 /home/simon/pandas/asv_bench/benchmarks/frame_methods.py:307(time_frame_mask_bools)
        1    0.000    0.000    0.051    0.051 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/generic.py:9116(mask)
        1    0.000    0.000    0.051    0.051 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/generic.py:8963(where)
        1    0.000    0.000    0.051    0.051 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/generic.py:8801(_where)
        5    0.000    0.000    0.049    0.010 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/internals/managers.py:380(apply)
        1    0.000    0.000    0.048    0.048 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/internals/managers.py:548(where)
      2/1    0.000    0.000    0.047    0.047 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/internals/blocks.py:1259(where)
        1    0.000    0.000    0.037    0.037 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/internals/blocks.py:544(_maybe_downcast)
        1    0.000    0.000    0.037    0.037 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/internals/blocks.py:551(<listcomp>)
        1    0.000    0.000    0.037    0.037 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/internals/blocks.py:553(downcast)
        1    0.001    0.001    0.037    0.037 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/internals/blocks.py:474(split_and_operate)
      500    0.000    0.000    0.021    0.000 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/internals/blocks.py:583(f)
      500    0.001    0.000    0.021    0.000 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/dtypes/cast.py:192(maybe_downcast_to_dtype)

commit before

(pandas-dev) simon@T3630:~/pandas/asv_bench (master)$ asv profile frame_methods.MaskBool.time_frame_mask_bools 3217b436c754bdf62081408f0417ff2061702e75~1|head -n 30
· Discovering benchmarks
·· Uninstalling from conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
·· Installing 6e579ed3 <v1.3.0rc0~1333> into conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
· Profile data does not already exist. Running profiler now.
·· Benchmarking conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
··· Running (frame_methods.MaskBool.time_frame_mask_bools--).
··· frame_methods.MaskBool.time_frame_mask_bools           5.29±0.03ms

Tue Jun 29 14:37:42 2021    /tmp/tmpqkntmn4t

         10274 function calls (10261 primitive calls) in 0.007 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.007    0.007 {built-in method builtins.exec}
        1    0.000    0.000    0.007    0.007 /home/simon/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py:540(method_caller)
        1    0.000    0.000    0.007    0.007 /home/simon/pandas/asv_bench/benchmarks/frame_methods.py:307(time_frame_mask_bools)
        1    0.000    0.000    0.007    0.007 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/generic.py:9116(mask)
        1    0.000    0.000    0.006    0.006 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/generic.py:8963(where)
        1    0.000    0.000    0.006    0.006 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/generic.py:8801(_where)
        5    0.000    0.000    0.005    0.001 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/internals/managers.py:380(apply)
        1    0.000    0.000    0.004    0.004 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/internals/managers.py:548(where)
        1    0.000    0.000    0.004    0.004 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/internals/blocks.py:1277(where)
      501    0.000    0.000    0.001    0.000 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/dtypes/common.py:1354(is_bool_dtype)
        1    0.001    0.001    0.001    0.001 {method 'take' of 'numpy.ndarray' objects}
        1    0.000    0.000    0.001    0.001 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/computation/expressions.py:239(where)
        1    0.000    0.000    0.001    0.001 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/computation/expressions.py:165(_where_numexpr)
        1    0.001    0.001    0.001    0.001 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/numexpr/necompiler.py:767(evaluate)
        1    0.000    0.000    0.001    0.001 /home/simon/pandas/asv_bench/env/11a1c20ede452de2525075dc4a15eb94/lib/python3.8/site-packages/pandas/core/internals/blocks.py:1351(<listcomp>)

I guess this is expected?

jbrockmendel · 2021-06-30T14:39:48Z

I guess this is expected?

Can't comment on the size of the perf hit, but the existence is unsurprising since we're casting to object

jbrockmendel and others added 7 commits December 24, 2020 13:47

TST: implement tm.check_setitem_equivalents

6ee5c5f

re-write as fixturized

33e39f9

CLN: algos.searchsorted (pandas-dev#38686)

5896615

REF: simplify coerce_to_target_dtype (pandas-dev#38683)

90a76ce

PERF: fix assert_frame_equal can be very slow (pandas-dev#38202)

0827cd0

API: cast to object when setting np.nan into Series[bool]

803f495

Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…

055617b

…g-block-setitem

jreback added the Indexing Related to indexing on series/frames, not to indexes themselves label Dec 27, 2020

Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…

3be818f

…g-block-setitem

jbrockmendel mentioned this pull request Dec 28, 2020

API: setting np.nan into Series[bool] #38692

Closed

jbrockmendel added 9 commits December 28, 2020 19:07

Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…

51ad10d

…g-block-setitem

Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…

279564b

…g-block-setitem

Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…

e3b2cec

…g-block-setitem

Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…

c4eb8e1

…g-block-setitem

Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…

ce4311a

…g-block-setitem

Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…

9e5cb27

…g-block-setitem

fix putmask

c01661d

Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…

23c69dc

…g-block-setitem

Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…

198768e

…g-block-setitem

jbrockmendel mentioned this pull request Jan 19, 2021

API: setitem copy/view behavior ndarray vs Categorical vs other EA #38896

Closed

jbrockmendel added 2 commits January 19, 2021 08:37

Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…

7c6fdb7

…g-block-setitem

Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…

e0c0194

…g-block-setitem

jbrockmendel mentioned this pull request Jan 20, 2021

BUG: setting dt64 values into Series[int] incorrectly casting dt64->int #39266

Merged

4 tasks

jbrockmendel added 2 commits January 25, 2021 10:57

Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…

f02f514

…g-block-setitem

Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…

0934ad1

…g-block-setitem

jreback requested changes Jan 27, 2021

View reviewed changes

jbrockmendel added 2 commits January 27, 2021 07:53

Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…

d966441

…g-block-setitem

whatsnew, test, maybe_promote->find_common_type

d2ac2b3

jreback added this to the 1.3 milestone Jan 28, 2021

jreback approved these changes Jan 28, 2021

View reviewed changes

jreback merged commit 3217b43 into pandas-dev:master Jan 28, 2021

jbrockmendel deleted the bug-block-setitem branch January 28, 2021 03:53

simonjayhawkins mentioned this pull request Jul 4, 2021

RLS: 1.3.1 #42343

Closed

ggold7046 mentioned this pull request Aug 10, 2023

Modified doc/make.py to run sphinx-build -b linkcheck #54265

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API: Series[bool][key] = np.nan -> cast to object #38709

API: Series[bool][key] = np.nan -> cast to object #38709

jbrockmendel commented Dec 27, 2020

jbrockmendel commented Jan 27, 2021

jreback left a comment

jreback Jan 27, 2021

jbrockmendel Jan 27, 2021

jreback Jan 27, 2021

jbrockmendel Jan 27, 2021

jbrockmendel commented Jan 27, 2021

jreback commented Jan 28, 2021

jbrockmendel commented Jan 28, 2021

simonjayhawkins commented Jun 29, 2021

jbrockmendel commented Jun 30, 2021

API: Series[bool][key] = np.nan -> cast to object #38709

API: Series[bool][key] = np.nan -> cast to object #38709

Conversation

jbrockmendel commented Dec 27, 2020

jbrockmendel commented Jan 27, 2021

jreback left a comment

Choose a reason for hiding this comment

jreback Jan 27, 2021

Choose a reason for hiding this comment

jbrockmendel Jan 27, 2021

Choose a reason for hiding this comment

jreback Jan 27, 2021

Choose a reason for hiding this comment

jbrockmendel Jan 27, 2021

Choose a reason for hiding this comment

jbrockmendel commented Jan 27, 2021

jreback commented Jan 28, 2021

jbrockmendel commented Jan 28, 2021

simonjayhawkins commented Jun 29, 2021

jbrockmendel commented Jun 30, 2021