BUG: Summing a series of bools doesn't return a number

[marberts]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

pd.Series([True, False], dtype="object").cumsum()

Issue Description

Certain mathematical operations (e.g., sum, prod) on a series with a single bool that has an object dtype return a bool instead of an int. Not all operations do this (e.g., mean, median).

>>> pd.Series([True], dtype="object").sum()
True
>>> pd.Series([True, False], dtype="object").sum()
1

>>> pd.Series([True], dtype="object").mean()
1.0

Cumsum and cumprod are a bit more interesting, as the first element in the resulting series is a bool.

>>> pd.Series([True, False], dtype="object").cumsum()
0    True
1       1
dtype: object

Expected Behavior

I expect the results to be the same as when the dtype is bool, e.g.,

>>> pd.Series([True, False]).cumsum()
0    1
1    1
dtype: int64

Installed Versions

INSTALLED VERSIONS

commit : e8093ba
python : 3.9.12.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-125-generic
Version : #141-Ubuntu SMP Wed Aug 10 13:42:03 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_CA.UTF-8
LOCALE : en_CA.UTF-8

pandas : 1.4.3
numpy : 1.22.3
pytz : 2022.1
dateutil : 2.8.2
setuptools : 59.4.0
pip : 21.3.1
Cython : None
pytest : None
hypothesis : None
sphinx : 5.0.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.3
IPython : 7.33.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli :
fastparquet : None
fsspec : None
gcsfs : None
markupsafe : 2.1.1
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.8.0
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None

[vorbrodt]

Do you think this issue could be suitable for a new contributor? If so, I would like to be assigned to it and give it a try.

[mzeitlin11]

Thanks for reporting this issue @marberts! @vorbrodt I'd recommend first waiting for consensus on what the correct output should be - I'd guess this result just comes from deferring to numpy:

>>> np.array([True, False], dtype="object").cumsum()
array([True, 1], dtype=object)

[asishm]

It also is consistent with stdlib itertools.accumulate

In [28]: list(itertools.accumulate([True, False]))
Out[28]: [True, 1]

[marberts]

Good point, @asishm, but the standard library doesn't produce this behavior consistently.

>>> pd.Series([True], dtype="object").prod()
True

>>> pd.Series([True]).prod()
1

>>> math.prod([True])
1

The same result is had with the built-in sum as well. (The result for accumulate makes sense, though, as the return value is documented as [True, True + False].)

I think this is particularly tricky with a bool series that has missing values, as the dtype will be object instead of bool. Logically,

pd.DataFrame({
    "x": [True, False, True, pd.NA], 
    "g": ["a", "a", "b", "b"]
}).groupby("g").sum()

should return the same value as

pd.DataFrame({
    "x": [True, False, True], 
    "g": ["a", "a", "b"]
}).groupby("g").sum()

[marberts]

@mzeitlin11 You're right, this is coming from numpy, and is documented in the docstring for np.ufunc.reduce. In particular, the initial value is set to None by default when the dtype is object, leading to the same behavior as accumulate above. Explicitly setting the initial value or dtype gives the expected result.

>>> np.array([True], dtype="object").sum(initial=0)
1

>>> np.array([True, False]).cumsum(dtype="int")
array([1, 1])

Neither of these arguments are available in the pandas equivalents.

[mzeitlin11]

I think this is particularly tricky with a bool series that has missing values, as the dtype will be object instead of bool

Missing values in object series are tricky (#32931). For a series of missing and boolean values, you can use the nullable boolean type (https://pandas.pydata.org/docs/user_guide/boolean.html) which should align closer to the behavior you're looking for.

In the future, a Series like you constructed above may be inferred as boolean instead of object, xref #33662

[marberts]

@mzeitlin11 Nullable booleans do seem to be what I want. Thanks for the advice. I'm new to pandas, so it's much appreciated.

[mzeitlin11]

@marberts going to close then to keep future discussion around the other issues mentioned. If you run into any issues using nullable booleans, feel free to open a new issue of course!