BUG: DataFrame with Int64 columns casts to float64 with .max()/.min() #34210

simonjayhawkins · 2020-05-16T15:35:35Z

closes DataFrame with Int64 columns casts to float64 with .max()/.min() #32651
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

simonjayhawkins · 2020-05-16T17:30:17Z

It looks like #25777 needs to be fixed first, so closing to clear queue.

pandas/core/arrays/sparse/array.py

jreback · 2020-05-17T22:06:06Z

pandas/core/frame.py

-        if numeric_only is not None and axis in [0, 1]:
+        if (
+            self._is_homogeneous_type and self._mgr.any_extension_types and axis == 0
+        ) or (numeric_only is not None and axis in [0, 1]):


the axis in [0, 1] is superfluous

test_any_all_np_func passes axis=None for reduction along both axes.

jreback · 2020-05-17T22:06:38Z

pandas/core/frame.py

@@ -8419,7 +8419,9 @@ def _get_data(axis_matters):
                raise NotImplementedError(msg)
            return data

-        if numeric_only is not None and axis in [0, 1]:
+        if (
+            self._is_homogeneous_type and self._mgr.any_extension_types and axis == 0


do we really need all of these conditions?

no I don't think so, as this is hiding some bugs (or inconsistencies)

>>> import pandas as pd >>> >>> pd.__version__ '1.1.0.dev0+1600.gdede2c7a1e' >>> >>> df = pd.DataFrame({"A": pd.Series([0, 1], dtype="category")}) >>> df A 0 0 1 1 >>> >>> df.all() A False dtype: bool >>> >>> df.A.all() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Users\simon\pandas\pandas\core\generic.py", line 11394, in logical_func return self._reduce( File "C:\Users\simon\pandas\pandas\core\series.py", line 4028, in _reduce return delegate._reduce(name, skipna=skipna, **kwds) File "C:\Users\simon\pandas\pandas\core\arrays\categorical.py", line 2076, in _reduce raise TypeError(f"Categorical cannot perform the operation {name}") TypeError: Categorical cannot perform the operation all >>>

will need to fix #21020 first

…integerarray

…ataframes

simonjayhawkins · 2020-05-18T17:51:20Z

@jorisvandenbossche @jreback This PR in the current state would close #32651

BUT the issue would persist for DataFrames with non-numeric columns.

I see the benefit of the changes here is consistency of the code path taken regardless of numeric_only parameter for numeric only DataFrames. I can't find an issue for this, but could potentially concoct one.

There is overlap here with #32867 and #32867 would likely close #32651 anyway.

I'll leave this as draft for now, or could close to clear queue if preferred.

in the meantime, I assume investigating #21020 would also benefit #32867.

There's also an issue with PeriodArray raising TypeError: mean is not implemented for PeriodArray since the meaning is ambiguous when dispatching to the EA which is currently ignored with DataFrame.mean containing Period column that I could also look into.

>>> df = pd.DataFrame(
...     {
...         "A": np.arange(3),
...         "B": pd.date_range("2016-01-01", periods=3),
...         "C": pd.timedelta_range("1D", periods=3),
...         "D": pd.period_range("2016", periods=3, freq="A"),
...     }
... )
>>> df.mean()
<stdin>:1: FutureWarning: DataFrame.mean and DataFrame.median with numeric_only=None will include datetime64 and datetime64tz columns in a futu
re version.
A                  1
C    2 days 00:00:00
dtype: object
>>>

I guess we want to maintain the same behaviour. or is inconsistent behaviour not desirable.

>>> df = pd.DataFrame(
...     {
...         "D": pd.period_range("2016", periods=3, freq="A"),
...     }
... )
>>>
>>> df.mean()
Series([], dtype: float64)
>>>
>>> df.D.mean()
Traceback (most recent call last):
...
TypeError: mean is not implemented for PeriodArray since the meaning is ambiguous.  An alternative is obj.to_timestamp(how='start').mean()
>>>

jorisvandenbossche · 2020-05-18T17:55:28Z

I just wanted to comment that this overlaps with #32867, as you also just mentioned.

Regarding mean of periods, see also #33758

jreback

looks fine, just some questions.

jreback · 2020-05-25T22:10:50Z

pandas/core/frame.py

@@ -8441,6 +8443,10 @@ def blk_func(values):
            assert isinstance(res, dict)
            if len(res):
                assert len(res) == max(list(res.keys())) + 1, res.keys()
+            elif not out_dtype:
+                # The default dtype for empty Series will be 'object' instead of
+                #  'float64' in a future version.


do we have tests which currently hit this case?

this was added because of test failures in tests/frame/test_analytics.py (maybe others)

========================================================================= short test summary info ========================================================================== FAILED pandas/tests/frame/test_analytics.py::TestDataFrameAnalytics::test_stat_op_calc - AssertionError: Caused unexpected warning(s): [('DeprecationWarning', Deprecation... FAILED pandas/tests/frame/test_analytics.py::TestDataFrameAnalytics::test_median - AssertionError: Caused unexpected warning(s): [('DeprecationWarning', DeprecationWarnin... FAILED pandas/tests/frame/test_analytics.py::TestDataFrameAnalytics::test_mean_excludes_datetimes[None] - AssertionError: Caused unexpected warning(s): [('DeprecationWarn... FAILED pandas/tests/frame/test_analytics.py::TestDataFrameAnalytics::test_mean_excludes_datetimes[UTC] - AssertionError: Caused unexpected warning(s): [('DeprecationWarni...

do we have tests which currently hit this case?

so I guess on master, no tests hit this case

these cases were for empty DataFrames (after removing timestamp columns). using all for is_numeric was returning True for empty iterable (_mgr.blocks). changing is_numeric and this elif not out_dtype code block doesn't need to be added.

jreback · 2020-05-25T22:12:20Z

pandas/core/frame.py

@@ -8419,7 +8419,9 @@ def _get_data(axis_matters):
                raise NotImplementedError(msg)
            return data

-        if numeric_only is not None and axis in [0, 1]:
+        is_numeric = all(b.is_numeric for b in self._mgr.blocks)


do the added tests hit this sufficiently (e.g. some blocks numeric some blocks not)

no, but that is another issue #34520

jreback · 2020-06-14T22:53:53Z

can you merge master and update to comments

jorisvandenbossche · 2020-06-15T07:11:11Z

I think we should rather fix this in #32867 (the test could still be useful though, so that could be merged here as an xfail, or I can move it to that PR)

…integerarray

simonjayhawkins · 2020-07-10T09:14:27Z

I think we should rather fix this in #32867 (the test could still be useful though, so that could be merged here as an xfail, or I can move it to that PR)

closing in favour of #32867

BUG: DataFrame with Int64 columns casts to float64 with .max()/.min()

9728d81

simonjayhawkins added Bug NA - MaskedArrays Related to pd.NA and nullable extension arrays labels May 16, 2020

simonjayhawkins closed this May 16, 2020

temp fix

7765bac

simonjayhawkins reopened this May 17, 2020

simonjayhawkins commented May 17, 2020

View reviewed changes

pandas/core/arrays/sparse/array.py Outdated Show resolved Hide resolved

simonjayhawkins marked this pull request as draft May 17, 2020 11:28

jreback requested changes May 17, 2020

View reviewed changes

simonjayhawkins added 6 commits May 18, 2020 13:25

Merge remote-tracking branch 'upstream/master' into dataframe-minmax-…

c58e85d

…integerarray

Merge remote-tracking branch 'upstream/master' into dataframe-minmax-…

af366b8

…integerarray

changes condition

4a14032

make results consistent for numeric_only parameter and numeric_only d…

e4e6c8e

…ataframes

black fixup

9b5f41c

comments

e704f51

jreback reviewed May 25, 2020

View reviewed changes

simonjayhawkins added 3 commits June 30, 2020 10:46

Merge remote-tracking branch 'upstream/master' into dataframe-minmax-…

60ab03a

…integerarray

change is_numeric to be False on empty DataFrame

7b18d89

reorder condition

e850108

simonjayhawkins closed this Jul 10, 2020

simonjayhawkins mentioned this pull request Jul 15, 2020

BUG: Use correct ExtensionArray reductions in DataFrame reductions #35254

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: DataFrame with Int64 columns casts to float64 with .max()/.min() #34210

BUG: DataFrame with Int64 columns casts to float64 with .max()/.min() #34210

simonjayhawkins commented May 16, 2020

simonjayhawkins commented May 16, 2020

jreback May 17, 2020

simonjayhawkins May 18, 2020

jreback May 17, 2020

simonjayhawkins May 18, 2020

simonjayhawkins May 18, 2020

simonjayhawkins commented May 18, 2020

jorisvandenbossche commented May 18, 2020

jreback left a comment

jreback May 25, 2020

simonjayhawkins Jun 30, 2020

simonjayhawkins Jun 30, 2020

simonjayhawkins Jun 30, 2020

jreback May 25, 2020

simonjayhawkins Jun 30, 2020

jreback commented Jun 14, 2020

jorisvandenbossche commented Jun 15, 2020

simonjayhawkins commented Jul 10, 2020

BUG: DataFrame with Int64 columns casts to float64 with .max()/.min() #34210

BUG: DataFrame with Int64 columns casts to float64 with .max()/.min() #34210

Conversation

simonjayhawkins commented May 16, 2020

simonjayhawkins commented May 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonjayhawkins commented May 18, 2020

jorisvandenbossche commented May 18, 2020

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jun 14, 2020

jorisvandenbossche commented Jun 15, 2020

simonjayhawkins commented Jul 10, 2020