Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating _resolve_numeric_only function of GroupBy #43154

Merged
merged 38 commits into from
Sep 9, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
6b0f5e7
Updating _resolve_numeric_only function of GroupBy
kurchi1205 Aug 21, 2021
116534f
Update groupby.py
kurchi1205 Aug 21, 2021
7b5ecb4
Update test_aggregate.py
kurchi1205 Aug 21, 2021
7faf1fc
Update groupby.py
kurchi1205 Aug 21, 2021
d773e7a
Update groupby.py
kurchi1205 Aug 21, 2021
5b4e799
Update test_aggregate.py
kurchi1205 Aug 21, 2021
3d2e78e
Update groupby.py
kurchi1205 Aug 21, 2021
5836f91
Update groupby.py
kurchi1205 Aug 21, 2021
1eb0e25
Update groupby.py
kurchi1205 Aug 21, 2021
a0391e6
Update test_aggregate.py
kurchi1205 Aug 21, 2021
2a1835a
Update test_aggregate.py
kurchi1205 Aug 21, 2021
66ecb96
Update test_aggregate.py
kurchi1205 Aug 21, 2021
c80fa9e
Update groupby.py
kurchi1205 Aug 22, 2021
95b6533
Update groupby.py
kurchi1205 Aug 22, 2021
fa8a8ca
Update test_aggregate.py
kurchi1205 Aug 22, 2021
e3f6767
Update test_aggregate.py
kurchi1205 Aug 22, 2021
ed668e6
Update groupby.py
kurchi1205 Aug 22, 2021
f30855a
Update groupby.py
kurchi1205 Aug 22, 2021
65261b4
Update groupby.py
kurchi1205 Aug 22, 2021
2477aa3
Update groupby.py
kurchi1205 Aug 22, 2021
da24f29
Update groupby.py
kurchi1205 Aug 22, 2021
4075353
Update groupby.py
kurchi1205 Aug 24, 2021
463a7c9
Update test_aggregate.py
kurchi1205 Aug 24, 2021
dd562c5
Update frame.py
kurchi1205 Aug 24, 2021
f400174
Making the necessary to _resolve_numeric_only and adding testcase
kurchi1205 Aug 29, 2021
56f28d7
Checking for empty DataFrame
kurchi1205 Aug 29, 2021
4fd254b
Commiting
kurchi1205 Aug 30, 2021
a3b09e6
Adding test cases
kurchi1205 Aug 30, 2021
f7e2d59
No changes
kurchi1205 Aug 30, 2021
270549a
Changes in tests and warning
kurchi1205 Sep 1, 2021
17808bf
Solving the errors
kurchi1205 Sep 1, 2021
08b2429
Solving the errors
kurchi1205 Sep 1, 2021
e141123
Merge remote-tracking branch 'upstream/master' into pr43154
Dr-Irv Sep 9, 2021
45f54d6
whatsnew 1.3.3, move tests, restore mypy
Dr-Irv Sep 9, 2021
0012423
add back blank line
Dr-Irv Sep 9, 2021
a9a63e7
Merge branch 'master' into issue_43108
jreback Sep 9, 2021
d46091e
add FutureWarning. Avoid Int64Index
Dr-Irv Sep 9, 2021
a39e5ce
Merge remote-tracking branch 'upstream/master' into issue_43108
Dr-Irv Sep 9, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.3.3.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Fixed regressions
- Fixed regression in :meth:`DataFrame.__getitem__` raising error for slice of :class:`DatetimeIndex` when index is non monotonic (:issue:`43223`)
- Fixed regression in :meth:`.Resampler.aggregate` when used after column selection would raise if ``func`` is a list of aggregation functions (:issue:`42905`)
- Fixed regression in :meth:`DataFrame.corr` where Kendall correlation would produce incorrect results for columns with repeated values (:issue:`43401`)
- Fixed regression in :meth:`DataFrame.groupby` where aggregation on columns with object types dropped results on those columns (:issue:`42395`, :issue:`43108`)
- Fixed regression in :meth:`Series.fillna` raising ``TypeError`` when filling ``float`` ``Series`` with list-like fill value having a dtype which couldn't cast lostlessly (like ``float32`` filled with ``float64``) (:issue:`43424`)
-

Expand Down
8 changes: 8 additions & 0 deletions pandas/core/groupby/groupby.py
Original file line number Diff line number Diff line change
Expand Up @@ -1195,6 +1195,14 @@ def _resolve_numeric_only(self, numeric_only: bool | lib.NoDefault) -> bool:
if self.obj.ndim == 2:
# i.e. DataFrameGroupBy
numeric_only = True
# GH#42395 GH#43108 GH#43154
# Regression from 1.2.5 to 1.3 caused object columns to be dropped
obj = self._obj_with_exclusions
jreback marked this conversation as resolved.
Show resolved Hide resolved
check = obj._get_numeric_data()
if len(obj.columns) and not len(check.columns) and not obj.empty:
numeric_only = False
jreback marked this conversation as resolved.
Show resolved Hide resolved
# TODO: v1.4+ Add FutureWarning

else:
numeric_only = False

Expand Down
3 changes: 2 additions & 1 deletion pandas/tests/groupby/aggregate/test_cython.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,8 @@ def test_cython_agg_nothing_to_agg():

frame = DataFrame({"a": np.random.randint(0, 5, 50), "b": ["foo", "bar"] * 25})

result = frame[["b"]].groupby(frame["a"]).mean()
with tm.assert_produces_warning(FutureWarning, check_stacklevel=False):
result = frame[["b"]].groupby(frame["a"]).mean()
expected = DataFrame([], index=frame["a"].sort_values().drop_duplicates())
tm.assert_frame_equal(result, expected)

Expand Down
62 changes: 62 additions & 0 deletions pandas/tests/groupby/test_groupby.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
MultiIndex,
RangeIndex,
Series,
Timedelta,
Timestamp,
date_range,
read_csv,
Expand Down Expand Up @@ -2392,6 +2393,67 @@ def test_groupby_empty_multi_column(as_index, numeric_only):
tm.assert_frame_equal(result, expected)


def test_groupby_aggregation_non_numeric_dtype():
# GH #43108
df = DataFrame(
[["M", [1]], ["M", [1]], ["W", [10]], ["W", [20]]], columns=["MW", "v"]
)

expected = DataFrame(
{
"v": [[1, 1], [10, 20]],
},
index=Index(["M", "W"], dtype="object", name="MW"),
)

gb = df.groupby(by=["MW"])
result = gb.sum()
tm.assert_frame_equal(result, expected)


def test_groupby_aggregation_multi_non_numeric_dtype():
# GH #42395
df = DataFrame(
{
"x": [1, 0, 1, 1, 0],
"y": [Timedelta(i, "days") for i in range(1, 6)],
"z": [Timedelta(i * 10, "days") for i in range(1, 6)],
}
)

expected = DataFrame(
{
"y": [Timedelta(i, "days") for i in range(7, 9)],
"z": [Timedelta(i * 10, "days") for i in range(7, 9)],
},
index=Index([0, 1], dtype="int64", name="x"),
)

gb = df.groupby(by=["x"])
result = gb.sum()
tm.assert_frame_equal(result, expected)


def test_groupby_aggregation_numeric_with_non_numeric_dtype():
# GH #43108
df = DataFrame(
{
"x": [1, 0, 1, 1, 0],
"y": [Timedelta(i, "days") for i in range(1, 6)],
"z": list(range(1, 6)),
}
)

expected = DataFrame(
{"z": [7, 8]},
index=Index([0, 1], dtype="int64", name="x"),
)

gb = df.groupby(by=["x"])
result = gb.sum()
tm.assert_frame_equal(result, expected)


def test_groupby_filtered_df_std():
# GH 16174
dicts = [
Expand Down