Skip to content

Commit

Permalink
BUG: fix issue with sparse concatting
Browse files Browse the repository at this point in the history
This was originally brought up in :issue:`18686` and :issue:`18914`.
Basically the problem is when you use get_dummies with sparse=True it
will return a SparseDataFrame with sparse and dense columns. This is in
fact not what we want. What we want is a DataFrame with sparse and dense
columns.

Inside of pandas.core.dtypes.concat is a function that defines the
factory class which needed to be changed.
  • Loading branch information
hexgnu committed Dec 28, 2017
1 parent dbec3c9 commit a37ea0c
Show file tree
Hide file tree
Showing 4 changed files with 10 additions and 2 deletions.
1 change: 1 addition & 0 deletions doc/source/whatsnew/v0.23.0.txt
Original file line number Diff line number Diff line change
Expand Up @@ -353,6 +353,7 @@ Reshaping
- Bug in :func:`Series.rank` where ``Series`` containing ``NaT`` modifies the ``Series`` inplace (:issue:`18521`)
- Bug in :func:`cut` which fails when using readonly arrays (:issue:`18773`)
- Bug in :func:`Dataframe.pivot_table` which fails when the ``aggfunc`` arg is of type string. The behavior is now consistent with other methods like ``agg`` and ``apply`` (:issue:`18713`)
- Bug in :func:`pandas.core.dtypes.concat._get_series_result_type` which returns SparseDataFrame even if not all contained in Frame are sparse. (:issue:`18914`)


Numeric
Expand Down
2 changes: 1 addition & 1 deletion pandas/core/dtypes/concat.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ def _get_frame_result_type(result, objs):
if any block is SparseBlock, return SparseDataFrame
otherwise, return 1st obj
"""
if any(b.is_sparse for b in result.blocks):
if all(b.is_sparse for b in result.blocks):
from pandas.core.sparse.api import SparseDataFrame
return SparseDataFrame
else:
Expand Down
1 change: 0 additions & 1 deletion pandas/core/sparse/series.py
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,6 @@ def __init__(self, data=None, index=None, sparse_index=None, kind='block',
if index is None:
index = data.index.view()
else:

data = data.reindex(index, copy=False)

else:
Expand Down
8 changes: 8 additions & 0 deletions pandas/tests/reshape/test_reshape.py
Original file line number Diff line number Diff line change
Expand Up @@ -454,6 +454,14 @@ def test_dataframe_dummies_preserve_categorical_dtype(self, dtype):

tm.assert_frame_equal(result, expected)

def test_get_dummies_dont_sparsify_all_columns(self):
# GH18914
df = DataFrame.from_items([('GDP', [1, 2]), ('Nation', ['AB', 'CD'])])
df = get_dummies(df, columns=['Nation'], sparse=True)
df2 = df.reindex(columns=['GDP'])

tm.assert_index_equal(df.index, df2.index)


class TestCategoricalReshape(object):

Expand Down

0 comments on commit a37ea0c

Please sign in to comment.