BUG: Sparse master issue

[kawochen]

These other issues will be fixed by this

• Support dtypes other than float in sparse data structures #667 other dtypes PR ENH: Sparse int64 and bool dtype support enhancement #13849
• non-NDFFrame object error using pandas.SparseSeries.from_coo() function #10818 from_coo issues (same as TypeError in SparseSeries.__repr__ when series longer than max_rows #10560)
• BUG: sparse empty construction buggy #10079 empty construction
• TypeError in SparseSeries.__repr__ when series longer than max_rows #10560 __repr__ broken, PR BUG: Sparse misc fixes including __repr__ #12779
• Concat does not work for sparse series #10536 concat SparseSeries, PR BUG: SparseSeries concat results in dense #12844
• concat() on Sparse dataframe returns strange results #12174 concat SparseDataFrame? PR BUG: Sparse concat may fill fill_value with NaN #12966
• TST: more testing of indexing with sparse #4400 more testing of indexing (TST: Add more Sparse indexing tests #12848)
• API, DOC: SparseArray Interface and Documentation is Confusing #12794 more/better sparse docs PR b13ddd5
• API: Sparse Return Types #12855 return types
• BUG: SparseSeries/DataFrame non-float dtypes repr #13110 non float repr PR ENH: Sparse int64 and bool dtype support enhancement #13849
• _combine_const() in pandas.sparse.frame does not have uniform method signature with pandas.core.frame #13001 signatures PR ENH: Sparse int64 and bool dtype support enhancement #13849
• SparseSeries throws a ValueError exception when printing large arrays #13144 repr of large arrays (BUG: Misc fixes for SparseSeries indexing with MI #13163)
• SparseSeries with MultiIndex indexing (BUG: Misc fixes for SparseSeries indexing with MI #13163)
• SparseDataFrame/Series with MultiIndex indexing (Wrong result of pandas.sparse.series.SparseSeries.loc with indexer of length 1 #15447)
• SparseSeries with non-unique index indexing
• SparseDataFrame with non-unique index indexing
• slicing / reindexing SparseDataFrame to return subclass (BUG: Slicing subclasses of SparseDataFrames. #13787)
• concat (Concatting dense and sparse dataframes breaks many common operations #16874)

The fundamental issue is that slicing is broken in sparse.

In [11]: s = Series([1]+[np.nan]*5).to_sparse()

In [12]: s
Out[12]: 
0     1
1   NaN
2   NaN
3   NaN
4   NaN
5   NaN
dtype: float64
BlockIndex
Block locations: array([0], dtype=int32)
Block lengths: array([1], dtype=int32)

In [13]: s.iloc[0:3]
Out[13]: 
[1.0, nan, nan]
Fill: nan
BlockIndex
Block locations: array([0], dtype=int32)
Block lengths: array([1], dtype=int32)

In [14]: type(s.iloc[0:3])
Out[14]: pandas.sparse.array.SparseArray

In [15]: type(s)
Out[15]: pandas.sparse.series.SparseSeries
[14] should be a SparseSeries (not SparseArray which is the underlying object the SparseSeries holds). its not getting wrapped when sliced somewhere.

[jreback]

not sure why this is an issue
can u show an example?

[kawochen]

For Series we get back the same type:

>>> type(Series([1, 2, 3]).iloc[1:2])
<class 'pandas.core.series.Series'>
>>> type(Series([1, 2, 3]).ix[1:2])
<class 'pandas.core.series.Series'>

>>> concat([Series([1, 2, 3]).iloc[0:1], Series([1, 2, 3]).iloc[2:3]])
0    1
2    3
dtype: int64
>>> concat([SparseSeries([1, 2, 3]).iloc[0:1], SparseSeries([1, 2, 3]).iloc[2:3]])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kawoc/pandas-kawochen/pandas/tools/merge.py", line 755, in concat
    copy=copy)
  File "/home/kawoc/pandas-kawochen/pandas/tools/merge.py", line 806, in __init__
    raise TypeError("cannot concatenate a non-NDFrame object")
TypeError: cannot concatenate a non-NDFrame object

[jreback]

well are the same issue then

[kawochen]

hmm I suppose so, since that was the issue I was looking at. I checked the history and the difference between SparseSeries's fastpath and Series's fastpath seems deliberate (and tests do fail if I just make them the same), so I just wanted to point out SparseSeries hits paths that are not covered in tests, and we see things like SparseSeries([1,2,3]).iloc[0:3] and SparseSeries([1,2,3]).iloc[:] being different.

[kawochen]

#10079 is also the same issue (construction doesn't error, .__repr__ does, but not for smaller frames)

[jreback]

@gfyoung hole host of issues w.r.t. sparse, note @sinhrks has recently fixed some of these

[gfyoung]

@jreback : Maybe I'll join the party here once I "finish" with fromnumeric.py. 😄 Btw, I think scipy.sparse might be a useful resource in terms of getting ideas about functions, as they do a lot of their implementation in pure Python as well.

[jreback]

I don't think I made an issue for this, but we are open to taking sparse/array.py (and limited iinfrastructure), and making a pandas-sparse package to (that pandas would then depend). This would simplify the interface/API.

[gfyoung]

#667, #12794, #13001, and #13110 have all been closed for some time now. Should all be checked off.