Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: passing a SparseArray to pd.Index - overly eager FutureWarning #45618

Closed
2 of 3 tasks
jsignell opened this issue Jan 25, 2022 · 1 comment · Fixed by #45668
Closed
2 of 3 tasks

BUG: passing a SparseArray to pd.Index - overly eager FutureWarning #45618

jsignell opened this issue Jan 25, 2022 · 1 comment · Fixed by #45668
Labels
Bug Sparse Sparse Data Type Warnings Warnings that appear or should be added to pandas
Milestone

Comments

@jsignell
Copy link
Contributor

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

In [1]: import pandas as pd
   ...: 
   ...: s = pd.Series(pd.Categorical(["a", "b", "a"], categories=["a", "b", "c"]))
   ...: exp = pd.get_dummies(s, sparse=True)
   ...: exp.sort_values(by=exp.columns.tolist())
<ipython-input-1-312167088340>:5: FutureWarning: In a future version, passing a SparseArray to pd.Index will store that array directly instead of converting to a dense numpy ndarray. To retain the old behavior, use pd.Index(arr.to_numpy()) instead
  exp.sort_values(by=exp.columns.tolist())
<ipython-input-1-312167088340>:5: FutureWarning: In a future version, passing a SparseArray to pd.Index will store that array directly instead of converting to a dense numpy ndarray. To retain the old behavior, use pd.Index(arr.to_numpy()) instead
  exp.sort_values(by=exp.columns.tolist())
<ipython-input-1-312167088340>:5: FutureWarning: In a future version, passing a SparseArray to pd.Index will store that array directly instead of converting to a dense numpy ndarray. To retain the old behavior, use pd.Index(arr.to_numpy()) instead
  exp.sort_values(by=exp.columns.tolist())
Out[1]: 
   a  b  c
1  0  1  0
0  1  0  0
2  1  0  0

Issue Description

In pandas 1.4.0 there is a FutureWarning that I believe is too aggressive. In the example above it is not clear what to change in the code snippet. For context this came up in the dask test suite.

Expected Behavior

I imagine there might be some internal code that is coercing the array that I provided into an index. It this is the expected behavior then I think the error message needs to be changed to make it clear how to fix the code snippet.

Installed Versions

INSTALLED VERSIONS

commit : bb1f651
python : 3.9.9.final.0
python-bits : 64
OS : Linux
OS-release : 5.11.0-7633-generic
Version : #35163010093020.04~ae2753e-Ubuntu SMP Mon Aug 30 18:23:52 UTC
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.4.0
numpy : 1.21.5
pytz : 2021.3
dateutil : 2.8.2
pip : 21.3.1
setuptools : 59.8.0
Cython : None
pytest : 6.2.5
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.3
IPython : 8.0.1
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : 0.7.2
fsspec : 2022.01.0
gcsfs : None
matplotlib : 3.5.1
numba : 0.55.0
numexpr : 2.8.0
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 6.0.1
pyreadstat : None
pyxlsb : None
s3fs : 2022.01.0
scipy : 1.7.3
sqlalchemy : None
tables : 3.6.1
tabulate : None
xarray : 0.20.2
xlrd : None
xlwt : None
zstandard : None

@jsignell jsignell added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jan 25, 2022
@phofl phofl added the Sparse Sparse Data Type label Jan 25, 2022
@phofl phofl added this to the 1.4.1 milestone Jan 25, 2022
@jorisvandenbossche
Copy link
Member

This seems to be because in the lexsort code, we create a Categorical from the column to sort by, which in turn will create an Index from the unique values of that column, in this case from a sparse array:

for k, order in zip(keys, orders):
cat = Categorical(k, ordered=True)

So we will need to silence the warnings there.

@jorisvandenbossche jorisvandenbossche removed the Needs Triage Issue that has not been reviewed by a pandas team member label Jan 25, 2022
@simonjayhawkins simonjayhawkins added the Warnings Warnings that appear or should be added to pandas label Feb 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Sparse Sparse Data Type Warnings Warnings that appear or should be added to pandas
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants