Pandas index type can't be viewed in variable viewer

[mg3146]

Hi, it seems that certain (Int64Index in this case) built-in pandas.Index don't work with the variable viewer.

I'm unsure if this is unique to Spyder 6 (Windows) or all versions (I'm using the standalone spyder 6 installer), and there could be other index types that dont work that I'm unaware of.

Here is a bit of reproducible code:

import pandas as pd
import datetime as dt

cols = pd.Index(range(5))
index = pd.Index(pd.bdate_range(dt.date.today(),periods=10))
data = {key : [i/1.25 for i in range(10)] for key in cols}

df = pd.DataFrame(data, index = index, columns = cols)
print(f'I work!:    {df.columns}')

df.columns = pd.Index([i for i in range(5)])

print(f'I dont work!:    {df.columns}')

(PS. Hopefully I dont make myself look silly by ignoring the very direct "dont report to github" message :) )

[ccordoba12]

Hey @mg3146, thanks for reporting. What's the variable you tried to open in the Variable Explorer which gave you the error above? cols or index?

(PS. Hopefully I dont make myself look silly by ignoring the very direct "dont report to github" message :) )

Don't worry about it, that's fine.

[mg3146]

so in both cases I'm trying to open 'df' in the variable viewer. The difference between the 2 cases is what index type df has.

[ccordoba12]

I can't reproduce this on Linux.

@dalthviz, could you give me a hand reviewing this on Windows? Thanks!

[dalthviz]

Checked running the code provided from a temporary file on a Spyder 6.0.0a1 installation done with the Windows installer but seems like no issue is raised:

To run the code I used the default Python interpreter that comes with Spyder and used the default run configuration.

Installation details:

• Spyder version: 6.0.0a1 0d7fb75 (standalone)
• Python version: 3.9.16 64-bit
• Qt version: 5.15.6
• PyQt5 version: 5.15.7
• Operating System: Windows-10-10.0.19045-SP0

Maybe there are other steps to follow in order to reproduce this @mg3146 ?

[mg3146]

Thanks - I’ll take a look.  Which spyder kernels are you using ?

…

On Aug 8, 2023 at 4:38 PM -0400, Daniel Althviz Moré ***@***.***>, wrote: Checked running the code provided from a temporary file on a Spyder 6.0.0a1 installation done with the Windows installer but seems like no issue is raised: To run the code I used the default Python interpreter that comes with Spyder and used the default run configuration. Installation details: • Spyder version: 6.0.0a1 0d7fb75 (standalone) • Python version: 3.9.16 64-bit • Qt version: 5.15.6 • PyQt5 version: 5.15.7 • Operating System: Windows-10-10.0.19045-SP0 Maybe there are other steps to follow in order to reproduce this @mg3146 ? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[ccordoba12]

3.0b1.

[dalthviz]

Could it be that the error comes from using a custom interpreter?

[mg3146]

This seems to be working for me now - I must have had some conflict or something in my interpeter. Sorry for the windup!

[mg3146]

I should have finished my coffee before looking at this. Indeed its still not working - and yup, I'm using a custom interpreter. Let me know if you want me to post all the packages in it - but a few of the ones below. Spyder is installed with the standalone installer.

spyder-kernels            3.0.0b1                  pypi_0    pypi
ipykernel                 6.19.2           py39hd4e2768_0
ipynb-py-convert          0.4.6              pyh9f0ad1d_0    conda-forge
ipython                   8.14.0                   pypi_0    pypi
ipython_genutils          0.2.0              pyhd3eb1b0_1
ipywidgets                8.0.4            py39haa95532_0
pandas                    1.5.3            py39hf11a4ad_0

[dalthviz]

Not totally sure but maybe a mismatch between the installer pandas version and the one you have installed in your custom interpreter env could be causing the dialog to appear. I think the installers come with pandas 2.0.3 and seems like the pandas version related with your custom environment has pandas 1.5.3. Could it be possible for you to update the pandas version in your custom Python interpreter env from 1.5.3 to 2.0.3?

[mg3146]

its kind of dicey on my end, it’s a production env (I’m not a developer, I’d call myself an end user).  If you have a way of trying it without causing much pain I’d much appreciate it.   But if not, I can try to cobble something together

…

On Aug 9, 2023 at 11:10 AM -0400, Daniel Althviz Moré ***@***.***>, wrote: Not totally sure but maybe a mismatch between the installer pandas version and the one you have installed in your custom interpreter env could be causing the dialog to appear. I think the installers come with pandas 2.0.3 and seems like the pandas version related with your custom environment has pandas 1.5.3. Could it be possible for you to update the pandas version in your custom Python interpreter env from 1.5.3 to 2.0.3? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[ccordoba12]

@dalthviz, could you check this with Pandas 1.5.3? Maybe the problem is that serialization of Dataframes between Pandas 2 and 1.5 is incompatible (we've seen that before a couple of times), so we need to account for that somehow.

[dalthviz]

I was able to reproduce this when using a custom interpreter which has pandas 1.5.3 while using Spyder 6.0.0a1 from the Windows installer:

Seems like the error indeed comes from the serialization. Running Spyder in debug mode I see the following message:

2023-08-09 18:42:41,770 [DEBUG] [spyder_kernels.comms.commbase] -> Exception in cloudpickle.loads : No module named 'pandas.core.indexes.numeric'

[mg3146]

anything else needed from me at this point?

[ccordoba12]

You need to either update Pandas to its 2.0 version in your environment or create a new environment with Spyder 6a1 and Pandas 1.5 and then connect it to your custom env.

[Ax-sudo]

Hi there,
jumping on board of this thread with a question. I had the same problem while running Pandas 2.03 in a custom env, while Pandas 1.5 was installed in base environment. Updating that in base environment solved the problem for me in the custom one. Why would updating packages in the base environment affect the custom one?

Python 3.11.4
Spyder 5.4.3
Windows 10
Anaconda