Inconsistent Return Types Between numpy 1.26.4 and numpy 2.1.0 in pandas 2.2.2

[xinrong-meng]

When using numpy 2.1.0, certain methods in pandas (e.g. first_valid_index() and .at[] access) return numpy.int64 instead of plain Python integers as seen when using numpy 1.26.4. This creates inconsistencies in behavior.

To reproduce

1. Environment 1: Python 3.10, pandas 2.2.2, numpy 1.26.4

conda create -n pd_np_1 python=3.10 pandas=2.2.2 numpy=1.26.4

2. Environment 2: Python 3.10, pandas 2.2.2, numpy 2.1.0

conda create -n pd_np_2 python=3.10 pandas=2.2.2 -c conda-forge
conda activate pd_np_2
pip install numpy==2.1.0

import pandas as pd
import numpy as np

pd.Series([None, None, 3], index=[1, 2, 3]).first_valid_index()
pd.DataFrame([[0, 1, 2]], index=['a'], columns=['A', 'B', 'C']).at['a', 'A']

Issue

Environment 1 (numpy 1.26.4)

>>> pd.Series([None, None, 3], index=[1, 2, 3]).first_valid_index()
3
>>> pd.DataFrame([[0, 1, 2]], index=['a'], columns=['A', 'B', 'C']).at['a', 'A']
0

Environment 2 (numpy 2.1.0)

>>> pd.Series([None, None, 3], index=[1, 2, 3]).first_valid_index()
np.int64(3)
>>> pd.DataFrame([[0, 1, 2]], index=['a'], columns=['A', 'B', 'C']).at['a', 'A']
np.int64(0)

Discusion

Is this intended behavior, or is it a compatibility issue between pandas 2.2.2 and numpy 2.1.0?

[asishm]

Thanks for the report, the result in both cases are numpy integers.

If you check type(pd.Series([None, None, 3], index=[1, 2, 3]).first_valid_index()) in both, you'll see that they are np.int64.

numpy 2.0 has changed the representation of scalars: see https://numpy.org/devdocs/release/2.0.0-notes.html#representation-of-numpy-scalars-changed

[xinrong-meng]

Thank you @asishm ! That really makes sense. Closing the issue.