BUG: behaviour of fillna(NaN) for nullable dtype (or, how to fill nullable array with NaN)

[cynddl]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

s1 = pd.Series([pd.NA, -1], dtype='Int64')
s2 = pd.Series([pd.NA, -1])  # the dtype is now object

Problem description

s1.fillna(np.nan) and s2.fillna(np.nan) return different series. Specifically, the first one keeps the pd.NA cell untouched and does not replace it with np.nan. Replacing with any other value that np.nan does not trigger this bug.

Expected Output

> s2.fillna(np.nan)
0    NaN
1   -1.0
dtype: float64

Output of pd.show_versions()

INSTALLED VERSIONS

commit : f00ed8f
python : 3.8.5.final.0
python-bits : 64
OS : Linux
OS-release : 4.4.0-210-generic
Version : #242-Ubuntu SMP Fri Apr 16 09:57:56 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.0
numpy : 1.19.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.2.2
setuptools : 50.3.0
Cython : 0.29.17
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.18.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.2
numexpr : 2.7.1
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[mzeitlin11]

Thanks for reporting this @cynddl! While certainly confusing, I believe this is intended behavior. For object type, pd.NA and np.nan are both valid missing values and treated as distinct.

For nullable types like Int64, there is a notion of a single missing value, pd.NA. Generally (though there are probably inconsistencies here), using other missing values than pd.NA with these types will still treat that value as pd.NA. For example using a missing value in construction,

In [3]: pd.Series([np.nan, -1], dtype="Int64")
Out[3]:
0    <NA>
1      -1
dtype: Int64

If you want pandas to treat pd.NA and np.nan differently, your best bet would be object type data (or perhaps nullable float data depending on how #32265 goes)

[simonjayhawkins]

If you want pandas to treat pd.NA and np.nan differently, your best bet would be object type data (or perhaps nullable float data depending on how #32265 goes)

adding no action label for now. needs to considered as part of discussion in #32265. cc @jorisvandenbossche

[simonjayhawkins]

if the outcome of the discussion is that fillna() on a FloatingArray should be changed...

>>> s = pd.Series([pd.NA, -1], dtype='Float64')
>>> s
0    <NA>
1    -1.0
dtype: Float64
>>> 
>>> s.fillna(np.nan)
0    <NA>
1    -1.0
dtype: Float64

then for the IntegerArray, which does not support floats, then the expected behavior would need to be revisted, but for now the behaviour of IntegerArray and FloatingArray are consistent.

[jorisvandenbossche]

As long as we allow NaN in FloatArray (which we currently do, but dependent on #32265), it should be possible to fill your missing values with NaN, so the example above of pd.Series([pd.NA, -1], dtype='Float64').fillna(np.nan) still returning a Series with NA seems wrong. We should change that to have a Series with NaN.

The problem is that this generates some inconsistencies.
In general, when the user uses np.nan as input value, we automatically convert that to NA for nullable dtypes. For example, pd.Series([np.nan, 1], dtype="Float64") converts the NaN into NA, as shown above. But this behaviour for constructors also propagates to other cases. Eg in setitem, such as s[mask] = values, if values is NaN or contains NaNs, those are also treated as NA.
Following that, it makes some sense that also the input argument (the value to fill with) to fillna is treated consistently, and NaN is treated as NA. And that gives the current behaviour, since filling with NA doesn't do anything.

So as long as we automatically convert NaN to NA on input by default, we will probably need a way to specify if you don't want this conversion and you actually want to use NaN (such as the Series[float].fillna(np.nan if you actually want to replace NA with NaN). For methods as fillna, that could maybe be done with a keyword.

We might also want to start raise a UserWarning, to avoid suprises, if NaN is silently treated as NA, especially in cases where it is the input to methods like fillna.

[phofl]

This is basically a duplicate of #39926 and #25288 so closing here