clip fails on NA/nan values

[burnpanck]

The following code fails in the last line with TypeError: boolean value of NA is ambiguous:

import numpy as np
import pandas as pd
import pint
import pint_pandas

print(f"{pd.__version__=}")
print(f"{pint.__version__=}")
print(f"{pint_pandas.__version__=}")

u = pint.get_application_registry()

a = np.r_[1,2,np.nan,4,10]
print(f"{a=}")
print(f"{np.clip(a,3,5)=}")

s = pd.Series(data=a)
print(f"{s=}")
print(f"{np.clip(s,3,5)=}")

qs = pd.Series(data=pint_pandas.PintArray.from_1darray_quantity(a*u.m))
print(f"{qs=}")
print(f"{np.clip(qs,3*u.m,5*u.m)=}")

It's output before failing in my environment is:

pd.__version__='2.2.3'
pint.__version__='0.24.4'
pint_pandas.__version__='0.6.2'
a=array([ 1.,  2., nan,  4., 10.])
np.clip(a,3,5)=array([ 3.,  3., nan,  4.,  5.])
s=0     1.0
1     2.0
2     NaN
3     4.0
4    10.0
dtype: float64
np.clip(s,3,5)=0    3.0
1    3.0
2    NaN
3    4.0
4    5.0
dtype: float64
qs=0     1.0
1     2.0
2     nan
3     4.0
4    10.0
dtype: pint[meter]

So, clearly, numpy chose to pass-through nan values in np.clip, and so does pandas.

[andrewgsavage]

does this PR #247 help at all?

…

On Mon, Nov 11, 2024 at 11:06 AM Yves Delley ***@***.***> wrote: The following code fails in the last line with TypeError: boolean value of NA is ambiguous: import numpy as npimport pandas as pdimport pintimport pint_pandas print(f"{pd.__version__=}")print(f"{pint.__version__=}")print(f"{pint_pandas.__version__=}") u = pint.get_application_registry() a = np.r_[1,2,np.nan,4,10]print(f"{a=}")print(f"{np.clip(a,3,5)=}") s = pd.Series(data=a)print(f"{s=}")print(f"{np.clip(s,3,5)=}") qs = pd.Series(data=pint_pandas.PintArray.from_1darray_quantity(a*u.m))print(f"{qs=}")print(f"{np.clip(qs,3*u.m,5*u.m)=}") np.clip(s,3*u.m, 5*u.m) It's output before failing in my environment is: pd.__version__='2.2.3' pint.__version__='0.24.4' pint_pandas.__version__='0.6.2' a=array([ 1., 2., nan, 4., 10.]) np.clip(a,3,5)=array([ 3., 3., nan, 4., 5.]) s=0 1.0 1 2.0 2 NaN 3 4.0 4 10.0 dtype: float64 np.clip(s,3,5)=0 3.0 1 3.0 2 NaN 3 4.0 4 5.0 dtype: float64 qs=0 1.0 1 2.0 2 nan 3 4.0 4 10.0 dtype: pint[meter] So, clearly, numpy chose to pass-through nan values in np.clip, and so does pandas. — Reply to this email directly, view it on GitHub <#255>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADEMLEBMHRVS65DTN5ABZX32ACFTNAVCNFSM6AAAAABRRSZFUGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGY2DSMBQGYYDMMI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[burnpanck]

I wasn't aware of "subdtypes" or that PR yet. However, I don't immediately see how it would help. In principle, I'm a big fan of proper nullable types ("not-a-number" is not the same as "not-available" to me). However, in the case of clip, I would argue NA most definitely should pass through rather than raising an exception, as NaN always did outside of pint_pandas. So while "subdtypes" should give me control over whether I'd like to use NaN or NA, I feel we're essentially seeing a bug here, for which no API change should be needed.

[andrewgsavage]

I've added __array_function__, but it doesnt work with Series so you'll need to use np.clip(qs.values, ...)