API/BUG: Series(floating, dtype=intlike) ignores dtype, DataFrame casts

[jbrockmendel]

In most contexts, Series is strict about dtype, so will always either return the given dtype or raise. DataFrame is the opposite, often silently ignoring dtype (xref #24435) (i think on the theory that dtype may be intended to apply to some columns but not others).

With floating data and integer dtypes, its the opposite:

arr = np.random.randn(5)

>>> pd.Series(arr, dtype="int16")
0    1.002695
1    0.259332
2   -1.111468
3   -0.680714
4   -0.008943

>>> pd.DataFrame(arr, dtype="int16")
   0
0  1
1  0
2 -1
3  0
4  0

We have exactly one test that is broken if we change the latter behavior, and that is mostly by coincidence. There are other bugs (e.g. Series(bigints, dtype="int8") silently overflowing) that would be easier to fix if maintaining this behavior weren't a consideration.

Is this intentional? cc @jreback @jorisvandenbossche @TomAugspurger

[jorisvandenbossche]

No idea what the reason would be if it was intentional.

So I would be fine with changing it. I am only thinking if there is some way to deprecate it (which doesn't seem easy without introducing an extra keyword just to manager the deprecation ..), since it is a breaking change (actaul numbers would change)

[jorisvandenbossche]

It seems this changed behaviour before, as a long time ago we actually honored the dtype (I checked pd 0.20)

[jbrockmendel]

xref #26919

[jbrockmendel]

Tried changing this to only disallow float-int casting that included NaNs, but it turns out we explicitly test that Series([1, 2, 3.5], dtype=any_int_dtype) raises.

Is there reasoning/precedent for why we would treat list[float] differently from ndarray[float]? cc @jreback

[jreback]

only recollection is that we didn't want to silently cast floats to ints like numpy (but agreed this is a bit odd that we allow for array but not list)

[jbrockmendel]

any preference for changing/deprecating one or the other?

[jreback]

yeah i think we should raise
i find numpy a bit odd here

[jbrockmendel]

yeah i think we should raise

hmm not what i expected. in most cases we think of Series(foo, dtype=bar) as behaving like Series(foo).astype(bar), which wouldn't raise

[jbrockmendel]

update: Index behaves like Series in this case

[jbrockmendel]

This actually gets worse:

In [3]: arr = np.random.randn(5)

In [4]: pd.Series(arr, dtype="i8")
Out[4]: 
0   -1.329695
1    0.721636
2    0.781266
3   -0.413694
4   -0.125569
dtype: float64

In [5]: pd.Series(list(arr), dtype="i8")
ValueError: Trying to coerce float values to integers

The latter behavior we have a specific test for added in #21456

@pandas-dev/pandas-core thoughts on making this consistent?

[rhshadrach]

Instead of raising, when the specified dtype could result in data loss (e.g. float -> int), does a warning make sense? That way it is visible to interactive users and effectively an error for a production job running with -Werror or in a test framework.

[jbrockmendel]

yeah i think we should raise
i find numpy a bit odd here

@jreback for the most part we like to have Series(values, dtype=dtype) === Series(values).astype(dtype). If you're suggesting we raise on the constructor version, would you also want to disallow the astype? Or we could leave the astype as the "do this instead", but sacrifice the equivalency

[jreback]

yeah i think we should raise
i find numpy a bit odd here

@jreback for the most part we like to have Series(values, dtype=dtype) === Series(values).astype(dtype). If you're suggesting we raise on the constructor version, would you also want to disallow the astype? Or we could leave the astype as the "do this instead", but sacrifice the equivalency

I would still have this axiom hold. My point is that (and maybe this is a bigger change / deprecation), casting to ints (not extension) should fail if we have NaN (i think we don this now in all cases) or if these are not exactly the same. This would be 'safe' casting.

IOW the numpy behavior i view as really bad actully (if you want to round/floor/ceil, then that should be explicit).

So I think that both construction & astype should fail here.

[jbrockmendel]

should fail if we have NaN

totally agreed. the case in question here is where we have e.g. np.array([1.5, 2.5]), i.e. no NaNs, and not round

[jreback]

should fail if we have NaN

totally agreed. the case in question here is where we have e.g. np.array([1.5, 2.5]), i.e. no NaNs, and not round

I think we should raise on this as well. This is a big departure from numpy, but IMHO totally unexpected to directly cast these to int should raise. (wether by constror dtype or via .astype)

[jbrockmendel]

OK, ive got a branch almost ready that changes the Series constructor (though not DataFrame, which is a separate branch to ...). I'd be OK with deprecating instead of directly changing, too.

[jreback]

yeah i think have to deprecate

[TomAugspurger]

I don't have a strong opinion here, but one question: What's the performance impact of verifying / validating that we should raise? Does this have to look at each value, or is it equivalent to ndarray.astype(casting="safe")?

[jbrockmendel]

What's the performance impact of verifying / validating that we should raise? Does this have to look at each value, or is it equivalent to ndarray.astype(casting="safe")?

It is not equivalent to casting="safe", no. It goes through maybe_cast_to_integer_array which when passed an ndarray[floaty] boils down to

casted = arr.astype(dtype, copy=copy)
if np.array_equal(arr, casted):
     return casted

if not np.isfinite(arr).all():  # we do this check because our fallback behavior depends on it
    raise IntCastingNaNError(...)
raise ValueError(..)

I've opened numpy/numpy#19146 to discuss how to implement this (well, the uint analogue of this) more efficiently.

[jbrockmendel]

closed by #45142 + #45136 + #41770