__iter__ makes Quantity look list_like when it's not

[MichaelTiemannOSC]

The code defining a PlainQuantity includes the following:

def __iter__(self: PlainQuantity[Iterable[S]]) -> Iterator[S]:
        # Make sure that, if self.magnitude is not iterable, we raise TypeError as soon
        # as one calls iter(self) without waiting for the first element to be drawn from
        # the iterator
        it_magnitude = iter(self.magnitude)

When self.magnitude is not iterable, TypeError will be raised as per the comment, but that may be too later for the loosely defined is_list_like, which will report True if it so much finds an iter attribute (even an iter attribute that wants to raise a TypeError the instant anybody tries to use it).

What if we hooked getattr so that if it's called with iter it will return False if self.magnitude is not iterable? Would that not both simplify the implementation as well as protecting is_list_like from getting the wrong idea how this particular Quantity identifies?

[hgrecco]

I don't think that is the right way in Python.

According to the docs link

The only reliable way to determine whether an object is iterable is to call iter(obj).

Which is what Pint is doing.

[MichaelTiemannOSC]

So maybe this code from Pandas needs to be fixed?

cdef inline bint c_is_list_like(object obj, bint allow_sets) except -1:
    # first, performance short-cuts for the most common cases
    if util.is_array(obj):
        # exclude zero-dimensional numpy arrays, effectively scalars
        return not cnp.PyArray_IsZeroDim(obj)
    elif isinstance(obj, list):
        return True
    # then the generic implementation
    return (
        # equiv: `isinstance(obj, abc.Iterable)`
        getattr(obj, "__iter__", None) is not None and not isinstance(obj, type)
        # we do not count strings/unicode/bytes as list-like
        and not isinstance(obj, (str, bytes))
        # exclude zero-dimensional duck-arrays, effectively scalars
        and not (hasattr(obj, "ndim") and obj.ndim == 0)
        # exclude sets if allow_sets is False
        and not (allow_sets is False and isinstance(obj, abc.Set))
    )

Here's a reproducer of the problem, treating Pandas as just a nice way to organize random data (such as Quantities) into DataFrames that can otherwise be a nice way to slice and dice:

import pandas as pd
from pint import Quantity as Q_
qq = pd.DataFrame({'qq': [Q_(1.0,'m'), Q_(2.0, 'm')]})
# /Users/michael/opt/miniconda3/envs/pint-vanilla/lib/python3.9/site-packages/pandas/core/dtypes/cast.py:1784: UnitStrippedWarning: The unit of the quantity is stripped when downcasting to ndarray.
  result[:] = values
qq.iloc[0, 0] = Q_(3.0,'m')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/michael/opt/miniconda3/envs/pint-vanilla/lib/python3.9/site-packages/pandas/core/indexing.py", line 818, in __setitem__
    iloc._setitem_with_indexer(indexer, value, self.name)
  File "/Users/michael/opt/miniconda3/envs/pint-vanilla/lib/python3.9/site-packages/pandas/core/indexing.py", line 1797, in _setitem_with_indexer
    self._setitem_single_block(indexer, value, name)
  File "/Users/michael/opt/miniconda3/envs/pint-vanilla/lib/python3.9/site-packages/pandas/core/indexing.py", line 2078, in _setitem_single_block
    self.obj._mgr = self.obj._mgr.setitem(indexer=indexer, value=value)
  File "/Users/michael/opt/miniconda3/envs/pint-vanilla/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 388, in setitem
    return self.apply("setitem", indexer=indexer, value=value)
  File "/Users/michael/opt/miniconda3/envs/pint-vanilla/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 347, in apply
    applied = getattr(b, f)(**kwargs)
  File "/Users/michael/opt/miniconda3/envs/pint-vanilla/lib/python3.9/site-packages/pandas/core/internals/blocks.py", line 983, in setitem
    casted = setitem_datetimelike_compat(values, len(vi), casted)
  File "/Users/michael/opt/miniconda3/envs/pint-vanilla/lib/python3.9/site-packages/pint/quantity.py", line 1868, in __len__
    return len(self._magnitude)
TypeError: object of type 'float' has no len()

Here's the Pint-Pandas way of doing it:

import pandas as pd
from pint import Quantity as Q_
from pint_pandas import PintArray as PA_
>>> xx = pd.DataFrame({'qq': PA_([1.0, 2.0], 'pint[m]')})
>>> xx.iloc[0, 0] = Q_(3.0,'m')
>>> xx
# /Users/michael/opt/miniconda3/envs/pint-vanilla/lib/python3.9/site-packages/pint_pandas/pint_array.py:648: UnitStrippedWarning: The unit of the quantity is stripped when downcasting to ndarray.
  return np.array(qtys, dtype="object", copy=copy)
    qq
0  3.0
1  2.0

Maybe the answer is YOU CANNOT USE PINT IN PANDAS UNLESS YOU USE PINT-PANDAS CORRECTLY. But it would be so nice to use the object nature of Quantities for Ad Hoc data that could be sliced and diced into PintArrays when we wanted to do math. Again, the Pandas people might say "all Pandas operations are math, including split/transform/combine". Seems like a harsh set of rules to adhere to, especially if they won't test iter-ability according to the rules.

[hgrecco]

I understand and have been bitten by the same issue. I am sure that there is a good reason that Panda's is_list_like but I think that in general it is safer to adhere to Python's guidelines and that is what I will be pushing in Pint.

Having said that, it is worth noting that it has been discussed in the past (after pint-pandas 0.1 was released) if we should create two classes: QuantityScalar and QuantityArray. (See #1128)

[MichaelTiemannOSC]

I'm looking at how Pandas got to where it got to, and I think they are arguably in the wrong place for two reasons.

The commit (lines 945-947 replaced with 934-941): pandas-dev/pandas@63ae9aa#diff-55001624a0932c1b6cee2e6ddb65dea85c1faf0dee84812c0ca0c32916a71438

Reason 1 is what we are talking about. Reason 2 is that is_list_like can actually be overridden as defined in pandas/core/dtypes/interence.py:

is_list_like = lib.is_list_like

But the code they committed specifically overrides the override! So I cannot test my cool theory without opening that can of worms.

[andrewgsavage]

pandas will recognise Quantity scalars as scalars if they have ndim = 0. #1599 adds the ndim attribute to Quantity, returning 0 for scalars. This is enough to get is_list_like working.