Update for dask 2024.11.0

[khaeru]

Dask 2024.11.0 was released on 2024-11-08, triggering CI failures, for instance here.

These include (click to expand):

=========================== short test summary info ============================
FAILED genno/tests/core/test_computer.py::TestComputer::test_deprecated_aggregate - genno.core.exceptions.ComputationError: computing <x:t-y:agg2> using:

<Task <x:t-y:agg2> aggregate(<x:t-y>, <Task ('anom', 0) literal<type=dict>()>, True)>

Use Computer.describe(...) to trace the computation.

Computation traceback:
  File "/home/runner/work/genno/genno/genno/operator.py", line 187, in aggregate
    coords = result.coords[dim].data
             ^^^^^^^^^^^^^
AttributeError: 'Key' object has no attribute 'coords'

FAILED genno/tests/core/test_computer.py::test_eval - genno.core.exceptions.ComputationError: computing <b:t-y> using:

<Task <b:t-y> add(<a:t-y>, <a:t-y>)>

Use Computer.describe(...) to trace the computation.

Computation traceback:
  File "/home/runner/work/genno/genno/genno/operator.py", line 153, in add
    assert_quantity(*quantities)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/runner/work/genno/genno/genno/core/quantity.py", line 22, in assert_quantity
    raise TypeError(
        f"arg #{i+1} ({repr(arg)[:20]}) is not Quantity; likely an incorrect key"
    )
TypeError: arg #1 (<a:t-y>) is not Quantity; likely an incorrect key

FAILED genno/tests/core/test_computer.py::test_order - genno.core.exceptions.ComputationError: computing 'c' using:

<Task 'c' func(<a:y-x>, <b:x-y>)>

Use Computer.describe(...) to trace the computation.

Computation traceback:
  File "/home/runner/work/genno/genno/genno/tests/core/test_computer.py", line 378, in func
    return sum(args)
TypeError: unsupported operand type(s) for +: 'int' and 'Key'

FAILED genno/tests/core/test_computer.py::test_add_product - AssertionError: assert <class 'genno.core.attrseries.AttrSeries'> is <class 'genno.core.key.Key'>
 +  where <class 'genno.core.attrseries.AttrSeries'> = type(t     y   \nfoo1  2000    0.164447\n      2010    0.022211\n      2020    0.389674\n      2030    0.333649\n      2040    0.056085\n      2050    0.830137\nfoo2  2000    0.929491\n      2010    0.215743\n      2020    0.074485\n      2030    0.336839\n      2040    0.202948\n      2050    0.497683\nfoo3  2000    0.059524\n      2010    0.969819\n      2020    0.070786\n      2030    0.320431\n      2040    0.634081\n      2050    0.272614\nbar4  2000    0.409739\n      2010    0.030235\n      2020    0.112415\n      2030    0.026342\n      2040    0.186713\n      2050    0.267134\nbar5  2000    0.105754\n      2010    0.095589\n      2020    0.244615\n      2030    0.151573\n      2040    0.083683\n      2050    0.336899\nbar6  2000    0.188441\n      2010    0.513554\n      2020    0.479825\n      2030    0.698588\n      2040    0.196385\n      2050    0.397112\nName: Quantity X, dtype: float64, units: kilogram ** 2)
 +  and   <class 'genno.core.key.Key'> = type(<x:t-y-t-y>)

FAILED genno/tests/core/test_computer.py::test_dantzig - genno.core.exceptions.ComputationError: computing <d:i> using:

<Task <d:i> sum(<d:i-j>)>

Use Computer.describe(...) to trace the computation.

Computation traceback:
  File "/home/runner/work/genno/genno/genno/core/operator.py", line 50, in __call__
    return self.func(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/runner/work/genno/genno/genno/operator.py", line 1001, in sum
    return quantity._keep((quantity * _w).sum(dim=dimensions) / w_total, name=True)
           ^^^^^^^^^^^^^^
AttributeError: 'Key' object has no attribute '_keep'

FAILED genno/tests/compat/test_pyam.py::test_computer_as_pyam - genno.core.exceptions.ComputationError: computing 'ACT IAMC' using:

<Task 'ACT IAMC' as_pyam(Alias(scenario->scenario), <ACT:nl-t-yv-ya-m-h>)>

Use Computer.describe(...) to trace the computation.

Computation traceback:
  File "/home/runner/work/genno/genno/genno/core/operator.py", line 50, in __call__
    return self.func(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/runner/work/genno/genno/genno/compat/pyam/operator.py", line 118, in as_pyam
    assign = dict(unit=quantity.units)
                       ^^^^^^^^^^^^^^
AttributeError: 'Key' object has no attribute 'units'

FAILED genno/tests/compat/test_pyam.py::test_concat - genno.core.exceptions.ComputationError: computing <tom:nl-t-yv-ya> using:

<Task <tom:nl-t-yv-ya> add(<fom:nl-t-yv-ya>, <vom:nl-t-yv-ya>)>

Use Computer.describe(...) to trace the computation.

Computation traceback:
  File "/home/runner/work/genno/genno/genno/operator.py", line 153, in add
    assert_quantity(*quantities)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/runner/work/genno/genno/genno/core/quantity.py", line 22, in assert_quantity
    raise TypeError(
        f"arg #{i+1} ({repr(arg)[:20]}) is not Quantity; likely an incorrect key"
    )
TypeError: arg #1 (<fom:nl-t-yv-ya>) is not Quantity; likely an incorrect key

FAILED genno/tests/core/test_computer.py::TestComputer::test_add_aggregate - genno.core.exceptions.ComputationError: computing <x:t-y:agg2> using:

<Task <x:t-y:agg2> aggregate(<x:t-y>)>

Use Computer.describe(...) to trace the computation.

Computation traceback:
  File "/home/runner/work/genno/genno/genno/operator.py", line 187, in aggregate
    coords = result.coords[dim].data
             ^^^^^^^^^^^^^
AttributeError: 'Key' object has no attribute 'coords'

FAILED genno/tests/core/test_quantity.py::TestQuantity::test_size[attrseries] - AttributeError: 'Key' object has no attribute 'to_series'

FAILED genno/tests/test_operator.py::test_select_bigmem[attrseries] - genno.core.exceptions.ComputationError: computing 'random indexers' using:

<Task 'random indexers' random_indexers(<q_00:a-b-c-d-e-f-g-h-i-j-k-l-m-n-o-p-q>)>

Use Computer.describe(...) to trace the computation.

Computation traceback:
  File "/home/runner/work/genno/genno/genno/tests/test_operator.py", line 894, in random_indexers
    coords = qty.coords
             ^^^^^^^^^^
AttributeError: 'Key' object has no attribute 'coords'

FAILED genno/tests/core/test_exceptions.py::test_computationerror_ipython - AssertionError: computing 'test' using:
  
  <Task 'test' fail()>
  
  Use Computer.describe(...) to trace the computation.
  
  Computation traceback:
    File "/tmp/ipykernel_2972/1121372927.py", line 4, in fail
      'x' + 3.4  # Raises TypeError
      ~~~~^~~~~
  TypeError: can only concatenate str (not "float") to str
  
assert None
 +  where None = <built-in method match of re.Pattern object at 0x557722a17dd0>('computing \'test\' using:\n\n<Task \'test\' fail()>\n\nUse Computer.describe(...) to trace the computation.\n\nComputation traceback:\n  File "/tmp/ipykernel_2972/1121372927.py", line 4, in fail\n    \'x\' + 3.4  # Raises TypeError\n    ~~~~^~~~~\nTypeError: can only concatenate str (not "float") to str\n')
 +    where <built-in method match of re.Pattern object at 0x557722a17dd0> = re.compile('computing \'test\' using:\n\n\\(<function fail at \\w+>,\\)\n\nUse Computer.describe\\(...\\) to trace the computation\\.\n\nComputation traceback:\n  File ".*", line 4, in fail\n    \'x\' \\+ 3.4  #).match

To resolve:

• Investigate the root cause and failure mechanism.
  • This is likely related to “Introduce new Task class” / Add a Task class to replace tuples for task specification dask/dask#11248 first mentioned in dask 2024.9.0. Although that PR description says “…with full backwards compatibility”, further internal changes in 2024.11.0 probably employed this new class in a backwards-incompatible manner.
  • Stricter data type for dask keys dask/dask#10485 (released with dask 2023.9.1) narrowed the allowable types for keys to Union[str, bytes, int, float, tuple["Key", ...]]. genno.Key is not a subtype of any of these, but seemed to work until the above changes.
• File any bug(s) upstream. Upstream didn't communicate the changes loudly, but the behaviour is as advertised/specified.
• Adapt code in genno that interacts with dask.

To mitigate/work around:

• Use genno with dask 2024.10.0 or earlier.