Dependence of dask_awkward.lib.testutils on pyarrow is not declared

[jpivarski]

Fresh installation:

mamba create -n quickie python==3.11
mamba activate quickie
mamba list

# packages in environment at /home/jpivarski/mambaforge/envs/quickie:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_0    conda-forge
libgomp                   13.2.0               h807b86a_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsqlite                 3.43.0               h2797004_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
openssl                   3.1.2                hd590300_0    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
python                    3.11.0          he550d4f_1_cpython    conda-forge
readline                  8.2                  h8228510_1    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge

And then

pip install dask-awkward

Collecting dask-awkward
  Obtaining dependency information for dask-awkward from https://files.pythonhosted.org/packages/66/d0/87a32662f2e1834ad3999ba7b85a301aa142869ad6dc7a675df923bff35a/dask_awkward-2023.9.1-py3-none-any.whl.metadata
  Downloading dask_awkward-2023.9.1-py3-none-any.whl.metadata (3.3 kB)
Collecting awkward>=2.4.0 (from dask-awkward)
  Obtaining dependency information for awkward>=2.4.0 from https://files.pythonhosted.org/packages/c5/be/531c95dd71da5e6814c36672e8ec58e26a084418908717162556bc73e5eb/awkward-2.4.2-py3-none-any.whl.metadata
  Downloading awkward-2.4.2-py3-none-any.whl.metadata (6.9 kB)
Collecting dask>=2023.04.0 (from dask-awkward)
  Obtaining dependency information for dask>=2023.04.0 from https://files.pythonhosted.org/packages/3d/2f/f08c8fdc77fafcd15c14dd80b891600184b35ad60784461ddc9fea44a549/dask-2023.9.1-py3-none-any.whl.metadata
  Downloading dask-2023.9.1-py3-none-any.whl.metadata (3.6 kB)
Collecting awkward-cpp==23 (from awkward>=2.4.0->dask-awkward)
  Obtaining dependency information for awkward-cpp==23 from https://files.pythonhosted.org/packages/cd/d6/44becf38423091f47467ae12bb702c2eebd5719f7e5310921dd55e7f2a4a/awkward_cpp-23-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading awkward_cpp-23-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.2 kB)
Collecting importlib-metadata>=4.13.0 (from awkward>=2.4.0->dask-awkward)
  Obtaining dependency information for importlib-metadata>=4.13.0 from https://files.pythonhosted.org/packages/cc/37/db7ba97e676af155f5fcb1a35466f446eadc9104e25b83366e8088c9c926/importlib_metadata-6.8.0-py3-none-any.whl.metadata
  Using cached importlib_metadata-6.8.0-py3-none-any.whl.metadata (5.1 kB)
Collecting numpy>=1.18.0 (from awkward>=2.4.0->dask-awkward)
  Obtaining dependency information for numpy>=1.18.0 from https://files.pythonhosted.org/packages/32/6a/65dbc57a89078af9ff8bfcd4c0761a50172d90192eaeb1b6f56e5fbf1c3d/numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)
Collecting packaging (from awkward>=2.4.0->dask-awkward)
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting click>=8.0 (from dask>=2023.04.0->dask-awkward)
  Obtaining dependency information for click>=8.0 from https://files.pythonhosted.org/packages/00/2e/d53fa4befbf2cfa713304affc7ca780ce4fc1fd8710527771b58311a3229/click-8.1.7-py3-none-any.whl.metadata
  Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting cloudpickle>=1.5.0 (from dask>=2023.04.0->dask-awkward)
  Downloading cloudpickle-2.2.1-py3-none-any.whl (25 kB)
Collecting fsspec>=2021.09.0 (from dask>=2023.04.0->dask-awkward)
  Obtaining dependency information for fsspec>=2021.09.0 from https://files.pythonhosted.org/packages/3a/9f/b40e8e5be886143379000af5fc0c675352d59e82fd869d24bf784161dc77/fsspec-2023.9.0-py3-none-any.whl.metadata
  Downloading fsspec-2023.9.0-py3-none-any.whl.metadata (6.7 kB)
Collecting partd>=1.2.0 (from dask>=2023.04.0->dask-awkward)
  Downloading partd-1.4.0-py3-none-any.whl (18 kB)
Collecting pyyaml>=5.3.1 (from dask>=2023.04.0->dask-awkward)
  Obtaining dependency information for pyyaml>=5.3.1 from https://files.pythonhosted.org/packages/7b/5e/efd033ab7199a0b2044dab3b9f7a4f6670e6a52c089de572e928d2873b06/PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Collecting toolz>=0.10.0 (from dask>=2023.04.0->dask-awkward)
  Downloading toolz-0.12.0-py3-none-any.whl (55 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.8/55.8 kB 6.6 MB/s eta 0:00:00
Collecting zipp>=0.5 (from importlib-metadata>=4.13.0->awkward>=2.4.0->dask-awkward)
  Obtaining dependency information for zipp>=0.5 from https://files.pythonhosted.org/packages/8c/08/d3006317aefe25ea79d3b76c9650afabaf6d63d1c8443b236e7405447503/zipp-3.16.2-py3-none-any.whl.metadata
  Using cached zipp-3.16.2-py3-none-any.whl.metadata (3.7 kB)
Collecting locket (from partd>=1.2.0->dask>=2023.04.0->dask-awkward)
  Downloading locket-1.0.0-py2.py3-none-any.whl (4.4 kB)
Downloading dask_awkward-2023.9.1-py3-none-any.whl (68 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.5/68.5 kB 7.9 MB/s eta 0:00:00
Downloading awkward-2.4.2-py3-none-any.whl (702 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 703.0/703.0 kB 47.6 MB/s eta 0:00:00
Downloading awkward_cpp-23-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 65.0 MB/s eta 0:00:00
Downloading dask-2023.9.1-py3-none-any.whl (1.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 69.3 MB/s eta 0:00:00
Downloading click-8.1.7-py3-none-any.whl (97 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.9/97.9 kB 12.3 MB/s eta 0:00:00
Downloading fsspec-2023.9.0-py3-none-any.whl (173 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 173.2/173.2 kB 21.9 MB/s eta 0:00:00
Using cached importlib_metadata-6.8.0-py3-none-any.whl (22 kB)
Downloading numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 62.8 MB/s eta 0:00:00
Downloading PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (757 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 757.7/757.7 kB 54.9 MB/s eta 0:00:00
Using cached zipp-3.16.2-py3-none-any.whl (7.2 kB)
Installing collected packages: zipp, toolz, pyyaml, packaging, numpy, locket, fsspec, cloudpickle, click, partd, importlib-metadata, awkward-cpp, dask, awkward, dask-awkward
Successfully installed awkward-2.4.2 awkward-cpp-23 click-8.1.7 cloudpickle-2.2.1 dask-2023.9.1 dask-awkward-2023.9.1 fsspec-2023.9.0 importlib-metadata-6.8.0 locket-1.0.0 numpy-1.25.2 packaging-23.1 partd-1.4.0 pyyaml-6.0.1 toolz-0.12.0 zipp-3.16.2

mamba list

# packages in environment at /home/jpivarski/mambaforge/envs/quickie:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
awkward                   2.4.2                    pypi_0    pypi
awkward-cpp               23                       pypi_0    pypi
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
click                     8.1.7                    pypi_0    pypi
cloudpickle               2.2.1                    pypi_0    pypi
dask                      2023.9.1                 pypi_0    pypi
dask-awkward              2023.9.1                 pypi_0    pypi
fsspec                    2023.9.0                 pypi_0    pypi
importlib-metadata        6.8.0                    pypi_0    pypi
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_0    conda-forge
libgomp                   13.2.0               h807b86a_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsqlite                 3.43.0               h2797004_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
locket                    1.0.0                    pypi_0    pypi
ncurses                   6.4                  hcb278e6_0    conda-forge
numpy                     1.25.2                   pypi_0    pypi
openssl                   3.1.2                hd590300_0    conda-forge
packaging                 23.1                     pypi_0    pypi
partd                     1.4.0                    pypi_0    pypi
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
python                    3.11.0          he550d4f_1_cpython    conda-forge
pyyaml                    6.0.1                    pypi_0    pypi
readline                  8.2                  h8228510_1    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toolz                     0.12.0                   pypi_0    pypi
tzdata                    2023c                h71feb2d_0    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zipp                      3.16.2                   pypi_0    pypi

When we try to import:

>>> import dask_awkward

is fine but

>>> import dask_awkward.lib.testutils
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jpivarski/mambaforge/envs/quickie/lib/python3.11/site-packages/dask_awkward/lib/testutils.py", line 8, in <module>
    import pyarrow as pa
ModuleNotFoundError: No module named 'pyarrow'

If this is an optional dependency, it should say which library needs it (dask-awkward) in a sense like https://github.com/scikit-hep/uproot5/blob/main/src/uproot/extras.py

I encountered this in an Uproot CI test that hadn't been failing before and didn't know that it had anything to do with dask-awkward until I dug into it. An error message like

ModuleNotFoundError: dask-awkward.lib.testutils.FUNCTION_NAME requires pyarrow, which can be installed with...

would tell users and downstream dependents why they're getting this error (maybe they don't really need FUNCTION_NAME) and an immediate course of action.

[douglasdavis]

Ah this leaked through to xfail a few tests based on awkward and pyarrow versions, but we now require an awkward version where the issue was fixed. We can just remove the pyarrow import completely