p4p incompatible with NumPy 2.0

[DiamondJoseph]

On attempting to load p4p with the latest release of NumPy raises errors due to being compiled against the previous version.

File "src/p4p/_p4p.pyx", line 1, in init p4p._p4p
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

[mdavidsaver]

Oh goody... Any idea how to accommodate both numpy 1.x and 2.x series without (somehow) doubling the .whl files?

fyi. the error comes from cython.

[callumforrester]

Unsure... @coretl might have some ideas...

[AlexanderWells-diamond]

I've done some digging and it looks like Numpy is pulling some magic under the covers to make 2.0 builds compatible with 1.* builds. They have a guide here, which seems to simply say that you build against 2.* and it'll work for 1.*. There's a caveat about what Python versions it may work with though; Numpy 2.0 is Python3.9+, so care will have to be taken to continue supporting older Python versions.

[tacaswell]

For reference, look at how scipy or Matplotlib are doing this.

For unsupported versions of Python (<=3.8) you will need to fallback to oldest-supported-numpy to build the wheels with a version of numpy that will work with any numpy that has a wheels for that Python version.

[finnoshea]

p4p now seems to require numpy 2.0. Is this intended? It appears to make p4p impossible to install.
From my latest attempts to install p4p:

67.85 The conflict is caused by:
67.85 The user requested numpy<1.23
67.85 p4p 3.5.0 depends on numpy>=2.0.1

This shows up in setup.py:

install_requires = [
        epicscorelibs.version.abi_requires(),
        pvxslibs.version.abi_requires(),
        # assume ABI forward compatibility as indicated by
        # https://github.com/numpy/numpy/blob/master/numpy/core/setup_common.py#L28
        'numpy >=%s'%numpy.version.short_version,
        'nose2>=0.8.0',
        'ply', # for asLib
    ]

I wonder if this is inheriting a numpy 2.0 dependence somehow?

Yep, the problem is here. The problem code appears to have been committed by @OCopping 10 hours ago.

[mdavidsaver]

@finnoshea Which version of python are you using? Which platform? Do you expect/want to use a pre-built wheel?

I think you are correct that the 'numpy >=%s'%numpy.version.short_version, dependency is now too strict post numpy 2.x.

The original logic was to build wheels against a fairly old numpy, which was in practice ABI compatible with newer versions at runtime.

For numpy 2.x, I'm not sure what the oldest backwards compatible version is wrt. ABI.

install_requires = [ ... ] # existing deps excluding numpy
if numpy.lib.NumpyVersion(numpy.__version__) >= '2.0.0b1':
    install_requires += ['numpy >= 1.7', 'numpy < 3']
else:
    install_requires += ['numpy >=%s'%numpy.version.short_version, 'numpy < 2']

I pick 1.7 somewhat arbitrarily. I think that whichever version is picked will probably need to inject eg. :

#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION

[tacaswell]

Please read the manual (also already linked above): https://numpy.org/devdocs/dev/depending_on_numpy.html#adding-a-dependency-on-numpy

[mdavidsaver]

Please read the manual ...

I did last month, and I have again now. I admit that I was confusing the existing NPY_NO_DEPRECATED_API with the new NPY_TARGET_VERSION macro. So the answer to my question of what the oldest possible backwards compatible version for 2.x builds is in fact 1.7.

https://github.com/numpy/numpy/blob/97ac25534b359f35e690df49696405462abd9909/numpy/_core/include/numpy/numpyconfig.h#L64

This document does not address also supporting source builds against 1.x, which I think P4P needs to support. So I think the condition I wrote above will be necessary, with the addition of defining the per-processor macro NPY_TARGET_VERSION to NPY_1_7_API_VERSION.

[finnoshea]

Apologies for the verbose dump in this comment. I've been trying to install p4p in a Docker container.

Here is the Dockerfile I inherited:

ARG PYTHON_VERSION=3.10-slim-bullseye

FROM python:${PYTHON_VERSION}

SHELL ["/bin/bash", "-c"]
ARG DEBIAN_FRONTEND=noninteractive
ENV PYTHONFAULTHANDLER=1 \
    PYTHONUNBUFFERED=1 \
    PYTHONHASHSEED=random \
    PIP_NO_CACHE_DIR=off \
    PIP_DISABLE_PIP_VERSION_CHECK=on \
    PIP_DEFAULT_TIMEOUT=100

# Install build dependencies
RUN apt-get update \
    && apt-get install -y --no-install-recommends build-essential curl git vim poppler-utils pkg-config libhdf5-serial-dev \
    && apt-get autoremove -y \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

# Install poetry as package manager
#ENV POETRY_HOME="/opt/poetry" \
#    POETRY_VIRTUALENVS_IN_PROJECT=1 \
#    POETRY_NO_INTERACTION=1
#ARG POETRY_VERSION=1.4.1
#RUN curl -sSL https://install.python-poetry.org | POETRY_HOME=${POETRY_HOME} POETRY_VERSION=${POETRY_VERSION} python3 -

# Setup venv
RUN python3 -m venv /opt/venv
RUN pip3 install --upgrade pip setuptools wheel

# Install packages
#ENV PATH="/opt/venv/bin:$POETRY_HOME/bin:$PATH"

WORKDIR /opt/SLAC-AD
RUN source /opt/venv/bin/activate
#COPY pyproject.toml poetry.lock ./
RUN source /opt/venv/bin/activate
#RUN poetry config virtualenvs.create false \
#    && poetry install --no-interaction --no-ansi --no-root

COPY requirements.txt ./
RUN pip3 install -r requirements.txt
COPY p4p_reqs.txt ./
RUN pip3 install -r p4p_reqs.txt

#ADD p4p_install /opt/p4p
#RUN pip3 install epicscorelibs pvxslibs /opt/p4p  

# Set env var needed to load EPICS data (as expected by the meme/p4p packages)
# Valid only on machines where mccas0 is a known alias
ENV EPICS_PVA_ADDR_LIST=mccas0

# Set env var needed to load BSA data
# Requires the BSA data directory to be mounted to this path
ENV BSA_DATA_DIR=/opt/data/bsa

# Set env var to project home
# Requires the project home directory to be mounted to this path
ENV PROJECT_HOME=/opt/project

WORKDIR /opt/SLAC-AD

I've had to excise the poetry stuff from there, because it does not play well with pytorch. The two files of interest to you are requirements.txt:

colorama #==0.4.6 ; python_version >= "3.10" and python_version < "3.11" and platform_system == "Windows"
#epicscorelibs==7.0.7.99.1.1a2  #==7.0.7.99.0.2 ; python_version >= "3.10" and python_version < "3.11"
h5py  #==3.11.0 ; python_version >= "3.10" and python_version < "3.11"
joblib  #==1.4.2 ; python_version >= "3.10" and python_version < "3.11"
kaleido  #==0.2.1 ; python_version >= "3.10" and python_version < "3.11"
llvmlite  #==0.39.1 ; python_version >= "3.10" and python_version < "3.11"
#meme @ git+https://github.com/slaclab/meme@af07afe431fa4503664aca8d9f83f786f85abbf0 ; python_version >= "3.10" and python_version < "3.11"
#nose2  #==0.15.1 ; python_version >= "3.10" and python_version < "3.11"
numba  #==0.56.4 ; python_version >= "3.10" and python_version < "3.11"
numpy  #==1.23.5 ; python_version >= "3.10" and python_version < "3.11"
#p4p  #4.1.12 ; python_version >= "3.10" and python_version < "3.11"
packaging  #==24.1 ; python_version >= "3.10" and python_version < "3.11"
pandas  #==1.5.3 ; python_version >= "3.10" and python_version < "3.11"
plotly  #==5.23.0 ; python_version >= "3.10" and python_version < "3.11"
ply  #==3.11 ; python_version >= "3.10" and python_version < "3.11"
#pvxslibs  #==1.3.1 ; python_version >= "3.10" and python_version < "3.11"
pynndescent  #==0.5.13 ; python_version >= "3.10" and python_version < "3.11"
python-dateutil  #==2.9.0.post0 ; python_version >= "3.10" and python_version < "3.11"
pytz  #==2022.7.1 ; python_version >= "3.10" and python_version < "3.11"
scikit-learn  #==1.5.1 ; python_version >= "3.10" and python_version < "3.11"
scipy  #==1.14.0 ; python_version >= "3.10" and python_version < "3.11"
setuptools-dso  #==2.10 ; python_version >= "3.10" and python_version < "3.11"
setuptools  #==72.2.0 ; python_version >= "3.10" and python_version < "3.11"
six  #==1.16.0 ; python_version >= "3.10" and python_version < "3.11"
tenacity  #==9.0.0 ; python_version >= "3.10" and python_version < "3.11"
threadpoolctl  #==3.5.0 ; python_version >= "3.10" and python_version < "3.11"
tqdm  #==4.66.5 ; python_version >= "3.10" and python_version < "3.11"
umap-learn  #==0.5.6 ; python_version >= "3.10" and python_version < "3.11"
xlrd  #==2.0.1 ; python_version >= "3.10" and python_version < "3.11"

and p4p_reqs.txt:

epicscorelibs
meme @ git+https://github.com/slaclab/meme@af07afe431fa4503664aca8d9f83f786f85abbf0 ; python_version >= "3.10" and python_version < "3.11"
nose2  #==0.15.1 ; python_version >= "3.10" and python_version < "3.11"
p4p  #4.1.12 ; python_version >= "3.10" and python_version < "3.11"

Below is the error I get when trying to pip3 install p4p_reqs.txt. I get the same error with 3.11-slim-bullseye as the base. If I limit numpy to numpy<1.23, it appears to download 2.0.1 anyway and then I get the same error.

0.207 Collecting meme@ git+https://github.com/slaclab/meme@af07afe431fa4503664aca8d9f83f786f85abbf0 (from -r p4p_reqs.txt (line 2))                                                                                                                  
0.208   Cloning https://github.com/slaclab/meme (to revision af07afe431fa4503664aca8d9f83f786f85abbf0) to /tmp/pip-install-12gmrgwt/meme_66b649dcaeb54ab2ab297066b8f9da80                                                                            
0.208   Running command git clone --filter=blob:none --quiet https://github.com/slaclab/meme /tmp/pip-install-12gmrgwt/meme_66b649dcaeb54ab2ab297066b8f9da80                                                                                         
1.209   Running command git rev-parse -q --verify 'sha^af07afe431fa4503664aca8d9f83f786f85abbf0'                                                                                                                                                     
1.210   Running command git fetch -q https://github.com/slaclab/meme af07afe431fa4503664aca8d9f83f786f85abbf0
1.483   Running command git checkout -q af07afe431fa4503664aca8d9f83f786f85abbf0
1.831   Resolved https://github.com/slaclab/meme to commit af07afe431fa4503664aca8d9f83f786f85abbf0
1.835   Preparing metadata (setup.py): started
1.944   Preparing metadata (setup.py): finished with status 'done'
2.118 Collecting epicscorelibs (from -r p4p_reqs.txt (line 1))
2.227   Downloading epicscorelibs-7.0.7.99.0.2.tar.gz (1.6 MB)
2.289      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 116.5 MB/s eta 0:00:00
2.432   Installing build dependencies: started
3.266   Installing build dependencies: finished with status 'done'
3.267   Getting requirements to build wheel: started
3.405   Getting requirements to build wheel: finished with status 'done'
3.406   Preparing metadata (pyproject.toml): started
3.529   Preparing metadata (pyproject.toml): finished with status 'done'
3.542 Collecting nose2 (from -r p4p_reqs.txt (line 3))
3.548   Downloading nose2-0.15.1-py3-none-any.whl.metadata (7.3 kB)
3.613 Collecting p4p (from -r p4p_reqs.txt (line 4))
3.624   Downloading p4p-4.1.12.tar.gz (110 kB)
3.649   Installing build dependencies: started
60.98   Installing build dependencies: finished with status 'done'
60.98   Getting requirements to build wheel: started
62.21   Getting requirements to build wheel: finished with status 'done'
62.21   Preparing metadata (pyproject.toml): started
62.54   Preparing metadata (pyproject.toml): finished with status 'done'
62.54 Requirement already satisfied: setuptools in /usr/local/lib/python3.10/site-packages (from epicscorelibs->-r p4p_reqs.txt (line 1)) (72.2.0)
62.54 Requirement already satisfied: setuptools-dso>=2.9a1 in /usr/local/lib/python3.10/site-packages (from epicscorelibs->-r p4p_reqs.txt (line 1)) (2.10)
62.54 Requirement already satisfied: numpy in /usr/local/lib/python3.10/site-packages (from epicscorelibs->-r p4p_reqs.txt (line 1)) (1.22.4)
62.54 Requirement already satisfied: pandas in /usr/local/lib/python3.10/site-packages (from meme@ git+https://github.com/slaclab/meme@af07afe431fa4503664aca8d9f83f786f85abbf0->-r p4p_reqs.txt (line 2)) (2.2.2)
62.55 Collecting epicscorelibs (from -r p4p_reqs.txt (line 1))
62.55   Downloading epicscorelibs-7.0.7.99.1.1a2.tar.gz (1.6 MB)
62.62      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 25.7 MB/s eta 0:00:00
62.75   Installing build dependencies: started
63.59   Installing build dependencies: finished with status 'done'
63.59   Getting requirements to build wheel: started
63.73   Getting requirements to build wheel: finished with status 'done'
63.73   Preparing metadata (pyproject.toml): started
63.87   Preparing metadata (pyproject.toml): finished with status 'done'
63.91 Collecting pvxslibs<1.4.0a1,>=1.3.1 (from p4p->-r p4p_reqs.txt (line 4))
63.92   Downloading pvxslibs-1.3.1.tar.gz (660 kB)
63.95      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 660.3/660.3 kB 27.8 MB/s eta 0:00:00
63.99   Installing build dependencies: started
87.24   Installing build dependencies: finished with status 'done'
87.24   Getting requirements to build wheel: started
87.34   Getting requirements to build wheel: finished with status 'done'
87.34   Preparing metadata (pyproject.toml): started
87.44   Preparing metadata (pyproject.toml): finished with status 'done'
87.53 Collecting numpy (from epicscorelibs->-r p4p_reqs.txt (line 1))
87.53   Downloading numpy-2.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (62 kB)
87.54 Requirement already satisfied: ply in /usr/local/lib/python3.10/site-packages (from p4p->-r p4p_reqs.txt (line 4)) (3.11)
87.56 Collecting setuptools-dso>=2.11a2 (from epicscorelibs->-r p4p_reqs.txt (line 1))
87.56   Downloading setuptools_dso-2.11a2-py2.py3-none-any.whl.metadata (1.6 kB)
87.58 Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/site-packages (from pandas->meme@ git+https://github.com/slaclab/meme@af07afe431fa4503664aca8d9f83f786f85abbf0->-r p4p_reqs.txt (line 2)) (2.9.0.post0)
87.58 Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/site-packages (from pandas->meme@ git+https://github.com/slaclab/meme@af07afe431fa4503664aca8d9f83f786f85abbf0->-r p4p_reqs.txt (line 2)) (2024.1)
87.58 Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/site-packages (from pandas->meme@ git+https://github.com/slaclab/meme@af07afe431fa4503664aca8d9f83f786f85abbf0->-r p4p_reqs.txt (line 2)) (2024.1)
87.58 Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->meme@ git+https://github.com/slaclab/meme@af07afe431fa4503664aca8d9f83f786f85abbf0->-r p4p_reqs.txt (line 2)) (1.16.0)
87.59 Downloading nose2-0.15.1-py3-none-any.whl (211 kB)
87.60 Downloading numpy-2.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (13.9 MB)
88.05    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.9/13.9 MB 31.6 MB/s eta 0:00:00
88.06 Downloading setuptools_dso-2.11a2-py2.py3-none-any.whl (23 kB)
88.09 Building wheels for collected packages: meme, p4p, epicscorelibs, pvxslibs
88.09   Building wheel for meme (setup.py): started
88.21   Building wheel for meme (setup.py): finished with status 'done'
88.21   Created wheel for meme: filename=meme-0.7.0-py3-none-any.whl size=17432 sha256=6a358a00f3e92a0a08ab9fe8c5311d109ba573690de60d3d5872534cf6308afd
88.21   Stored in directory: /tmp/pip-ephem-wheel-cache-p78zn6r9/wheels/09/cf/97/1bd7c844fc10fb8547d0072a9008a65cfaacf2858ff43489cd
88.21   Building wheel for p4p (pyproject.toml): started
92.49   Building wheel for p4p (pyproject.toml): finished with status 'error'
92.49   error: subprocess-exited-with-error
92.49   
92.49   × Building wheel for p4p (pyproject.toml) did not run successfully.
92.49   │ exit code: 1
92.49   ╰─> [88 lines of output]
92.49       <string>:11: DeprecationWarning:
92.49       
92.49         `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
92.49         of the deprecation of `distutils` itself. It will be removed for
92.49         Python >= 3.12. For older Python versions it will remain present.
92.49         It is recommended to use `setuptools < 60.0` for those Python versions.
92.49         For more details, see:
92.49           https://numpy.org/devdocs/reference/distutils_status_migration.html
92.49       
92.49       
92.49       running bdist_wheel
92.49       running build
92.49       running build_py
92.49       creating build
92.49       creating build/lib.linux-aarch64-cpython-310
92.49       creating build/lib.linux-aarch64-cpython-310/p4p
92.49       copying src/p4p/gw.py -> build/lib.linux-aarch64-cpython-310/p4p
92.49       copying src/p4p/disect.py -> build/lib.linux-aarch64-cpython-310/p4p
92.49       copying src/p4p/__init__.py -> build/lib.linux-aarch64-cpython-310/p4p
92.49       copying src/p4p/rpc.py -> build/lib.linux-aarch64-cpython-310/p4p
92.49       copying src/p4p/util.py -> build/lib.linux-aarch64-cpython-310/p4p
92.49       copying src/p4p/version.py -> build/lib.linux-aarch64-cpython-310/p4p
92.49       copying src/p4p/wrapper.py -> build/lib.linux-aarch64-cpython-310/p4p
92.49       creating build/lib.linux-aarch64-cpython-310/p4p/nt
92.49       copying src/p4p/nt/__init__.py -> build/lib.linux-aarch64-cpython-310/p4p/nt
92.49       copying src/p4p/nt/ndarray.py -> build/lib.linux-aarch64-cpython-310/p4p/nt
92.49       copying src/p4p/nt/scalar.py -> build/lib.linux-aarch64-cpython-310/p4p/nt
92.49       copying src/p4p/nt/common.py -> build/lib.linux-aarch64-cpython-310/p4p/nt
92.49       copying src/p4p/nt/enum.py -> build/lib.linux-aarch64-cpython-310/p4p/nt
92.49       creating build/lib.linux-aarch64-cpython-310/p4p/client
92.49       copying src/p4p/client/__init__.py -> build/lib.linux-aarch64-cpython-310/p4p/client
92.49       copying src/p4p/client/cli.py -> build/lib.linux-aarch64-cpython-310/p4p/client
92.49       copying src/p4p/client/thread.py -> build/lib.linux-aarch64-cpython-310/p4p/client
92.49       copying src/p4p/client/Qt.py -> build/lib.linux-aarch64-cpython-310/p4p/client
92.49       copying src/p4p/client/asyncio.py -> build/lib.linux-aarch64-cpython-310/p4p/client
92.49       copying src/p4p/client/cothread.py -> build/lib.linux-aarch64-cpython-310/p4p/client
92.49       copying src/p4p/client/raw.py -> build/lib.linux-aarch64-cpython-310/p4p/client
92.49       creating build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/test_nt.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/test_asyncio.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/asynciotest.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/__init__.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/qttest.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/test_server.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/test_sharedpv.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/test_rpc.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/test_type.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/test_gw.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/test_asLib.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/test_cothread.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/cothreadtest.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/test_value.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/test_client_raw.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/utils.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/test_qt.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       copying src/p4p/test/test_client_thread.py -> build/lib.linux-aarch64-cpython-310/p4p/test
92.49       creating build/lib.linux-aarch64-cpython-310/p4p/server
92.49       copying src/p4p/server/__init__.py -> build/lib.linux-aarch64-cpython-310/p4p/server
92.49       copying src/p4p/server/cli.py -> build/lib.linux-aarch64-cpython-310/p4p/server
92.49       copying src/p4p/server/thread.py -> build/lib.linux-aarch64-cpython-310/p4p/server
92.49       copying src/p4p/server/asyncio.py -> build/lib.linux-aarch64-cpython-310/p4p/server
92.49       copying src/p4p/server/cothread.py -> build/lib.linux-aarch64-cpython-310/p4p/server
92.49       copying src/p4p/server/raw.py -> build/lib.linux-aarch64-cpython-310/p4p/server
92.49       creating build/lib.linux-aarch64-cpython-310/p4p/asLib
92.49       copying src/p4p/asLib/lex.py -> build/lib.linux-aarch64-cpython-310/p4p/asLib
92.49       copying src/p4p/asLib/yacc.py -> build/lib.linux-aarch64-cpython-310/p4p/asLib
92.49       copying src/p4p/asLib/__init__.py -> build/lib.linux-aarch64-cpython-310/p4p/asLib
92.49       copying src/p4p/asLib/pvlist.py -> build/lib.linux-aarch64-cpython-310/p4p/asLib
92.49       copying src/p4p/example.conf -> build/lib.linux-aarch64-cpython-310/p4p
92.49       copying src/p4p/pvagw@.service -> build/lib.linux-aarch64-cpython-310/p4p
92.49       running build_ext
92.49       building 'p4p._p4p' extension
92.49       Warning: Can't read registry to find the necessary compiler setting
92.49       Make sure that Python modules winreg, win32api or win32con are installed.
92.49       INFO: C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC
92.49       
92.49       creating build/temp.linux-aarch64-cpython-310
92.49       creating build/temp.linux-aarch64-cpython-310/src
92.49       creating build/temp.linux-aarch64-cpython-310/src/p4p
92.49       INFO: compile options: '-D_GNU_SOURCE -D_DEFAULT_SOURCE -Dlinux -DUNIX -D_GLIBCXX_USE_CXX11_ABI=1 -D__PYX_EXTERN_C=extern -DPY_ARRAY_UNIQUE_SYMBOL=PVXS_PyArray_API -DPVXS_ENABLE_EXPERT_API -I/tmp/pip-build-env-yhvusxpq/overlay/lib/python3.10/site-packages/numpy/_core/include -I/tmp/pip-build-env-yhvusxpq/overlay/lib/python3.10/site-packages/epicscorelibs/include -I/tmp/pip-build-env-yhvusxpq/overlay/lib/python3.10/site-packages/pvxslibs/include -Isrc -Isrc/p4p -I/usr/local/include/python3.10 -c'
92.49       extra options: '-g0 -std=c++11'
92.49       INFO: gcc: src/p4p/_p4p.cpp
92.49       INFO: gcc: src/pvxs_client.cpp
92.49       INFO: gcc: src/pvxs_sharedpv.cpp
92.49       INFO: gcc: src/pvxs_source.cpp
92.49       INFO: gcc: src/pvxs_type.cpp
92.49       INFO: gcc: src/pvxs_value.cpp
92.49       error: Command "gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -D_GNU_SOURCE -D_DEFAULT_SOURCE -Dlinux -DUNIX -D_GLIBCXX_USE_CXX11_ABI=1 -D__PYX_EXTERN_C=extern -DPY_ARRAY_UNIQUE_SYMBOL=PVXS_PyArray_API -DPVXS_ENABLE_EXPERT_API -I/tmp/pip-build-env-yhvusxpq/overlay/lib/python3.10/site-packages/numpy/_core/include -I/tmp/pip-build-env-yhvusxpq/overlay/lib/python3.10/site-packages/epicscorelibs/include -I/tmp/pip-build-env-yhvusxpq/overlay/lib/python3.10/site-packages/pvxslibs/include -Isrc -Isrc/p4p -I/usr/local/include/python3.10 -c src/pvxs_value.cpp -o build/temp.linux-aarch64-cpython-310/src/pvxs_value.o -g0 -std=c++11" failed with exit status 1
92.49       [end of output]
92.49   
92.49   note: This error originates from a subprocess, and is likely not a problem with pip.
92.49   ERROR: Failed building wheel for p4p
92.49   Building wheel for epicscorelibs (pyproject.toml): started
111.7   Building wheel for epicscorelibs (pyproject.toml): finished with status 'done'
111.7   Created wheel for epicscorelibs: filename=epicscorelibs-7.0.7.99.1.1a2-cp310-cp310-linux_aarch64.whl size=5074916 sha256=48c1a2e610bb9c1aaf55ef8479f4e1d1a8d126e778fa68fdf33b22764b25d72c
111.7   Stored in directory: /tmp/pip-ephem-wheel-cache-p78zn6r9/wheels/d4/8b/22/f36a826f9e51101328b7af1e32bcdc10a020bb4adf23564851
111.7   Building wheel for pvxslibs (pyproject.toml): started
121.4   Building wheel for pvxslibs (pyproject.toml): finished with status 'done'
121.4   Created wheel for pvxslibs: filename=pvxslibs-1.3.1-cp310-cp310-linux_aarch64.whl size=2441106 sha256=96afddbe0de298287a8d9fb81bdb36eb56f4de595ba65fa97fc80e72149c8f4b
121.4   Stored in directory: /tmp/pip-ephem-wheel-cache-p78zn6r9/wheels/7e/f4/76/05f2f0bc6dfafa9355714578b7683c6e38a4ea4c4fb9bcd7ca
121.4 Successfully built meme epicscorelibs pvxslibs
121.4 Failed to build p4p

I've tried cloning the p4p repo and adjusting the contents of setup.py and pyproject.toml, but then I run into incompatibility issues with pvxslibs. I don't know how necessary those required dependencies are, so I don't want to just bulldoze through them all.

[finnoshea]

At the risk of commenting too much, when I change pyproject.toml to be

[build-system]
requires = [
    "setuptools",
    "setuptools_dso>=2.7a1,<=2.10; python_version<='3.11'",
    "setuptools_dso>=2.11a2; python_version>='3.12'",
    "wheel",
    "numpy",
    #"numpy>=2.0.1; python_version>='3.10'",
    "numpy<1.23; python_version>='3.10'",
    "Cython>=0.20",
    #"epicscorelibs==7.0.7.99.0.2; python_version<='3.11'",
    #"epicscorelibs>=7.0.7.99.1.1a2; python_version>='3.12'",
    "pvxslibs==1.3.1; python_version<='3.11'",
    "pvxslibs>=1.3.2a1; python_version>='3.12'",
]

in a cloned repo, I get the following error:

0.202 Processing /opt/p4p                                                                                                                                                                                                                             
0.203   Installing build dependencies: started                                                                                                                                                                                                        
25.19   Installing build dependencies: finished with status 'error'                                                                                                                                                                                   
25.19   error: subprocess-exited-with-error                                                                                                                                                                                                           
25.19   
25.19   × pip subprocess to install build dependencies did not run successfully.
25.19   │ exit code: 1
25.19   ╰─> [44 lines of output]
25.19       Ignoring setuptools_dso: markers 'python_version >= "3.12"' don't match your environment
25.19       Ignoring pvxslibs: markers 'python_version >= "3.12"' don't match your environment
25.19       Collecting setuptools
25.19         Downloading setuptools-72.2.0-py3-none-any.whl.metadata (6.6 kB)
25.19       Collecting setuptools_dso<=2.10,>=2.7a1
25.19         Downloading setuptools_dso-2.10-py2.py3-none-any.whl.metadata (1.6 kB)
25.19       Collecting wheel
25.19         Downloading wheel-0.44.0-py3-none-any.whl.metadata (2.3 kB)
25.19       Collecting numpy
25.19         Downloading numpy-2.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (62 kB)
25.19         Downloading numpy-1.22.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (2.0 kB)
25.19       Collecting Cython>=0.20
25.19         Downloading Cython-3.0.11-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (3.2 kB)
25.19       Collecting pvxslibs==1.3.1
25.19         Downloading pvxslibs-1.3.1.tar.gz (660 kB)
25.19            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 660.3/660.3 kB 28.9 MB/s eta 0:00:00
25.19         Installing build dependencies: started
25.19         Installing build dependencies: finished with status 'done'
25.19         Getting requirements to build wheel: started
25.19         Getting requirements to build wheel: finished with status 'done'
25.19         Preparing metadata (pyproject.toml): started
25.19         Preparing metadata (pyproject.toml): finished with status 'done'
25.19       Collecting epicscorelibs<7.0.7.99.2,>=7.0.7.99.1.1a2 (from pvxslibs==1.3.1)
25.19         Downloading epicscorelibs-7.0.7.99.1.1a2.tar.gz (1.6 MB)
25.19            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 29.8 MB/s eta 0:00:00
25.19         Installing build dependencies: started
25.19         Installing build dependencies: finished with status 'done'
25.19         Getting requirements to build wheel: started
25.19         Getting requirements to build wheel: finished with status 'done'
25.19         Preparing metadata (pyproject.toml): started
25.19         Preparing metadata (pyproject.toml): finished with status 'done'
25.19       INFO: pip is looking at multiple versions of epicscorelibs to determine which version is compatible with other requirements. This could take a while.
25.19       ERROR: Cannot install pvxslibs, pvxslibs==1.3.1 and setuptools_dso<=2.10 and >=2.7a1 because these package versions have conflicting dependencies.
25.19       
25.19       The conflict is caused by:
25.19           The user requested setuptools_dso<=2.10 and >=2.7a1
25.19           pvxslibs 1.3.1 depends on setuptools-dso>=2.7a1
25.19           epicscorelibs 7.0.7.99.1.1a2 depends on setuptools-dso>=2.11a2
25.19       
25.19       To fix this you could try to:
25.19       1. loosen the range of package versions you've specified
25.19       2. remove package versions to allow pip to attempt to solve the dependency conflict
25.19       
25.19       ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
25.19       [end of output]
25.19   
25.19   note: This error originates from a subprocess, and is likely not a problem with pip.
25.19 error: subprocess-exited-with-error
25.19 
25.19 × pip subprocess to install build dependencies did not run successfully.
25.19 │ exit code: 1
25.19 ╰─> See above for output.
25.19 
25.19 note: This error originates from a subprocess, and is likely not a problem with pip.

[mdavidsaver]

Apologies for the verbose dump ...

This verbosity can be avoided by attaching as a text file.