Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Publish PEP 517 sdist #8

Closed
ijl opened this issue Jan 24, 2019 · 29 comments
Closed

Publish PEP 517 sdist #8

ijl opened this issue Jan 24, 2019 · 29 comments

Comments

@ijl
Copy link
Owner

ijl commented Jan 24, 2019

Needs PyO3/maturin#2.

@gatopeich
Copy link

Meanwhile, is there any other way to build orjson on alpine Linux?

@ijl
Copy link
Owner Author

ijl commented Mar 19, 2019

You can try the instructions in https://github.com/ijl/orjson#install. Building a pyo3 project against Python using musl libc may not succeed right now, I don't know.

@bsergean
Copy link

I would love to use orjson as well, but I like my small alpine containers.

Sounds like pyo3 is the project that would know about musl / alpine linux things.

@bsergean
Copy link

The pyo3 project says to use --manylinux when building.

(your instructions do not mention this flag)

git clone https://github.com/ijl/orjson.git && cd orjson
pip install --upgrade pyo3-pack
pyo3-pack build --release --strip --interpreter python3.7

I'll give it a try to see if this can be built from scratch on Alpine.

@ijl
Copy link
Owner Author

ijl commented Aug 29, 2019

2.0.7 includes an sdist using pyo3-pack.

@konstin, thank you for adding that to pyo3-pack.

@ijl ijl closed this as completed Aug 29, 2019
@bsergean
Copy link

I should probably create a different github issue, but here's my failed attempt at installing 2.0.7 on Alpine (in docker). I tried with and without the --user flag, as root and as a normal user. I have (latest ?) pip (19.2.3).

 ---> Running in f6955bd4d145
Collecting orjson==2.0.7
  Downloading https://files.pythonhosted.org/packages/8c/9a/f1d72af21340fae73b5ae0d75f015ee7caf27c808e0fae3659d2246ff1e7/orjson-2.0.7.tar.gz (480kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-99ysc57q/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'pyo3-pack>=0.7.0.beta11,<0.8'
       cwd: None
  Complete output (33 lines):
  Collecting pyo3-pack<0.8,>=0.7.0.beta11
    Downloading https://files.pythonhosted.org/packages/15/90/9c0ae42b8c9bfcad8a897353185f64ba47064d898e99b89957695858251f/pyo3_pack-0.7.0_beta.11.tar.gz (78kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    ERROR: Command errored out with exit status 2:
     command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-hc8y5lnr/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- pyo3-pack
         cwd: None
    Complete output (23 lines):
    Collecting pyo3-pack
      Using cached https://files.pythonhosted.org/packages/15/90/9c0ae42b8c9bfcad8a897353185f64ba47064d898e99b89957695858251f/pyo3_pack-0.7.0_beta.11.tar.gz
    ERROR: Exception:
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in main
        status = self.run(options, args)
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 345, in run
        resolver.resolve(requirement_set)
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 196, in resolve
        self._resolve_one(requirement_set, req)
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 359, in _resolve_one
        abstract_dist = self._get_abstract_dist_for(req_to_install)
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 307, in _get_abstract_dist_for
        self.require_hashes
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 213, in prepare_linked_requirement
        with self.req_tracker.track(req):
      File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
        return next(self.gen)
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 94, in track
        self.add(req)
      File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 63, in add
        % (link, fp.read()))
    LookupError: https://files.pythonhosted.org/packages/15/90/9c0ae42b8c9bfcad8a897353185f64ba47064d898e99b89957695858251f/pyo3_pack-0.7.0_beta.11.tar.gz#sha256=93bf2789bb5e27c57af4b9b9b03f48ed84ff99c62852626c9ac51036845df6d3 (from https://pypi.org/simple/pyo3-pack/) is already being built: pyo3-pack<0.8,>=0.7.0.beta11 from https://files.pythonhosted.org/packages/15/90/9c0ae42b8c9bfcad8a897353185f64ba47064d898e99b89957695858251f/pyo3_pack-0.7.0_beta.11.tar.gz#sha256=93bf2789bb5e27c57af4b9b9b03f48ed84ff99c62852626c9ac51036845df6d3
    ----------------------------------------
  ERROR: Command errored out with exit status 2: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-hc8y5lnr/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- pyo3-pack Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-99ysc57q/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'pyo3-pack>=0.7.0.beta11,<0.8' Check the logs for full command output.
The command '/bin/sh -c pip install orjson==2.0.7 --user' returned a non-zero code: 1

@ijl
Copy link
Owner Author

ijl commented Aug 29, 2019

musl libc isn't supported.

@bsergean
Copy link

bsergean commented Aug 29, 2019 via email

@bsergean
Copy link

bsergean commented Aug 29, 2019 via email

@gatopeich
Copy link

I managed to get it working on Alpine Linux by hacking the dist into believing it is "manylinux" with a couple lines.
Will share it later...

@gatopeich
Copy link

gatopeich commented Aug 30, 2019

Here is my Docker recipe to have orjson on Alpine Linux:

FROM python:3.7-alpine
RUN echo 'manylinux1_compatible = True' > /usr/local/lib/python3.7/_manylinux.py
RUN pip3 install orjson==2.0.9
RUN rm /usr/local/lib/python3.7/_manylinux.py

And yes it works.

Update: versions > 2.0.9 fail with "_register_atfork: symbol not found"

@bsergean
Copy link

bsergean commented Aug 30, 2019 via email

@goodking-bq
Copy link

Here is my Docker recipe to have orjson on Alpine Linux:

FROM python:3.7-alpine
RUN echo 'manylinux1_compatible = True' > /usr/local/lib/python3.7/_manylinux.py
RUN pip3 install orjson
RUN rm /usr/local/lib/python3.7/_manylinux.py

And yes it works.

i had try this,success install ,,but import raise error

/ # python -c "import orjson"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: Error relocating /usr/local/lib/python3.7/site-packages/orjson.cpython-37m-x86_64-linux-gnu.so: __register_atfork: symbol not found

have any idear?

@gatopeich
Copy link

ImportError: Error relocating /usr/local/lib/python3.7/site-packages/orjson.cpython-37m-x86_64-linux-gnu.so: __register_atfork: symbol not found

This annoying symbol dependency was added in v2.0.10.
My current solution is to install orjson==2.0.9

@scuml
Copy link

scuml commented Jan 2, 2020

Got the latest orjson working on on alpine3.11

FROM python:3.8.1-alpine3.11 as base

# Install rustup to get nightly build
RUN wget -O init.sh https://sh.rustup.rs
RUN sh init.sh -y
RUN cp $HOME/.cargo/bin/* /usr/local/bin

RUN rustup install nightly
RUN rustup default nightly

# use static linking to allow rust to compile orjson
ENV RUSTFLAGS "-C target-feature=-crt-static"

RUN pip install orjson

@gatopeich
Copy link

Got the latest orjson working on on alpine3.11
...
ENV RUSTFLAGS "-C target-feature=-crt-static"

How does this install orjson?

@scuml
Copy link

scuml commented Jan 7, 2020

How does this install orjson?

This allows one to install a python requirements file that includes the latest orjson. Without these lines, one will run into a compilation error. I've added RUN pip install orjson to the end to help clarify.

@ashleysommer
Copy link

ashleysommer commented Mar 5, 2020

An update for anyone coming to this issue looking to run newest orjson on Alpine.
I've worked out how to do it on vanilla alpine without installing glibc and without using the rust toolchain. (I'm using alpine 3.9 and python 3.6 but should work with newer versions).
Its based on the comment by @gatopeich above: #8 (comment)
The key is that orjson v2.5.1 now builds manylinux1-compatible wheels.
Also required is gcompat (a true manylinux1-compatible build shouldn't require gcompat, but alas this seems to still need it)
Here's my alpine recipe:

FROM alpine:3.9
RUN echo "https://dl-3.alpinelinux.org/alpine/v3.9/main" >> /etc/apk/repositories
RUN echo "https://dl-3.alpinelinux.org/alpine/v3.9/community" >> /etc/apk/repositories
RUN apk add --no-cache python3 gcompat patchelf
RUN patchelf --add-needed libgcompat.so.0 /usr/bin/python3.6
RUN echo 'manylinux1_compatible = True' > /usr/lib/python3.6/_manylinux.py &&\
    pip3 install "orjson>=2.5.1" &&\
    rm /usr/lib/python3.6/_manylinux.py

I have this running in two different performance-sensitive web applications right now and seems to be doing perfectly fine.

@ashleysommer
Copy link

ashleysommer commented Mar 5, 2020

Tested, the above recipe also works on Alpine 3.10, with Python 3.7 using the following recipe, but it doesn't work on Alpine 3.11, with Python 3.8.

FROM alpine:3.10
RUN echo "https://dl-3.alpinelinux.org/alpine/v3.10/main" >> /etc/apk/repositories
RUN echo "https://dl-3.alpinelinux.org/alpine/v3.10/community" >> /etc/apk/repositories
RUN apk add --no-cache python3 gcompat patchelf
RUN patchelf --add-needed libgcompat.so.0 /usr/bin/python3.7
RUN echo 'manylinux1_compatible = True' > /usr/lib/python3.7/_manylinux.py &&\
    pip3 install "orjson>=2.5.1" &&\
    rm /usr/lib/python3.7/_manylinux.py

When trying the same in Alpine 3.11 and Python 3.8, the python3 executable always crashes after being patched with gcompat, this includes running running normal python applications like PIP too.

@gatopeich
Copy link

gatopeich commented Apr 18, 2020

Working in easy steps with gcompat + patchelf, without explicit patching, on alpine:3.11:

$ docker run -it alpine:3.11                       
/ # apk add --no-cache py3-pip gcompat patchelf
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/15) Installing libucontext (0.1.3-r1)
(2/15) Installing gcompat (0.4.0-r0)
(3/15) Installing libgcc (9.2.0-r4)
(4/15) Installing libstdc++ (9.2.0-r4)
(5/15) Installing patchelf (0.10-r0)
(6/15) Installing libbz2 (1.0.8-r1)
(7/15) Installing expat (2.2.9-r1)
(8/15) Installing libffi (3.2.1-r6)
(9/15) Installing gdbm (1.13-r1)
(10/15) Installing xz-libs (5.2.4-r0)
(11/15) Installing ncurses-terminfo-base (6.1_p20200118-r3)
(12/15) Installing ncurses-libs (6.1_p20200118-r3)
(13/15) Installing readline (8.0.1-r0)
(14/15) Installing sqlite-libs (3.30.1-r1)
(15/15) Installing python3 (3.8.2-r0)
Executing busybox-1.31.1-r9.trigger
OK: 66 MiB in 29 packages
/ # echo 'manylinux1_compatible = True' > /usr/lib/python3.8/_manylinux.py
/ # pip3 install orjson
Collecting orjson
  Downloading https://files.pythonhosted.org/packages/98/90/a82ea40bc8615e227bad34b
66be60f40978044c8f098968c7ec39a765fb8/orjson-2.6.5-cp38-cp38-manylinux1_x86_64.whl
(222kB)
     |████████████████████████████████| 225kB 12.5MB/s
Installing collected packages: orjson
Successfully installed orjson-2.6.5
WARNING: You are using pip version 19.2.3, blah blah...
/ # python3
Python 3.8.2 (default, Feb 29 2020, 17:03:31)
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import orjson
>>> orjson.dumps(list(range(10)))
b'[0,1,2,3,4,5,6,7,8,9]'
>>> _

@ashleysommer
Copy link

That's great news. Perhaps it's a newer version of gcompat now on alpine 3.11?

@gatopeich
Copy link

gatopeich commented Apr 18, 2020

Must be a combination of gcompat and patchelf because if I don't install patchelf there is an import error.
But note I am not using it directly.

@NotoriousRebel
Copy link

@gatopeich Any chance you know why it doesn't work on python3.8-alpine3.10?

@gatopeich
Copy link

@NotoriousRebel there was some library symbol missing on 3.10, and no supported way to amend it.

@ashleysommer
Copy link

ashleysommer commented Dec 4, 2020

I know its been 6 months since I've commented here, but revisiting this now for a different project I'm working on, and I've investigated the issue with Python 3.8 and Alpine 3.11 a bit more.
The reason it worked for @gatopeich in April, but did not work for me in March, is not due to a different gcompat or patchelf version. But actually due to a newer orjson version. I was using orjson version 2.5.1 (the first with proper manylinux1 compatibility) but @gatopeich in April used the newer orjosn v2.6.5. Theres some reason due to a combination of using Python 3.8 and orjson 2.6.5, that means the python3 binary does not need to be patched using patchelf.
gcompat still needs to be installed, because orjson needs ld-linux-x86_64.so.2 library that is provided by gcompat.

However, in the last 6 months there has been a lot of changes in orjson, particularly in relation to building and publishing. Eg, manylinux1 builds are no longer published since 3.4.0, so we need to use v3.3.1 or older. Also, orjson v2.6.5 and above requires libgcc_s.so.1 that is provided by libgcc. That is why it worked for @gatopeich when they installed patchelf. Even though patchelf isn't used, it depends on libgcc which brings in that library.

So here is a basic docker recipe for using orjson in alpine v3.11 with Python 3.8

FROM alpine:3.11
RUN echo "https://dl-3.alpinelinux.org/alpine/v3.11/main" >> /etc/apk/repositories
RUN echo "https://dl-3.alpinelinux.org/alpine/v3.11/community" >> /etc/apk/repositories
RUN apk add --no-cache python3 gcompat libgcc
RUN echo 'manylinux1_compatible = True' > /usr/lib/python3.8/_manylinux.py &&\
    pip3 install "orjson>=2.6.5,<3.4.0" &&\
    rm /usr/lib/python3.8/_manylinux.py
RUN python3 -c "import orjson; print(orjson.dumps(list(range(10))))"

@ashleysommer
Copy link

And finally, I just did a build on Alpine 3.12 with Python 3.8.5. Theres not much different here, except we need to install py3-pip to use pip3. In this case we can now use the manylinux2014 builds, so we can use orjson v3.3.1 up to the current version.

FROM alpine:3.12
RUN echo "https://dl-3.alpinelinux.org/alpine/v3.12/main" >> /etc/apk/repositories
RUN echo "https://dl-3.alpinelinux.org/alpine/v3.12/community" >> /etc/apk/repositories
RUN apk add --no-cache python3 py3-pip gcompat libgcc
RUN echo 'manylinux2014_compatible = True' > /usr/lib/python3.8/_manylinux.py &&\
    pip3 install "orjson>=3.3.1" &&\
    rm /usr/lib/python3.8/_manylinux.py
RUN python3 --version
RUN python3 -c "import orjson; print(orjson.dumps(list(range(10))))"

@aspin
Copy link

aspin commented Dec 23, 2020

The above solution doesn't seem to work for the latest version of pip. For now, I've pinned pip==20.2.2 when trying to install orjson. Not really sure what's changed in the latest pip (20.3.3 as of writing), but on my image it doesn't seem to even get to the check for the _manylinux.py file, seemingly because pip isn't able to find glibc in the system

@gatopeich
Copy link

gatopeich commented Mar 21, 2021

And finally, I just did a build on Alpine 3.12 with Python 3.8.5.

Thanks, @ashleysommer, that works like a charm even today with Python 3.8.8 and orjson 3.5.1:

$ docker run -it alpine:3.12                                                                                                   
/ # echo "https://dl-3.alpinelinux.org/alpine/v3.12/main" >> /etc/apk/repositories                                                                           
/ # echo "https://dl-3.alpinelinux.org/alpine/v3.12/community" >> /etc/apk/repositories                                                                      
/ # apk add --no-cache python3 py3-pip gcompat libgcc                                                                                                        
...
(40/40) Installing py3-pip (20.1.1-r0)
...
/ # echo 'manylinux2014_compatible = True' > /usr/lib/python3.8/_manylinux.py
/ # pip3 install "orjson>=3.3.1"
Collecting orjson>=3.3.1
  Downloading orjson-3.5.1-cp38-cp38-manylinux2014_x86_64.whl (233 kB)
     |████████████████████████████████| 233 kB 2.6 MB/s
Installing collected packages: orjson
Successfully installed orjson-3.5.1
/ # rm /usr/lib/python3.8/_manylinux.py
/ # python3 --version
Python 3.8.8
/ # python3 -c "import orjson; print(orjson.dumps(list(range(10))))"
b'[0,1,2,3,4,5,6,7,8,9]'

@aspin, I can only guess you are not using the bundled py3-pip for your own reasons.

@AngryUbuntuNerd
Copy link

I came up with a solution combining a few of the above mentioned:

FROM python:3.6-alpine

RUN echo 'manylinux2014_compatible = True' > /usr/local/lib/python3.6/_manylinux.py && \
    apk add --no-cache gcompat libgcc && \
    pip install "pip==20.2.2" --user && \
    pip install "orjson~=3.5.1" && \
    pip uninstall -y pip && \
    rm /usr/local/lib/python3.6/_manylinux.py

...  # Install everything else (pip is back to the image version)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants