feat: Cross-compile macos arm64 wheels #1348

armenzg · 2021-09-07T20:15:46Z

This uses cibuildwheel in order to cross-compile the arm64 wheels for Python 3.8, 3.9 & 3.10.

This uses [cibuildwheel](https://cibuildwheel.readthedocs.io/en/stable/) in order to cross-compile the arm64 wheels for Python 3.8, 3.9 & 3.10. Fixes psycopg#1286

armenzg · 2021-09-07T20:17:27Z

@dvarrazzo thanks for opening this question

This PR uses the cibuildwheel action to produce the arm64 wheels that the Apple M1 machines could use.
I know you don't want to switch to cibuildwheel before version 3, however, you might be able to release only the arm64 wheels via cibuildwheel as experimental.

I did not manage to produce the universal2 package.

This can also be tested locally with `cibuildwheel --platform macos --archs arm64

armenzg · 2021-09-07T20:23:36Z

I've pushed one more commit to name correctly the upload artifact.

Meanwhile, I've tested that it installs:

❯ brew uninstall postgresql
Uninstalling /opt/homebrew/Cellar/postgresql/13.4... (3,230 files, 43.6MB)
❯ pip install ~/Downloads/packages_macos/psycopg2-2.9.1-cp38-cp38-macosx_11_0_arm64.whl --no-cache-dir
Processing /Users/armenzg/Downloads/packages_macos/psycopg2-2.9.1-cp38-cp38-macosx_11_0_arm64.whl
Installing collected packages: psycopg2
Successfully installed psycopg2-2.9.1

armenzg · 2021-09-07T20:31:12Z

The latest artifacts will show up in here:
https://github.com/armenzg/psycopg2/actions/runs/1210897910

You should be able to see packages_macos _arm64 once the workflow completes. It should be about < 9MB with the three versions.

dvarrazzo · 2021-09-07T21:17:28Z

.github/workflows/packages.yml

@@ -51,16 +50,15 @@ jobs:
          --health-timeout 5s
          --health-retries 5

-


You don't like two spaces? 🥺

I forgot to mention. prettier change the formatting of the file. I will change it.

dvarrazzo · 2021-09-07T21:18:00Z

.github/workflows/packages.yml

-          - {tag: manylinux2014, arch: i686}
-          - {tag: manylinux_2_24, arch: aarch64}
-          - {tag: manylinux_2_24, arch: ppc64le}
+          - { tag: manylinux2014, arch: x86_64 }


dvarrazzo · 2021-09-07T21:18:10Z

.github/workflows/packages.yml

  build-macos:
    runs-on: macos-10.15
    strategy:
      fail-fast: false
      matrix:
-        python-version: ['3.6', '3.7', '3.8', '3.9']
+        python-version: ["3.6", "3.7", "3.8", "3.9"]


dvarrazzo · 2021-09-07T21:25:43Z

@armenzg thank you very much for trying this out!

I have seen that tests are disabled, I assume the reference is this, right?

dvarrazzo · 2021-09-07T21:29:36Z

The latest artifacts will show up in here:
https://github.com/armenzg/psycopg2/actions/runs/1210897910

You should be able to see packages_macos _arm64 once the workflow completes. It should be about < 9MB with the three versions.

I'm not sure it worked: the zip size is only 418k

dvarrazzo · 2021-09-07T21:32:51Z

Meanwhile, I've tested that it installs:

Did you try if the result imports and passes the test suite?

Honestly releasing packages that aren't tested makes me nervous...

armenzg · 2021-09-07T23:28:06Z

Meanwhile, I've tested that it installs:

Did you try if the result imports and passes the test suite?

Honestly releasing packages that aren't tested makes me nervous...

No worries. I will look at the code tomorrow and see what you have in place for testing. Thanks for the prompt feedback.

If you have any specific code you want me to look to see how to get it tested let me know.

dvarrazzo · 2021-09-08T01:07:57Z

If you have any specific code you want me to look to see how to get it tested let me know.

What make check does :) make check tests what was just built with make; because that's not the case for you (I assume you want to test, on your laptop, what gets built by GitHub action), you can run:

python -c "import tests; tests.unittest.main(defaultTest='tests.test_suite')"

you can tweak the env vars you find in tests/testconfig.py to point the test suite to a database to trash.

What I mean is more a general thing however: future releases will build package that will not be tested automatically by the CI, at least until M1 runners are available.

armenzg · 2021-09-09T20:29:34Z

scripts/build/build_macos.sh


 # Build the wheels
 wheeldir="${prjdir}/wheels"
-pip wheel -w ${wheeldir} .
+pip wheel --use-feature=in-tree-build -w ${wheeldir} .


There's a warning to switch to this.

armenzg · 2021-09-09T20:30:02Z

scripts/build/build_macos.sh

@@ -8,22 +8,22 @@
 set -euo pipefail
 set -x

-dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"


Please ignore auto-formatting changes for now. My IDE is all messed up. I will fix it up at the very end.

armenzg · 2021-09-09T20:30:31Z

scripts/build/build_macos.sh

@@ -52,28 +48,33 @@ if [[ -z "${LIBPQ:-}" ]]; then
 fi

 delocate-wheel ${wheeldir}/*.whl
-# https://github.com/MacPython/wiki/wiki/Spinning-wheels#question-will-pip-give-me-a-broken-wheel
-delocate-addplat --rm-orig -x 10_9 -x 10_10 ${wheeldir}/*.whl
+if [[ $(uname -m) != 'arm64' ]]; then


This is to help my M1 machine skip this step.

armenzg · 2021-09-09T20:31:18Z

scripts/build/test_macos_arm64.sh

@@ -0,0 +1,23 @@
+#!/bin/bash


Ignore this script for now.

armenzg · 2021-09-09T20:32:16Z

setup.py

@@ -45,6 +45,7 @@
 # for a consistent versioning pattern.

 PSYCOPG_VERSION = '2.9.1'
+PSYCOPG_NAME = os.environ.get("PACKAGE_NAME", "psycopg2")


This removes the need to touch the file to generate pyscopg2-binary.

armenzg · 2021-09-09T20:39:18Z

If I execute the build_macos.sh on my M1 machine I can get the wheel with the included library.
I'm stuck not being able to cross-compile and for the library to be included (It comes up as 140kb).
I will file an issue with cibuildwheel or delocate and see what comes from it.

I can work around this on Sentry by installing postgresql on the developer's machine and get the wheel built on their machines.

dvarrazzo · 2021-09-09T22:02:37Z

@armenzg, if Sentry would like to help with this work maybe they could lend a machine to use as external worker and do without cross compiling?

…ci/build-m1-wheel

armenzg · 2021-10-13T20:53:56Z

@armenzg, if Sentry would like to help with this work maybe they could lend a machine to use as external worker and do without cross compiling?

Hi @dvarrazzo I worked on finding ways to fund you instead. The process is out of my hand at the moment.

lithammer · 2021-11-12T14:09:10Z

I'm not very familiar with the build/test related differences between psycopg2 and psycopg3. But isn't it possible to more or less copy the workflows from psycopg3? For example, it would be nice to get musllinux wheels (psycopg/psycopg#141) for psycopg2 as well for use with Django.

dvarrazzo · 2021-11-12T14:46:05Z

@lithammer psycopg2 doesn't use cibuildwheel. Before recently, as you know, there wasn't a standard for musl wheels. Now there is, but it's probably easier to adapt the existing build pipeline to musl than to start using cibuildwheel.

The M1 issue is different: you can cross compile but you cannot test the produced artifacts and this is not really acceptable quality IMO.

armenzg added 2 commits September 7, 2021 15:58

feat: Cross-compile macos arm64 wheels

5ef2032

This uses [cibuildwheel](https://cibuildwheel.readthedocs.io/en/stable/) in order to cross-compile the arm64 wheels for Python 3.8, 3.9 & 3.10. Fixes psycopg#1286

Adjust glob matching

96ebfa9

Name the upload artifact correctly

fe11655

This was referenced Sep 7, 2021

psycogbinary2 does not install for Apple's M1 getsentry/sentry#26465

Closed

confluent-kafka does not install on Apple's M1 chipset getsentry/sentry#28429

Closed

dvarrazzo reviewed Sep 7, 2021

View reviewed changes

armenzg added 14 commits September 8, 2021 14:56

Try again

8d55da4

Undo prettier

7e6f7d4

Undo prettier

224ccd8

We will run tests locally

b999a5c

Script to test wheel on Apple Sillicon

f3c696d

Upload repaired_wheels

6d99aa2

Formatting

ec07426

More formatting

c551f2e

Fix

1f379d0

Create directory

e4d5059

Use absolute path

618496e

Try again

2c9c31b

Try again

8176468

Faster

d780f64

armenzg added 11 commits September 9, 2021 10:35

Use CIBW_REPAIR again

3205851

Try with push

b674800

New name

89974ac

Verbosity

af32975

Use armenzg action

ad58ed7

Try main

87d4afc

Try build script

eba27a5

Try again

9fee307

Small change

9fc985b

One more try

8adcdcb

Less changes

4913603

armenzg commented Sep 9, 2021

View reviewed changes

Only Python 3.8

de4fe1e

A tiny bit more

e8ddb29

armenzg added 4 commits September 10, 2021 12:08

New script

57383fb

WIP

415eaee

Merge branch 'ci/build-m1-wheel' of github.com:armenzg/psycopg2 into …

50d74a3

…ci/build-m1-wheel

Changes for Intel

30cbc68

armenzg force-pushed the ci/build-m1-wheel branch from 80a2a94 to 30cbc68 Compare September 10, 2021 18:15

armenzg mentioned this pull request Sep 16, 2021

Cannot install confluent-kafka with provided instructions on Apple's M1 confluentinc/confluent-kafka-python#1190

Closed

7 tasks

benthorner mentioned this pull request Mar 15, 2022

(Django)(MAC Big Sur 11.0.1) ImportError: dlopen #1216

Closed

armenzg closed this Jul 11, 2022

armenzg deleted the ci/build-m1-wheel branch July 11, 2022 16:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Cross-compile macos arm64 wheels #1348

feat: Cross-compile macos arm64 wheels #1348

armenzg commented Sep 7, 2021

armenzg commented Sep 7, 2021

armenzg commented Sep 7, 2021

armenzg commented Sep 7, 2021

dvarrazzo Sep 7, 2021

armenzg Sep 7, 2021 •

edited

Loading

dvarrazzo Sep 7, 2021

dvarrazzo Sep 7, 2021

dvarrazzo commented Sep 7, 2021

dvarrazzo commented Sep 7, 2021

dvarrazzo commented Sep 7, 2021

armenzg commented Sep 7, 2021

dvarrazzo commented Sep 8, 2021

armenzg Sep 9, 2021

armenzg Sep 9, 2021

armenzg Sep 9, 2021

armenzg Sep 9, 2021

armenzg Sep 9, 2021

armenzg commented Sep 9, 2021

dvarrazzo commented Sep 9, 2021

armenzg commented Oct 13, 2021

lithammer commented Nov 12, 2021 •

edited

Loading

dvarrazzo commented Nov 12, 2021

		@@ -51,16 +50,15 @@ jobs:
		--health-timeout 5s
		--health-retries 5

feat: Cross-compile macos arm64 wheels #1348

feat: Cross-compile macos arm64 wheels #1348

Conversation

armenzg commented Sep 7, 2021

armenzg commented Sep 7, 2021

armenzg commented Sep 7, 2021

armenzg commented Sep 7, 2021

Choose a reason for hiding this comment

armenzg Sep 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dvarrazzo commented Sep 7, 2021

dvarrazzo commented Sep 7, 2021

dvarrazzo commented Sep 7, 2021

armenzg commented Sep 7, 2021

dvarrazzo commented Sep 8, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

armenzg commented Sep 9, 2021

dvarrazzo commented Sep 9, 2021

armenzg commented Oct 13, 2021

lithammer commented Nov 12, 2021 • edited Loading

dvarrazzo commented Nov 12, 2021

armenzg Sep 7, 2021 •

edited

Loading

lithammer commented Nov 12, 2021 •

edited

Loading