Sponsorship requested to release Apple M1 packages

[dvarrazzo]

We have set up a workflow to build Apple Silicon packages for psycopg.

Building these packages comes at a cost, because of the scarcity that Apple forces on this hardware and because of Apple policies requiring to over-allocate build time for Apple Silicon workers. Further development cost was required to set up non-standard build scripts and further operational time will be required at every release because the Apple Silicon build procedure doesn't integrate easily with the existing build pipelines.

As a consequence, we ask the users of the Apple Silicon binary packages to sponsor their development and release. The cost of the packages is 2,000 USD: after this sponsorship level is cumulatively reached, Apple Silicon binary packages for the current and upcoming psycopg 2 and 3 releases will be released on PyPI. New versions of the packages will be released regularly, at least until changes in the build environment and policies stop the currently set up system from working and will require further effort to be restored.

If you are interested in the package release, you can use the project sponsorship channels to contribute. Please get in touch to make sure that your sponsorship is counted towards the release of the Apple Silicon packages. If you are already a sponsor, please get in touch if you wish to count your sponsorship towards the goal of releasing Apple Silicon packages.

Thank you very much for supporting the psycopg project.

[chadwhitacre]

Please apply Sentry's sponsorship towards this. Thanks. :)

[dvarrazzo]

@chadwhitacre sure, will do. Thank you very much for your sponsorship!

[armenzg]

Would using Cirrus CI save some of the cost in here? We have some work in progress to use their M1s (free for OSS projects).

Link: https://github.com/getsentry/prebuilt-pythons/blob/main/.cirrus.yml

[dvarrazzo]

Would using Cirrus CI save some of the cost in here?

At the cost of sinking more development time.

[Dzeri96]

What is the status on the funding currently? Is the $2000 sum required monthly or per release? I want to see if I can get something going in my company to support this.

[dvarrazzo]

@Dzeri96 we are about halfway and no, it is a one-off refund.

[mrowe]

Hi @dvarrazzo, can you please apply Art Processors' sponsorship to this? Thank you!

[mrowe]

Ugh, I fat-fingered that on the form and put the wrong amount, I was aiming for twice that. It won't let me change it now, so I'll do another one next month. 😬

[zyv]

@dvarrazzo you obviously know better what and why you need to provide M1 wheels, but since we were in the same situation not being able to use Pulumi on M1 without grpcio wheels, I thought that maybe posting this will help you some.

Basically, GitHub has free macOS runners, and even though M1 is not available yet, you can easily cross-compile to M1 with cibuildwheel. This is not qemu emulation, so it runs totally fast and only needs one workflow file:

https://github.com/moneymeets/grpcio-mac-arm-build

name: Build

on:
  workflow_dispatch:
    inputs:
      version:
        description: "Version of grpcio"
        type: string
        required: true
        default: "1.49.0"

jobs:
  build_wheels:
    name: Build grpcio==${{ github.event.inputs.version }} wheels on ${{ matrix.os }}, for Python ${{ matrix.python_build_version }}
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ "macOS-12" ]
        python_build_version: [ "cp310", "cp311" ]

    steps:
      # Used to host cibuildwheel
      - uses: actions/setup-python@v3
      
      - name: Download grpcio tarball
        run: |
          pip download --no-binary grpcio grpcio==${{ github.event.inputs.version }}
          tar xf *.tar.gz
          
      - name: Install cibuildwheel
        run: python -m pip install cibuildwheel==2.9.0

      - name: Build wheels
        run: |
          cd grpcio-*/
          python -m cibuildwheel --output-dir ../wheelhouse
        env:
          CIBW_ARCHS_MACOS: "arm64"
          CIBW_TEST_SKIP: "*_arm64"
          CIBW_BUILD: "${{ matrix.python_build_version }}-macosx_arm64"
          CIBW_BUILD_FRONTEND: pip
          CIBW_ENVIRONMENT_MACOS: "GRPC_PYTHON_BUILD_WITH_CYTHON=1"
          CIBW_BEFORE_BUILD: "pip install -r requirements.txt"

      - uses: actions/upload-artifact@v3
        with:
          name: artifacts-${{ github.event.inputs.version }}
          path: ./wheelhouse/*.whl

Kudos to @pietrodn who wrote the original code.

[zyv]

P.S. I think I understand now why the solution with simply using cross-compiling on GitHub (like shown above) has been discarded after reading the whole of #1286 - you can only test resulting artifacts via super-slow qemu emulation or opt for not testing them at all (apparently not an option for the maintainers).

It would be good to have an update in this ticket regarding the current state - I poked around a bit, and could not quickly find a way to see whether the goal has been reached already, and if not, how much is left and how to send money.

[dvarrazzo]

@zyv so far about $1500 have been collected

[zyv]

@zyv so far about $1500 have been collected

I have arranged a budget of $500 for M1 wheels via my company. Hope that helps...

[dvarrazzo]

Thank you very much, @zyv. The M1 wheels should be released in the coming days.

[dvarrazzo]

Apple M1 packages for psycopg2-binary version 2.9.3 are now available on PyPI.

Thank you very much to all the contributors. Happy hacking!