Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for manylinux2014 rollout #338

Closed
20 of 25 tasks
brainwane opened this issue Aug 16, 2019 · 36 comments
Closed
20 of 25 tasks

Tracking issue for manylinux2014 rollout #338

brainwane opened this issue Aug 16, 2019 · 36 comments

Comments

@brainwane
Copy link

brainwane commented Aug 16, 2019

Now that PEP 599 has been accepted to adopt the manylinux2014 standard, we need a tracking issue for implementation (I'm reusing structure and prose from #179).

The dates are guesses but would (as I figure) help us get all the infrastructure in place well before Nov 2020 when CentOS 6 (supported in manylinux2010) will go EOL, and in time for the August 2020 date that @gunan asked for; please argue with me. ;-)

There are a number of further steps needed before folks can actually make use of that new baseline, and the order of operations matters (since building manylinux2014 wheels isn't very useful if installers won't install them):

Prep (ideally done by 2 September 2019):

  • Gather personnel: tentatively strategize re: who's available to review & merge code, especially for repositories where maintainer time is scarce, re: who will make a build environment, and re: who will do testing or build testing infrastructure
  • Curate initial list of test cases regarding build environment, target Linux distributions, specific Python packages to be built, presence/absence of specific libraries

Essential client support (ideally done by 15 September 2019):

Enable publication of manylinux2014 wheel archives (ideally done by 30 October 2019):

Management of transition from manylinux1 and manylinux2010 (ideally done by 15 December 2019):

Additional projects to consider once core capability support has rolled out (ideally done by 1 March 2020):

Tagging @di as the PEP 599 author, @pfmoore as the BDFL-Delegate who requested further planning on rollout, and @ewdurbin from the PyPI/PSF side of things.

@trishankatdatadog
Copy link
Member

Minor comment, and probably too late now that the PEP has been accepted, but I feel it's relevant to avoid confusion: could we call it manylinux2024 (when Centos 7 actually expires) instead of manylinux2014 (sounds old and deprecated)?

@cjerdonek
Copy link
Member

Regarding this task:

Replace pip._internal.pep425tags w/ packaging.tags

Is switching pip to use packaging.tags really a blocker formanylinux2014? @di's manylinux2014 patch was super tiny, so it seems like something similar could be done to pip's pep425tags for the short-term. I'm just concerned because it seems like switching pip to use packaging.tags is relatively a much bigger task, with things to do that aren't related to the manylinux2014 changes. (These additional tasks aren't currently reflected in the checklist above.) So I'd rather not rush the switch if there isn't really a need to. With this approach, switching to packaging.tags could be done for the pip release after the next.

This was referenced Sep 22, 2019
mayeut added a commit to mayeut/warehouse that referenced this issue Sep 22, 2019
manylinux2014 platform tag is official per PEP599

see also pypa/manylinux#338
mayeut added a commit to mayeut/warehouse that referenced this issue Sep 23, 2019
manylinux2014 platform tag is official per PEP599

see also pypa/manylinux#338
@di
Copy link
Member

di commented Sep 27, 2019

Is switching pip to use packaging.tags really a blocker for manylinux2014?

No, definitely not, just figured it'd be nice to roll this in together since it seems like both are being worked on at the same time.

@di
Copy link
Member

di commented Sep 30, 2019

I just merged pypi/warehouse#6684, manylinux2014 wheels can now be uploaded to PyPI.

@xavfernandez
Copy link
Member

I've taken the liberty to add pypa/pip#7102 to the rollout.

@di
Copy link
Member

di commented Oct 11, 2019

Support in pip has been merged in pypa/pip#7102.

@xavfernandez
Copy link
Member

FYI pip 19.3 has been released with manylinux2014 support (& get-pip.py should soon follow).

@hrw
Copy link
Contributor

hrw commented Oct 24, 2019

multibuild uses Travis CI to do builds. Some time ago Travis CI got support for doing AArch64 builds.

Can you add aarch64 builds with manylinux2014 so users will not have to compile wheels on their systems?

@di
Copy link
Member

di commented Oct 24, 2019

@hrw manylinux2014 adds support for aarch64, but it will be the responsibility of individual package maintainers to add manylinux2014 wheels for their projects once all the tooling is rolled out as described in this issue.

@hrw
Copy link
Contributor

hrw commented Oct 24, 2019

@di I wonder how much work it would be to move from "send us source+wheels and we publish" to "send us source, we build wheels and publish".

Now it can take 45 minutes to do "pip install scikit-learn" on aarch64 compared to few seconds on x86-64 machine. Not counting time needed to find out which development packages to install first to get those Python packages built.

@di
Copy link
Member

di commented Oct 24, 2019

@hrw A lot. PyPI currently does zero introspection or execution of any user-supplied code. However, there are other repositories (piwheels.org, conda-forge.org) that do something similar, so it's not unprecedented.

However I think this is off-topic for this issue, a better place to discuss further would be in the PyPI issue tracker.

@matthew-brett
Copy link
Contributor

@hrw - for some wheels, it's not very difficult - if they build useful wheels with the default recipe (python setup.py install equivalent). For a lot of scientific Python wheels, there are other libraries that need building, options that need considering, and so on, and the package author needs to resolve.

@mayeut
Copy link
Member

mayeut commented Nov 7, 2019

auditwheel 3.0.0.0rc1 has been released.
The docker image PR has been updated with this.
The README still needs to be updated. Seems it's the last step for the build environment PR unless other reviewers want to give some feedback.
Before it can be merged, the corresponding quay.io repos have to be created (@takluyver ?)
Shall it be merged in master (and manylinux2010 moved in a branch) or shall it be merged in a branch keeping manylinux2010 the master until CentOS 6 EOL ? I'd go for the latter but any input is welcome.

@takluyver
Copy link
Member

I've just created manylinux2014 repositories for x86_64, i686 and aarch64:
https://quay.io/organization/pypa

Let me know if there's anything I've missed. I'm also happy to add you to the quay.io organisation - are you mayeut on quay.io as well?

@mayeut
Copy link
Member

mayeut commented Nov 8, 2019

I'm also happy to add you to the quay.io organisation - are you mayeut on quay.io as well?

@takluyver, yes, I'm mayeut on quay.io as well, thanks !

@takluyver
Copy link
Member

OK, I've invited you. If you create any more repositories, remember to give the bot account write permission for them - you can copy the details from any existing ones.

@mayeut
Copy link
Member

mayeut commented Nov 8, 2019

First draft for README pushed to #348
If there are no further reviews coming, I'll merge the PR in a manylinux2014 branch on Monday to get things going forward.

@mayeut
Copy link
Member

mayeut commented Nov 10, 2019

#348 has been merged in branch manylinux2014, first images now deployed in quay.io

@mayeut
Copy link
Member

mayeut commented Nov 11, 2019

Twine can upload manylinux2014 wheels: https://test.pypi.org/project/pybase64/1.0.1/#files

@mattip
Copy link
Contributor

mattip commented Nov 26, 2019

Is there a central location where all the supported pypa/manylinux* images are catalogued? The README here does not list them, and searching on quay.io does not seem to return anything for manylinux2014

@mayeut
Copy link
Member

mayeut commented Nov 26, 2019

@mattip, @jayfurmanek, images for manylinux2014 are catalogued in the README of the manylinux2014 branch https://github.com/pypa/manylinux/tree/manylinux2014
I thought I added a link from the master README (manylinux2010) but I didn't. Will do.
You can also see all images on quay.io https://quay.io/organization/pypa

@dwighthubbard
Copy link

Now that there is a manylinux2014 container the screwdrivercd project and the python-screwdrivercd packages used by the python_sdv4_templates screwdrivercd python templates have been updated to allow generating manylinux2010 and manylinux2014 packages and publish them using twine using the new containers.

We have updated the redislite package to publish both formats. I'd be interested in knowing if we're missing anything and if/when it might be appropriate to start promoting the manylinux2014 support.

@hrw
Copy link
Contributor

hrw commented Dec 19, 2019

@dwighthubbard next step will be building non-x86 wheels I hope.

@mattip
Copy link
Contributor

mattip commented Dec 19, 2019

I see you are not publishing a manylinux1 wheel. Are you getting complaints from people with old pip versions who do pip install (without --only-binary) and are surprised to get a sdist source tarball that they have to compile?

@ericzolf
Copy link

ericzolf commented Jan 4, 2020

Perhaps your top level README.rst file should already reference https://www.python.org/dev/peps/pep-0599 ? I got confused by manylinux2014 being listed but not referenced via a PEP.

@The-Compiler
Copy link

Is there a way to force older pip versions to install manylinux2014 wheels? I thought --platform manylinux2014 would help, but that errors out with ERROR: Can not use any platform or abi specific options unless installing via '--target'.

@The-Compiler
Copy link

Oh, another data point:

  • virtualenv v16.7.6 (2019-10-16) bundles pip 19.3 (thus works with manylinux2014)
  • However, Python's venv/ensurepip bundle pip 19.2.3 as of Python 3.8.1, thus fail to install manylinux2014 wheels.

There seems to be a PR against cpython to update the bundled versions here: python/cpython#16782

@di
Copy link
Member

di commented Jan 15, 2020

Now that pip==19.3 and auditwheel==3.0.0 have been published, and the PyPUG platform compatibility page has been updated, everything in the "Essential client support", "Enable publication" and "Management of transition" sections has been completed.

Thanks to @brainwane, @mayeut, @lkollar, @pradyunsg, @takluyver, @zachW, and especially @pfmoore for everyone's work here (as well as anyone I may be forgetting).

@brainwane
Copy link
Author

brainwane commented Jan 16, 2020

@di I still see this on the platform compat tagging page - am I missing something?

@di
Copy link
Member

di commented Jan 16, 2020

@brainwane In pypa/packaging.python.org#656 I reworded the caveat to say "manylinux2014" instead of "manylinux2010", is that not what you intended?

@brainwane
Copy link
Author

Whoops - sorry, I see that now, sorry @di .

Also, I see dockcross/manylinux2014-x64 , dockcross/manylinux2010-x64, dockcross/manylinux2010-x86, dockcross/manylinux1-x64, and dockcross/manylinux1-x86 in the dockcross README. So can we check that off?

@brainwane
Copy link
Author

@di could we take a moment sometime this week to refresh this issue, see whether anything in the checklist is finished or obsolete, and see what (if anything) is still waiting to close this?

@di
Copy link
Member

di commented Apr 7, 2020

I think we can call this finished! Thanks again everyone for your contributions here.

@di di closed this as completed Apr 7, 2020
brainwane added a commit to brainwane/peps that referenced this issue Apr 23, 2020
Per pypa/manylinux#338 , manylinux2014 is
now implemented in its entirety. Per PEP 1, since the reference
implementation is now complete and incorporated into the main source
code repository, this commit changes the status to "Final".

Signed-off-by: Sumana Harihareswara <sh@changeset.nyc>
pfmoore pushed a commit to python/peps that referenced this issue Apr 23, 2020
Per pypa/manylinux#338 , manylinux2014 is
now implemented in its entirety.
mnm678 pushed a commit to mnm678/peps that referenced this issue Oct 22, 2020
Per pypa/manylinux#338 , manylinux2014 is
now implemented in its entirety.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests