-
Notifications
You must be signed in to change notification settings - Fork 216
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Drop support for manylinux1 images January 1st, 2022 #994
Comments
Just to be complete, since the “very little need” is misleading, the reason 99% users get manylinux1 is not glibc version, but pip version. The version of pip in the most popular distro (Ubuntu), the most popular version (18.04) is pip 9. Even if you install Python 3.8 (it has 3.6 by default), you still get Pip 9. It’s quite unsupported to overwrite the system package manager’s pip, too. Making a new venv defaults to pip 9 even if you do update pip manually (such as a user install). So even though manylinux2014 is supported with a new enough pip, using the default tooling, only manylinux1 is. This issue should be fixed from recurring in the future by the perennial manylinuxes, once distros ship that. Edit: forgot about CentOS 7! |
Can you provide the source of this stat ? The stats in the linked comment suggested less than 25% last October I'm hopping that by the end of the year, things will have evolved in the right direction. Some major packages like You'll still be able to use |
I updated the stats by looking at February downloads on PyPI:
When looking at the data for all x86 cpus, this gives: When looking only at 32-bit x86 (which might lessen the CI bias a bit): |
I didn’t mean 99% of users can only get manylinux1, I meant that 99% of the times that manylinux1 is selected when manylinux2010 is available, it’s due to pip, not glibc limitations (the stat is easily computable from your stats above). I know there are a lot of (non-CI) Ubuntu 18.04 users because My point is this is a huge number of users on an OS with three years left to go, many of whom have no idea about virtual environments or other such things, and they will all suddenly be compiling when pip installing if a project drops manylinux1. I expect that’s why manylinux1 is still used by 75% of the cibuildwheel projects even though the default is manylinux2010. manylinux1 does have to die, and the best way is for everyone to drop it at the same time. But wanted to clarify it’s not only the “less than 0.1% of glibc systems” stat that matters. |
In your data, you can see CentOS 7 and Ubuntu 18.04 clearly listed with pip 9; there’s also a mystery 2.26 distro with pip 9. |
Thanks for the explanation of your stat. Indeed 99% of systems not manylinux2010 compliant systems are not compliant only because of
Again, projects do not have to drop manylinux1 and would still be able to use the last. The image just won't evolve as there won't be use for it to evolve unless some CPython 3.10 appears with pip <= 19.x Yes that means on packager side, there would be a need to split the build depending on CPython version being built. I guess python3.8 was added after in ubuntu 18.04 and it shares pip with the default python 3.6 so 9.x |
yes
|
Seeing Python version there could also be interesting. |
I dropped the patch version on everything not to get too many rows. https://docs.google.com/spreadsheets/d/1ZV-xf-2qXkPUdPp8D8i4qRq6Jo1GSNlJQBGuLyVkiGQ/edit?usp=sharing
|
Didn't polish off the spacing and clean the really small segments, but here's a basic pip & manylinux version pie chart for each Python version. Stuck 2.6 in to make the grid nicer. :) from hist import Hist
import pandas as pd
import matplotlib.pyplot as plt
data = pd.read_csv(
"results-20210227-133657 - results-20210227-133657.csv",
usecols=("cpu", "num_downloads", "python_version", "pip_version", "glibc_version", "policy"),
converters={
"python_version": str,
"pip_version": lambda x: int(x.split(".")[0]),
"glibc_version": lambda x: int(float(x.split("-")[0]) % 1 * 100),
},
)
h = Hist.from_columns(
data,
("cpu", "python_version", "pip_version", "policy"),
weight="num_downloads",
)
fig, axs = plt.subplots(2, 3, figsize=(12, 8))
for i, py in enumerate(["2.6", "2.7", "3.6", "3.7", "3.8", "3.9"]):
ax = axs.flatten()[i]
ph = h.project("python_version", "pip_version")[py, :]
ph.plot_pie(ax=ax, normalize=True, autopct='%1.0f%%', pctdistance=.8)
ax.set_title(f"Python {py} {int(ph.sum()) // 1000000:,} M")
plt.tight_layout()
plt.show()
fig, axs = plt.subplots(2, 3, figsize=(12, 8))
for i, py in enumerate(["2.6", "2.7", "3.6", "3.7", "3.8", "3.9"]):
ax = axs.flatten()[i]
ph = h.project("python_version", "policy")[py, :]
ph.plot_pie(ax=ax, normalize=True, autopct='%1.0f%%', pctdistance=.8)
ax.set_title(f"Python {py} {int(ph.sum()) // 1000000:,} M")
plt.show() Quick observation; Pip 9 is frighteningly large for 3.7 (6%) and even for Python 3.8 (3%). But even then, nothing like 3.6 (17%), so dropping Python 3.6 might be good point for most projects to drop manylinux1. |
Thanks @henryiii for those. c.f. also https://discuss.python.org/t/blog-post-about-manylinux-and-the-future-of-manylinux1/5734
I agree with you however, I think the can be guided into learning those and that would be beneficial for everybody. Let's see how it evolve in the coming months. |
for those who still want to use manylinux1 a bit more, its possible to add python 3.10 on top of the deprecated image using pypa scripts: |
Talking in the Pygame Discord, I came up with an idea that may help projects decommission manylinux1. The problem that Pygame (and projects that depend on it, like mine) has faced with wheels is that when wheels are unavailable pip falls back to trying to build an sdist. For Pygame this fails on Linux much more than it succeeds: you need a ton of dependencies. And if you have some but not all of the dependencies you can succeed in building a Pygame that is missing some capabilities (reduced set of sound and image file formats). Desupporting wheels for any platform/interpreter means that users will file issues saying that the package doesn't install on their system. I suggested filling slots we do not support in the compatibility matrix of wheels with deliberately bad wheels that refuse to install, but with a message that suggests which direction to travel to get a supported wheel (upgrade pip, upgrade Python, etc.) AFAIK wheels cannot themselves crash pip with an error so I suggested having empty wheels that have a dependency on an sdist-only package that contains the message (I made one). |
Here's the October 2021 data: Based on data https://github.com/mayeut/manylinux-timeline, code at https://gist.github.com/4f0715e64df7cf267d2d2490b5facee9. |
Quick question regarding which manylinux package a project should provide by default. Apart from missing Python 3.10 support, would it be sufficient to only provide a |
You can provide the lowest one, it will have the best compatibility. If you can build it with a newer image, that's even better - since the manylinux1 images are going to be discontinued soon. The final output depends on what glibc features are used. It doesn't make sense to try to build one older than your dependencies, though - so if you require NumPy, then why make a manylinux2010 Python 3.10 wheel? Anyone pre-2014 would have to build NumPy from SDist just to use your wheel before manylinux2014. |
OK, this makes sense.
I'll use the lowest supported one then.
How do I determine which manylinux to use from my dependencies? For example, sleepecg requires NumPy, and I'm building a manylinux2014 wheel. However, this workflow produces three additional manylinux1 wheels that seem to be copies of the 2014 ones. I have no idea what's up with those automatically generated manylinux1 wheels. |
This is the behavior of The behavior changed in Given the project in the example does not target python 2.7 or 3.5, I'd highly recommend updating cibuildwheel to the latest version c.f. also https://cibuildwheel.readthedocs.io/en/stable/faq/#automatic-updates in order to get newer manylinux images by default, support 3.10, configuration in
To my knowledge, if you want to match the ones used for your dependencies, the easiest is to check manually what's been deployed in PyPI. If you use cibuildwheel 2.2+, given wheels provided by numpy based on versions in https://pypi.org/project/oldest-supported-numpy, the
|
Thanks @mayeut, upgrading to the latest version solved the issue! |
This is to allow for building the Linux wheels with the more recent 'manylinux2010' image instead of the 'manylinux1' image. 'manylinux1' does not support CPython 3.10 pypa/manylinux#994 List of manylinux images: https://github.com/pypa/manylinux#docker-images
The active support for manylinux1 is now over. |
@mayeut Should this be closed now? |
|
The update workflow started to fail. |
CentOS 5 support - which is used as the base image for
manylinux1
- has been EOL for almost 4 years.There's now very little need for
manylinux1
compatible wheels (less than 0.1% of glibc systems downloads on PyPI in October 2020, c.f. #542 (comment))manylinux1
will not receive any new features from now on (e.g. no CPython 3.10).Support will end January 1st, 2022.
What this means on January 1st, 2022:
manylinux1
branch will be removed (a tag will be created to keep history)Amend on January 22nd, 2022:
The text was updated successfully, but these errors were encountered: