feat: move to flit backend #501

henryiii · 2023-01-27T18:47:23Z

This is an updated version and closes #436. Besides rebasing, this moves from using an adaptor of flake8 to using Ruff. Adaptors for pyproject.toml for flake8 are constantly (intentionally?) broken by changes in flake8. Ruff is Rust reimplementation that is 10-100x faster, has no dependencies, can do the work of multiple tools (and versions can't get out of sync, since it is one compiled package - that's why #499 is failing, flake8-bugbear updated), supports auto-fixing, and is natively configured via pyproject.toml. Build & cibuildwheel have already moved to Ruff.

Switched build backend to flit (original rebased and squashed commit from Switched build backend to flit #436)
chore: move to using Ruff
fix: include everything but setup.py in standards-based SDist builds

This solves the chicken-and-egg problem with setuptools and wheel when bootstrapping. Build and packaging have already moved to flit-core.

codecov · 2023-01-27T18:50:17Z

Codecov Report

Patch and project coverage have no change.

Comparison is base (182efc0) 68.35% compared to head (ce34fad) 68.35%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #501   +/-   ##
=======================================
  Coverage   68.35%   68.35%           
=======================================
  Files          12       12           
  Lines         929      929           
=======================================
  Hits          635      635           
  Misses        294      294

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

agronholm · 2023-01-27T21:22:24Z

Before you go any further, I would probably be only interested in switching to flit in v1.0.0 (the publicapi branch). I don't want to risk any compatibility issues in the v0.X series.

henryiii · 2023-01-27T22:13:16Z

I would probably be only interested

IMO, it would be easier to separate changes (this, then publicapi in a subsequent step). Also the wheel of wheel is unchanged; this only affects packaging distributors who want to build from sources, or a pure SDist build. It should not be a breaking change to anyone else, because it's not a user-facing change. Doing everything in one massive "1.0" release will likely make it unclear where breakages come from.

We've moved over packaging and build without any major issues, so I think this is fairly safe, and finally breaks the wheel - setuptools dependency cycle. Up to you, though.

Also, I'm following your lead of adding the backward-compatibility "setup.py" in the SDist. I don't think it's very useful, someone can use installer or bootstrap flit-core, but it's still there.

agronholm · 2023-01-28T11:09:30Z

this only affects packaging distributors who want to build from sources, or a pure SDist build.

And this precisely has been the source of trouble every time I try making a "radical" change to wheel.

henryiii · 2023-01-30T21:22:50Z

Yes, because you tried to move a year or two ago before anyone else had moved. Now packaging and build have moved, so I think most of the early-adopter pains have been dealt with - most distros should know how to package an non-setuptools package now. I can't promise it will be perfectly smooth, but it should be much easier than before. And one step at a time will be easier too - it's easier to deal with packagers and users separately - making a change that breaks both of them at the same time isn't ideal. ;)

FYI, running isort via pre-commit was broken two days ago by Poetry-core 1.5.0 (and they are the ones pushing for major version pinning, 🤣) - this branch works, but the main branch has a broken pre-commit due to isort. (noticed working on #422). Updating isort fixes it.

agronholm · 2023-01-30T21:57:00Z

Yes, because you tried to move a year or two ago before anyone else had moved. Now packaging and build have moved, so I think most of the early-adopter pains have been dealt with - most distros should know how to package an non-setuptools package now. I can't promise it will be perfectly smooth, but it should be much easier than before. And one step at a time will be easier too - it's easier to deal with packagers and users separately - making a change that breaks both of them at the same time isn't ideal. ;)

Neither packaging or build are used (vendored versions don't count) by the people who were previously affected by this. The ones immediately affected were people doing pip install --no-binary=:all:. I do indeed hope that things have improved since my last attempt. I will at least give people a heads-up at discuss.python.org a week before the next release that includes this PR.

FYI, running isort via pre-commit was broken two days ago by Poetry-core 1.5.0 (and they are the ones pushing for major version pinning, rofl) - this branch works, but the main branch has a broken pre-commit due to isort. (noticed working on #422). Updating isort fixes it.

I'm curious: what does this have to do with major version pinning?

henryiii · 2023-01-30T22:08:32Z

This should help with pip install --no-binary=:all:, as you no longer have a circular dependency with setuptools. This will affect users who are not using isolated builds, which is heavily packaging distributors. Those distributors are (hopefully) shipping at least those packages, and probably more, so have learned how to ship a package with a non-setuptools backend by now.

Poetry-core made a breaking change in a minor release that broke at least one major package (isort - which happens to be used via pre-commit, which builds from SDists unless you make a mirror). Poetry strongly insists on following SerVer and major version pinning (like poetry-core<2), while a lot of people (like me, I also link to others) really, really don't like. And they claim it protects you from exactly what they just did. :)

agronholm · 2023-01-30T22:13:15Z

Poetry-core made a breaking change in a minor release that broke at least one major package (isort - which happens to be used via pre-commit, which builds from SDists unless you make a mirror). Poetry strongly insists on following SerVer and major version pinning (like poetry-core<2), while a lot of people (like me, I also link to others) really, really don't like. And they claim it protects you from exactly what they just did. :)

That depends on whether or not they knew that the change was breaking. SemVer does give consumers some degree of protection from breaking changes, but that doesn't automatically protect from developer mistakes. I take it that they reverted the breaking change soon after? That is at least how SemVer is supposed to work.

agronholm · 2023-01-30T22:15:46Z

I went and fixed the pre-commit failure on main, and that caused the conflict you're seeing here. Sorry about that.
I'll give this PR a review either today or tomorrow.

henryiii · 2023-01-31T13:44:58Z

That's fine, I don't mind rebasing.

I take it that they reverted the breaking change

Nope, they considered it isort's problem.

agronholm

Some questions, remarks and change requests.

.pre-commit-config.yaml

pyproject.toml

tests/test_sdist.py

.github/workflows/publish.yml

.pre-commit-config.yaml

tox.ini

henryiii · 2023-01-31T17:08:46Z

I resolved the things I addressed with code changes. If I just respond, I didn't resolve. Tox's pyproject.toml support is... interesting. But I guess it works. :)

agronholm · 2023-02-01T00:26:11Z

Tox was supposed to get proper pyproject.toml support in v4.0 but IIRC it was deferred to a later release.

Per PEP-518 [0], for projects using setuptools as the build backend, both setuptools and wheel are to be specified, as wheel is a plugin for setuptools to support the format at all. Additionally, wheel itself will soon [1] not depend on setuptools, but rather the dependency tree will reverse. With hat: python Approved by: mentors (implicit), swills (maintainer, implicit) References: https://peps.python.org/pep-0518/#build-system-table [0] pypa/wheel#501 [1]

vishwin · 2023-02-24T23:43:37Z

Is test_sdist the only remaining blocker?

pyproject.toml

Fixed flit_scm installation Fixed FreeBSD test suite Switched build backend to flit_core Removed configuration required previously by flit-scm

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

pyproject.toml

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

agronholm · 2023-03-10T20:06:23Z

Thanks!

SpecLad · 2023-03-10T21:00:09Z

Shouldn't there be a runtime dependency on setuptools? The wheel convert command doesn't work without it:

> .\venv\Scripts\wheel.exe convert
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File ".\venv\Scripts\wheel.exe\__main__.py", line 7, in <module>
  File ".\venv\Lib\site-packages\wheel\cli\__init__.py", line 91, in main
    args.func(args)
  File ".\venv\Lib\site-packages\wheel\cli\__init__.py", line 29, in convert_f
    from .convert import convert
  File ".\venv\Lib\site-packages\wheel\cli\convert.py", line 10, in <module>
    from ..bdist_wheel import bdist_wheel
  File ".\venv\Lib\site-packages\wheel\bdist_wheel.py", line 24, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

agronholm · 2023-03-10T21:01:52Z

I think you should open a separate issue for that. Ideally, nothing else than bdist_wheel should depend on setuptools.

henryiii · 2023-03-10T21:03:48Z

Would this be fixed by #508, I wonder?

agronholm · 2023-03-10T21:04:32Z

No, that still imports setuptools.

SpecLad · 2023-03-10T21:11:23Z

Opened #510.

henryiii · 2023-03-10T21:55:07Z

Ahh, it’s via importing bdist_wheel.

nanonyme · 2023-03-11T06:47:42Z

To be fair, it might have been more sustainable to split this to wheel-core which does not runtime depend on setuptools and wheel that does. They can live in same repo. The core version would provide API for setuptools to build itself.

agronholm · 2023-03-11T12:00:39Z

The long term plan is to move bdist_wheel to setuptools, and refactor the wheel convert command to work independently of setuptools.

henryiii force-pushed the henryiii/feat/flit-backend branch 6 times, most recently from 3772ab3 to 7edc0c3 Compare January 27, 2023 21:20

henryiii force-pushed the henryiii/feat/flit-backend branch from 7edc0c3 to 8773bef Compare January 27, 2023 22:26

henryiii marked this pull request as ready for review January 27, 2023 22:26

henryiii force-pushed the henryiii/feat/flit-backend branch from 8773bef to 8b19da1 Compare January 30, 2023 22:49

agronholm requested changes Jan 31, 2023

View reviewed changes

astrojuanlu mentioned this pull request Feb 27, 2023

[FR] Allow bootstraping setuptools by running dependencies from source pypa/setuptools#2828

Open

1 task

henryiii force-pushed the henryiii/feat/flit-backend branch from 873deec to 7cb3d25 Compare March 5, 2023 18:36

agronholm reviewed Mar 5, 2023

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

agronholm and others added 3 commits March 10, 2023 11:48

Switched build backend to flit

170e2ec

Fixed flit_scm installation Fixed FreeBSD test suite Switched build backend to flit_core Removed configuration required previously by flit-scm

chore: move to using Ruff

99e7042

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

fix: include everything but setup.py in standards based SDist builds

24263ba

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

henryiii and others added 4 commits March 10, 2023 11:49

tests: add SDist test

1073f7c

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

refactor: addressing review feedback

10fc8d2

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

ci: only check SDists on Python 3.11 jobs

6df094b

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

chore: bump ruff version & add show-fixes

f24840d

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

henryiii force-pushed the henryiii/feat/flit-backend branch from 8eee7bc to f24840d Compare March 10, 2023 16:49

agronholm reviewed Mar 10, 2023

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

Update pyproject.toml

5f52063

agronholm reviewed Mar 10, 2023

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

Update pyproject.toml

5dd8410

agronholm reviewed Mar 10, 2023

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

chore: require flit-core 3.8+

ce34fad

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

agronholm approved these changes Mar 10, 2023

View reviewed changes

agronholm merged commit 0de0c1a into pypa:main Mar 10, 2023

henryiii deleted the henryiii/feat/flit-backend branch March 10, 2023 21:03

henryiii mentioned this pull request Mar 13, 2023

docs: add a couple of missing changelog entries #516

Merged

branchvincent mentioned this pull request Apr 6, 2023

python-gdbm@3.11 python-tk@3.11 python@3.11 3.11.3 Homebrew/homebrew-core#127641

Merged

carlocab mentioned this pull request Apr 8, 2023

python@3.10 python-tk@3.10 3.10.11 Homebrew/homebrew-core#127705

Merged

This was referenced Nov 1, 2023

python@3.11: bump python resources Homebrew/homebrew-core#153070

Merged

pypi_formula_mappings: extra_packages for python@X Homebrew/homebrew-core#153077

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: move to flit backend #501

feat: move to flit backend #501

henryiii commented Jan 27, 2023 •

edited

Loading

codecov bot commented Jan 27, 2023 •

edited

Loading

agronholm commented Jan 27, 2023

henryiii commented Jan 27, 2023 •

edited

Loading

agronholm commented Jan 28, 2023

henryiii commented Jan 30, 2023 •

edited

Loading

agronholm commented Jan 30, 2023

henryiii commented Jan 30, 2023

agronholm commented Jan 30, 2023

agronholm commented Jan 30, 2023

henryiii commented Jan 31, 2023

agronholm left a comment

henryiii commented Jan 31, 2023 •

edited

Loading

agronholm commented Feb 1, 2023

vishwin commented Feb 24, 2023

agronholm commented Mar 10, 2023

SpecLad commented Mar 10, 2023

agronholm commented Mar 10, 2023

henryiii commented Mar 10, 2023

agronholm commented Mar 10, 2023

SpecLad commented Mar 10, 2023

henryiii commented Mar 10, 2023 •

edited

Loading

nanonyme commented Mar 11, 2023 •

edited

Loading

agronholm commented Mar 11, 2023

feat: move to flit backend #501

feat: move to flit backend #501

Conversation

henryiii commented Jan 27, 2023 • edited Loading

codecov bot commented Jan 27, 2023 • edited Loading

Codecov Report

agronholm commented Jan 27, 2023

henryiii commented Jan 27, 2023 • edited Loading

agronholm commented Jan 28, 2023

henryiii commented Jan 30, 2023 • edited Loading

agronholm commented Jan 30, 2023

henryiii commented Jan 30, 2023

agronholm commented Jan 30, 2023

agronholm commented Jan 30, 2023

henryiii commented Jan 31, 2023

agronholm left a comment

Choose a reason for hiding this comment

henryiii commented Jan 31, 2023 • edited Loading

agronholm commented Feb 1, 2023

vishwin commented Feb 24, 2023

agronholm commented Mar 10, 2023

SpecLad commented Mar 10, 2023

agronholm commented Mar 10, 2023

henryiii commented Mar 10, 2023

agronholm commented Mar 10, 2023

SpecLad commented Mar 10, 2023

henryiii commented Mar 10, 2023 • edited Loading

nanonyme commented Mar 11, 2023 • edited Loading

agronholm commented Mar 11, 2023

henryiii commented Jan 27, 2023 •

edited

Loading

codecov bot commented Jan 27, 2023 •

edited

Loading

henryiii commented Jan 27, 2023 •

edited

Loading

henryiii commented Jan 30, 2023 •

edited

Loading

henryiii commented Jan 31, 2023 •

edited

Loading

henryiii commented Mar 10, 2023 •

edited

Loading

nanonyme commented Mar 11, 2023 •

edited

Loading