tag_build breaks prepare_metadata_for_build_wheel #2529

uranusjr · 2021-01-13T09:40:16Z

Previously reported in pypa/pip#9446.

This is similar to #1462, but for PEP 517 hooks, so I’m guessing a similar treatment is needed?

[egg_info]
tag_build = dev

[metadata]
name = pkg
version = 0.1

egg_info works as expected:

$ py -c 'from setuptools import *; setup()' egg_info >/dev/null
$ rg '^Version:' < pkg.egg-info/PKG-INFO 
Version: 0.1.dev0

But not the PEP 517 hook:

$ py -c 'import os; from setuptools.build_meta import *; prepare_metadata_for_build_wheel(os.getcwd())' >/dev/null
$ rg '^Version:' < pkg.dist-info/METADATA 
Version: 0.1.dev0dev

The text was updated successfully, but these errors were encountered:

jaraco · 2021-01-17T03:11:32Z

Due to #2536, the fix had to be rolled back and will need to be revisited in light of this new information.

zzzeek · 2021-01-17T14:50:35Z

im so impressed you understand how setuptools works thanks for the efforts! i still remember ten years ago when nobody could change anything about it.

zzzeek · 2021-01-27T22:51:46Z

Just checking that I also get this problem if I install straight from pip but also try to add an extras_require like this:

.venv/bin/pip install git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=sqlalchemy[postgresql] git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=sqlalchemy
Collecting sqlalchemy[postgresql]

# ...

ERROR: Cannot install sqlalchemy 1.4.0b2.dev0dev (from git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=sqlalchemy) and sqlalchemy[postgresql]==1.4.0b2.dev0dev because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested sqlalchemy 1.4.0b2.dev0dev (from git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=sqlalchemy)
    sqlalchemy[postgresql] 1.4.0b2.dev0dev depends on sqlalchemy 1.4.0b2.dev0dev (from git+https://github.com/sqlalchemy/sqlalchemy.git@master#egg=sqlalchemy[postgresql])

same thing right? I guess I have to start considering what tag_build=dev is getting me these days as this "dev" tag is now automatically appended in most cases installing from SCM ?

jaraco · 2021-01-27T23:59:01Z

I'm unsure why you would specify sqlalchemy twice to pip install. Why not do something like this:

~ $ pip-run --use-pep517 'sqlalchemy[postgresql]@git+https://github.com/sqlalchemy/sqlalchemy@master' -- -c pass
Collecting sqlalchemy[postgresql]@ git+https://github.com/sqlalchemy/sqlalchemy@master
  Cloning https://github.com/sqlalchemy/sqlalchemy (to revision master) to /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-e3mtb3dg/sqlalchemy_946f25bccb604571a8aa49e2d76bac6e
  Running command git clone -q https://github.com/sqlalchemy/sqlalchemy /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-e3mtb3dg/sqlalchemy_946f25bccb604571a8aa49e2d76bac6e
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting greenlet!=0.4.17
  Using cached greenlet-1.0.0-cp39-cp39-macosx_10_14_x86_64.whl (86 kB)
Collecting psycopg2>=2.7
  Using cached psycopg2-2.8.6-cp39-cp39-macosx_10_9_x86_64.whl
Building wheels for collected packages: sqlalchemy
  Building wheel for sqlalchemy (PEP 517) ... done
  Created wheel for sqlalchemy: filename=SQLAlchemy-1.4.0b2.dev0-cp39-cp39-macosx_10_9_x86_64.whl size=1414363 sha256=fdb1f57b673d1d270cd7004e566849fe21d2437003a10c71abc0feaae238b5b4
  Stored in directory: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-ephem-wheel-cache-pnjsfy6s/wheels/a5/aa/f0/c81b331ed799c5bb42262fbe93241571ac8c2110330a3d45f1
Successfully built sqlalchemy
Installing collected packages: greenlet, sqlalchemy, psycopg2
Successfully installed greenlet-1.0.0 psycopg2-2.8.6 sqlalchemy-1.4.0b2.dev0

This command also seems to work fine:

pip-run --use-pep517 'sqlalchemy[postgresql]@git+https://github.com/sqlalchemy/sqlalchemy@master' sqlalchemy@git+https://github.com/sqlalchemy/sqlalchemy@master -- -c pass

Note that I use pip-run in place of pip install to save the setup/teardown of a venv.

Perhaps using the more modern specifiers avoids whatever codepaths cause the duplicated tags. Does that help?

zzzeek · 2021-01-28T00:19:16Z

it's for a tox file. I dont really know how to do what I want:

[tox]

SQLA_REPO = {env:SQLA_REPO:git+https://github.com/sqlalchemy/sqlalchemy.git}

[testenv]

deps=pytest>4.6
     pytest-xdist
     mock
     sqla13: {[tox]SQLA_REPO}@rel_1_3#egg=sqlalchemy
     sqlamaster: {[tox]SQLA_REPO}@master#egg=sqlalchemy
     sqlalchemy: sqlalchemy>=1.3.0

     cov: pytest-cov
     mako
     python-editor>=0.3
     python-dateutil


     postgresql: sqlalchemy[postgresql]
     mysql: sqlalchemy[mysql]
     mysql: sqlalchemy[pymysql]
     oracle: sqlalchemy[oracle]
     mssql: sqlalchemy[mssql]

jaraco · 2021-01-30T00:15:54Z

How about this:

[tox]

SQLA_REPO = {env:SQLA_REPO:git+https://github.com/sqlalchemy/sqlalchemy.git}

[testenv]
skip_install = True
deps =
     pytest>4.6
     pytest-xdist
     mock
     sqla13: sqlalchemy@{[tox]SQLA_REPO}@rel_1_3
     sqlamaster: sqlalchemy@{[tox]SQLA_REPO}@master
     sqlalchemy: sqlalchemy>=1.3.0

     cov: pytest-cov
     mako
     python-editor>=0.3
     python-dateutil


     postgresql: sqlalchemy[postgresql]
     mysql: sqlalchemy[mysql]
     mysql: sqlalchemy[pymysql]
     oracle: sqlalchemy[oracle]
     mssql: sqlalchemy[mssql]

I was able to run tox -e sqlalchemy,sqla13,sqlamaster,postgresql with that tox file.

zzzeek · 2021-01-30T05:35:42Z

that looks great! I will try it in a bit but what's this syntax?

sqlalchemy@{[tox]SQLA_REPO}@rel_1_3

@ sign on both sides? I think I saw maybe a random stackoverflow post that referred to that syntax. how does that relate to #egg=<packagename> ?

zzzeek · 2021-01-30T05:39:33Z

it works, thanks a ton! [edit: sorry, it's not working]

uranusjr · 2021-01-30T05:51:36Z

You can think of the name @ URL format as the replacement of URL#egg=name. Easier to parse, and does not depend on the URL part to get the package name (which is an issue when comparing URL identity). The format is specified in PEP 508.

zzzeek · 2021-01-30T14:30:03Z

is "egg=name" deprecated / unnecessary ? (which would be awesome?)

uranusjr · 2021-01-30T14:53:04Z

I believe it’s not yet formally deprecated (there are still a couple of edge cases), but yeah, basically.

zzzeek · 2021-01-30T15:28:45Z

I was wrong. it doesn't work, it actually ignores the other packages. we run the tests such as:

tox -e py3-sqlamaster-sqlite-postgresql-mysql

it should be installing all the extras for postgresql and mysql. it's actually just ignoring them:

$ tox -e py3-sqlamaster-sqlite-postgresql --notest
GLOB sdist-make: /home/classic/dev/alembic/setup.py
py3-sqlamaster-sqlite-postgresql create: /home/classic/dev/alembic/.tox/py3-sqlamaster-sqlite-postgresql
py3-sqlamaster-sqlite-postgresql installdeps: pytest>4.6, pytest-xdist, mock, sqlalchemy@git+https://github.com/sqlalchemy/sqlalchemy.git@master, sqlalchemy[postgresql], mako, python-editor>=0.3, python-dateutil
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py3-sqlamaster-sqlite-postgresql inst: /home/classic/dev/alembic/.tox/.tmp/package/2/alembic-1.5.4.dev0.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py3-sqlamaster-sqlite-postgresql installed: alembic @ file:///home/classic/dev/alembic/.tox/.tmp/package/2/alembic-1.5.4.dev0.zip,apipkg==1.5,attrs==20.3.0,execnet==1.8.0,greenlet==1.0.0,iniconfig==1.1.1,Mako==1.1.4,MarkupSafe==1.1.1,mock==4.0.3,packaging==20.9,pluggy==0.13.1,py==1.10.0,pyparsing==2.4.7,pytest==6.2.2,pytest-forked==1.3.0,pytest-xdist==2.2.0,python-dateutil==2.8.1,python-editor==1.0.4,six==1.15.0,SQLAlchemy @ git+https://github.com/sqlalchemy/sqlalchemy.git@532026f97f402d6673cd9746f1a7daee99327a68,toml==0.10.2
_________________________________________________________________ summary _________________________________________________________________
  py3-sqlamaster-sqlite-postgresql: skipped tests
  congratulations :)
[classic@photon3 alembic:use_new_pip_format]$ .tox/py3-sqlamaster-sqlite-postgresql/bin/python
Python 3.8.3 (default, May 23 2020, 16:34:37) 
[GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'psycopg2'

jaraco · 2021-01-30T20:12:37Z

I think what you've found is an issue with pip. From that tox.ini and with tox -e sqlamaster-postgresql, tox constructs this pip install command:

python -m pip install 'pytest>4.6' pytest-xdist mock sqlalchemy@git+https://github.com/sqlalchemy/sqlalchemy.git@master mako 'python-editor>=0.3' python-dateutil 'sqlalchemy[postgresql]'

Running that command in clean environment goes into deep recursion on projects due to an issue in the new resolver (pypa/pip#8785). I gave up running the command after 10 minutes.

Disabling the new resolver, pip fails to install the postgresql extra:

draft $ env PIP_USE_DEPRECATED=legacy-resolver pip-run 'pytest>4.6' pytest-xdist mock sqlalchemy@git+https://github.com/sqlalchemy/sqlalc
hemy.git@master mako 'python-editor>=0.3' python-dateutil 'sqlalchemy[postgresql]'
Collecting pytest>4.6
  Using cached pytest-6.2.2-py3-none-any.whl (280 kB)
Collecting pytest-xdist
  Using cached pytest_xdist-2.2.0-py3-none-any.whl (36 kB)
Collecting mock
  Using cached mock-4.0.3-py3-none-any.whl (28 kB)
Collecting sqlalchemy@ git+https://github.com/sqlalchemy/sqlalchemy.git@master
  Cloning https://github.com/sqlalchemy/sqlalchemy.git (to revision master) to /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-lmxkw_v2/sqlalchemy
  Running command git clone -q https://github.com/sqlalchemy/sqlalchemy.git /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-lmxkw_v2/sqlalchemy
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Processing /Users/jaraco/Library/Caches/pip/wheels/cd/56/62/f7e8f1528d26bbdd5d1ea7693b945b4cbef4bbaa0db0bea2b0/Mako-1.1.4-py2.py3-none-any.whl
...
Collecting apipkg>=1.4
  Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Building wheels for collected packages: sqlalchemy
  Building wheel for sqlalchemy (PEP 517) ... done
  Created wheel for sqlalchemy: filename=SQLAlchemy-1.4.0b2.dev0-cp39-cp39-macosx_10_9_x86_64.whl size=1415486 sha256=657f025c579781e86261d89aa200119d575f4b359dfaa316f417f5cd6352ff99
  Stored in directory: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-ephem-wheel-cache-e_ib4n0j/wheels/74/f8/a3/5cdbf2d34871d0a2545a526befcf91053137b290f7bcda75c4
Successfully built sqlalchemy
Installing collected packages: pluggy, pyparsing, packaging, attrs, toml, iniconfig, py, pytest, apipkg, execnet, pytest-forked, pytest-xdist, mock, greenlet, sqlalchemy, MarkupSafe, mako, python-editor, six, python-dateutil
Successfully installed MarkupSafe-1.1.1 apipkg-1.5 attrs-20.3.0 execnet-1.8.0 greenlet-1.0.0 iniconfig-1.1.1 mako-1.1.4 mock-4.0.3 packaging-20.9 pluggy-0.13.1 py-1.10.0 pyparsing-2.4.7 pytest-6.2.2 pytest-forked-1.3.0 pytest-xdist-2.2.0 python-dateutil-2.8.1 python-editor-1.0.4 six-1.15.0 sqlalchemy-1.4.0b2.dev0 toml-0.10.2

Or more simply:

$ env PIP_USE_DEPRECATED=legacy-resolver pip-run sqlalchemy 'sqlalchemy[postgresql]' -- -c pass
Collecting sqlalchemy
  Using cached SQLAlchemy-1.3.22-cp39-cp39-macosx_10_14_x86_64.whl (1.2 MB)
Installing collected packages: sqlalchemy
Successfully installed sqlalchemy-1.3.22

My guess is there's already a bug for this unmet use-case.

jaraco · 2021-01-30T20:15:32Z

Looks like pypa/pip#4957 captures the weakness with the legacy resolver and the fix is "use 2020 resolver".

In the simple case, the 2020 resolver works for me:

$ pip-run sqlalchemy 'sqlalchemy[postgresql]' -- -c pass
Collecting sqlalchemy
  Using cached SQLAlchemy-1.3.22-cp39-cp39-macosx_10_14_x86_64.whl (1.2 MB)
Collecting psycopg2
  Using cached psycopg2-2.8.6-cp39-cp39-macosx_10_9_x86_64.whl
Installing collected packages: sqlalchemy, psycopg2
Successfully installed psycopg2-2.8.6 sqlalchemy-1.3.22

jaraco · 2021-01-30T20:31:28Z

Attempting to use sqlalchemy@master, however, pip is unable to reconcile the pre-release from master with any public release. Apparently it thinks that sqlalchemy[postgresql] depends on sqlalchemy of the same version, e.g.

sqlalchemy[postgresql] 1.3.22 depends on sqlalchemy 1.3.22

So it's never satisfied.

zzzeek · 2021-01-30T22:18:49Z

yes, that's the thing where it seems to go down a whole list of SQLAlchemy versions and doesn't find anything, wasn't sure if I should have reported that too as I wasnt sure which circumstances caused it, but if I add --use-feature-resolver-2020 that allows the non-SCM case to work, but the SCM case looks like:

Collecting sqlalchemy[postgresql]
  Using cached SQLAlchemy-1.3.22-cp38-cp38-manylinux2010_x86_64.whl (1.3 MB)
Collecting sqlalchemy[postgresql]
  Using cached SQLAlchemy-1.3.21-cp38-cp38-manylinux2010_x86_64.whl (1.3 MB)
Collecting sqlalchemy[postgresql]
  Using cached SQLAlchemy-1.3.20-cp38-cp38-manylinux2010_x86_64.whl (1.3 MB)
Collecting sqlalchemy[postgresql]
  Using cached SQLAlchemy-1.3.19-cp38-cp38-manylinux2010_x86_64.whl (1.3 MB)
Collecting sqlalchemy[postgresql]

... things go for a long time

WARNING: sqlalchemy 0.9.1 does not provide the extra 'postgresql'
Collecting sqlalchemy[postgresql]
  Using cached SQLAlchemy-0.9.0.tar.gz (4.3 MB)
WARNING: sqlalchemy 0.9.0 does not provide the extra 'postgresql'
Collecting sqlalchemy[postgresql]
  Using cached SQLAlchemy-0.8.7.tar.gz (3.5 MB)
ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device

thanks for the help so far!

di mentioned this issue Jan 15, 2021

Fix/2529 #2533

Merged

2 tasks

jaraco closed this as completed in #2533 Jan 16, 2021

jaraco reopened this Jan 17, 2021

uranusjr mentioned this issue Jan 30, 2021

build meta wheel does not respect metadata_directory per PEP-517 #1825

Closed

uranusjr mentioned this issue Apr 29, 2021

New resolver mangles development versions pypa/pip#9446

Closed

abravalheri mentioned this issue Jan 5, 2022

[Docs] Clarify setup.cfg-only Package Release Tagging #2819

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tag_build breaks prepare_metadata_for_build_wheel #2529

tag_build breaks prepare_metadata_for_build_wheel #2529

uranusjr commented Jan 13, 2021 •

edited

Loading

jaraco commented Jan 17, 2021

zzzeek commented Jan 17, 2021

zzzeek commented Jan 27, 2021

jaraco commented Jan 27, 2021

zzzeek commented Jan 28, 2021

jaraco commented Jan 30, 2021

zzzeek commented Jan 30, 2021 •

edited

Loading

zzzeek commented Jan 30, 2021 •

edited

Loading

uranusjr commented Jan 30, 2021 •

edited

Loading

zzzeek commented Jan 30, 2021

uranusjr commented Jan 30, 2021

zzzeek commented Jan 30, 2021 •

edited

Loading

jaraco commented Jan 30, 2021

jaraco commented Jan 30, 2021

jaraco commented Jan 30, 2021

zzzeek commented Jan 30, 2021

tag_build breaks prepare_metadata_for_build_wheel #2529

tag_build breaks prepare_metadata_for_build_wheel #2529

Comments

uranusjr commented Jan 13, 2021 • edited Loading

jaraco commented Jan 17, 2021

zzzeek commented Jan 17, 2021

zzzeek commented Jan 27, 2021

jaraco commented Jan 27, 2021

zzzeek commented Jan 28, 2021

jaraco commented Jan 30, 2021

zzzeek commented Jan 30, 2021 • edited Loading

zzzeek commented Jan 30, 2021 • edited Loading

uranusjr commented Jan 30, 2021 • edited Loading

zzzeek commented Jan 30, 2021

uranusjr commented Jan 30, 2021

zzzeek commented Jan 30, 2021 • edited Loading

jaraco commented Jan 30, 2021

jaraco commented Jan 30, 2021

jaraco commented Jan 30, 2021

zzzeek commented Jan 30, 2021

uranusjr commented Jan 13, 2021 •

edited

Loading

zzzeek commented Jan 30, 2021 •

edited

Loading

zzzeek commented Jan 30, 2021 •

edited

Loading

uranusjr commented Jan 30, 2021 •

edited

Loading

zzzeek commented Jan 30, 2021 •

edited

Loading