Add support for metadata 2.2

[FFY00]

Currently, twine will fail when a sdist uses metadata 2.2.

Checking dist/trampolim-0.0.1.tar.gz: InvalidDistribution: Invalid distribution metadata. Try upgrading twine if possible.

The error is fairly obscure, and the code too, from my quick look I wasn't able to find what is failing here, which I was expecting to be obvious. If I change the version number to 2.1 it works.

Here's the sdist I was trying to validate/upload.
trampolim-0.0.1.tar.gz

[bhrutledge]

Thanks @FFY00. I agree the error isn't helpful, and that's long been on my list of things to improve. Maybe I'll work on that in the PyCon sprints...

That said, I think the underlying issue is with pkginfo, which doesn't seem to support 2.2. I vaguely recall encountering some other issues with that dependency, so maybe it's worth revisiting.

[FFY00]

Yeah, I'd probably recommend that. The code to parse metadata is fairly simple. Here's the importlib.metadata implementation: https://github.com/python/importlib_metadata/blob/f5bfa3a2873ef9d9e2aa2899def51de63e446849/importlib_metadata/_adapters.py#L8, the only thing special is _repair_headers, the rest is just email.message.Message, you don't need the json stuff.

[bhrutledge]

I opened an issue on pkginfo: https://bugs.launchpad.net/pkginfo/+bug/1928729. That said, I'm out of my depth here, and I don't know what "adding support" involves. Maybe @pfmoore could help, who wrote PEP 643 and updated the Packaging User Guide in pypa/packaging.python.org#785? Or @di, who added Metadata 2.1 support to Twine in #319?

[pfmoore]

Sorry, I don't really know. Metadata version 2.2 introduces the Dynamic field, and (obviously) allows the metadata version to be 2.2, so twine needs to permit those details. See here for the current spec.

Basically, a version of 2.2 needs to be valid, and if the version is 2.2, the Dynamic field must be permitted, and valid (multiple use, must contain names of other metadata fields, Name and Version cannot be listed in Dynamic).

The rest of PEP 643 is more about how tools build sdists, and how build backends consume them, so I don't think it's relevant.

But I don't know what twine does here, so that's just based on the assumption that you just want to validate.

[di]

Here's what adding support for Metadata 2.1 looked like: https://bazaar.launchpad.net/~tseaver/pkginfo/trunk/revision/143.1.3

Looks like @tseaver is still actively maintaining this, so presumably if someone were to propose a revision to add Metadata 2.2 support it would be added and released in a reasonable amount of time.

[bhrutledge]

Thanks for the info. I'd like to get a sense of how urgent this is. Are there any implementations of Metadata 2.2? Running python3 -m build on https://github.com/bhrutledge/example-pkg-bhrutledge yields this PKG-INFO:

Metadata-Version: 2.1
Name: example-pkg-bhrutledge
Version: 0.0.5
Summary: A small example package
...

I'm guessing the 2.1 is coming from setuptools; maybe dist.get_metadata_version? Does @jaraco have any insight here?

[bhrutledge]

For reference, the reported error is coming from:

twine/twine/package.py

Lines 107 to 110 in eff3a45

	if not (meta.name and meta.version):
	raise exceptions.InvalidDistribution(
	"Invalid distribution metadata. Try upgrading twine if possible."
	)

where meta is presumably a pkginfo.SDist. This was added in #477.

[FFY00]

Yes, the metadata is generated by the backend, the current setuptools version is implementing 2.1. My backend is implementing 2.2, hence the error.

The current sdist spec mandates 2.2.
https://packaging.python.org/specifications/source-distribution-format/

I expect new backends to be implementing that and existing backends to update to it soon.

[jaraco]

I'm guessing the 2.1 is coming from setuptools; maybe dist.get_metadata_version? Does @jaraco have any insight here?

No insight really. Yes, Setuptools recently updated to unconditionally generate metadata 2.1 (versus previously falling back to the lowest version supported by the metadata). The 2.2 must be coming from other backends (as confirmed by FFY00).

I'd surely like to get support for metadata 2.2 into importlib.metadata, if that helps this issue.

[FFY00]

Any update on this? It is very annoying 🙃

[tseaver]

FWIW, I released pkginfo-1.8.0 with support for parsing the Dynamic header from Metadata-Version: 2.2 distributions.

[Freed-Wu]

I still met this bug:

$ twine upload dist/autotools_language_server-0.0.9-cp310-cp310-manylinux_2_37_x86_64.whl
Uploading autotools_language_server-0.0.9-cp310-cp310-manylinux_2_37_x86_64.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 178.5/178.5 kB • 00:01 • 45.7 MB/s
WARNING  Error during upload. Retry with the --verbose option for more details.
ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
         '2.2' is an invalid value for Metadata-Version. Error: Use a known metadata version. See https://packaging.python.org/specifications/core-metadata for more
         information.

scikit-build-core will generate Metadata-Version: 2.2 when dynamic = ["dependencies"] or dynamic = ["optional-dependencies"].

[jaraco]

Freed-Wu, your error is different, because the error message is coming from the server and not twine. You'll need to take it up with the owner of that index server, who probably has an out of date dependency.

[di]

The upstream issue to track for PyPI to support this is pypi/warehouse#9660.