Confusion about the effects of setting packages for sdist

[doronz88]

setuptools version

67.7.2

Python version

Python 3.11.3

OS

macOS 13.3.1

Additional environment information

No response

Description

My pyproject.toml explicitly specifies a packgaes list for setuptools to include.
However, tests/test_utils.py is still included in the generated .tar.gz archive.
Also, adding the exclude key didn't help.

Expected behavior

Not including the tests/test_utils.py

How to Reproduce

git clone git@github.com:doronz88/pymobiledevice3.git -b refactor/python_package
cd pymobiledevice3
python3 -m pip install -U build
python3 -m build

Open and view the dist/*.tar.gz file.

Output

The tests/test_utils.py is included

[abravalheri]

Hi @doronz88, the packages configuration will determine which files are copied in the end users machine when they install a package. Consequently, this configuration influences contents of the generated wheel (.whl) file.

However .tar.gz files are a different beast all together... They are what we call source distribution (sdist), which is an intermediate artefact, meant to be platform independent, and work in a different level of optimization than wheel files.

Most of the times, it includes more files than the ones you see in the wheel. When you install a .tar.gz file with pip, pip will first "compile it" into a .whl file (which will not contain the tests/test_utils.py) and then install it. Before everyone started using git, it was a common practice to include all the files that are used to develop your package in the sdist as they serve as a "git checkout on steroids"¹.

Sdists are still used when installing your package in a platform for which there are no wheels available and in some OS redistributions of Python packages (e.g. apt, dnf, pacman).

There is some level of agreement in the community that believe it is a good practice to include test files into the sdists, that is why setuptools include them by default.

If you don't want to include files in the sdist, that is still possible, but the configuration that you need to change is the one in MANIFEST.in: https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html (not packages).

Footnotes

1. They are more than a VCS checkout, as they include extra metadata generated by setuptools. ↩

[doronz88]

@abravalheri thanks for the amazingly fast and detailed response!

However, both when I remove the packages key and leave it as-is, I still get an unexpected result. The tests module remains, but the it doesn't include the inner modules: tests.cli & tests.services

[abravalheri]

Hi @doronz88, the default set of files automatically included in MANIFEST.in come from the original distutils implementation: https://github.com/pypa/distutils/blob/4435cec31b8eb5712aa8bf993bea3f07051c24d8/distutils/command/sdist.py#L233-L246, and it includes tests/test*.py and test/test*.py.

To do something different than the default, it is necessary to configure MANIFEST.in.

If you want all the files in git to be part of the sdist, you can also use setuptools-scm instead of configuring MANIFEST.in. (That is usually what I do in my own projects and I recommend people doing).

[doronz88]

Thanks! I completely understand the rational and behavior now 😃