Users expect [options.packages.find] exclude to be recursive, but it is not (adding .* to the pattern is needed)

[KOLANICH]

setuptools version

57.0.1.dev3+ga5131f0b.post.20210528a

Python version

cpython 3.9

OS

Ubuntu 21.04

Additional environment information

No response

Description

[options.packages.find]
exclude =
    arpeggio.tests

doesn't work. In fact it doesn't work in a lot of other projects. But in other projects there usually no more than 2 levels in the excluded dir, so the workaround tests and tests.* works.

Expected behavior

Should exclude the whole hierarchy if a dir is mentioned.

How to Reproduce

https://github.com/textX/Arpeggio/tree/f24ceb0e79f6c9216dc035de123a388d113a332e

Output

....
adding 'arpeggio/tests/__init__.py'
adding 'arpeggio/tests/test_decorator_combine.py'
adding 'arpeggio/tests/test_default_semantic_action.py'
adding 'arpeggio/tests/test_eolterm.py'
adding 'arpeggio/tests/test_error_reporting.py'
adding 'arpeggio/tests/test_examples.py'
....

Code of Conduct

• I agree to follow the PSF Code of Conduct

[abravalheri]

Hi @KOLANICH, thank you very much for bringing this topic for discussion.

The current implementation specifically suggests that this behaviour is intentional, although I don't know what is the rationale behind it. So I will re-label this from bug to enhancement.

(I can conjecture that if the exclusion is automatically recursive there is no way of "re-including" a nested package, so in some sense the current implementation is more general/flexible than the proposal - not sure how relevant this rationale is).

I don't know what would be the impacts of changing this behaviour (it would definitely not be backwards compatible), so I will leave the assessment of the proposal for the other maintainers.

[jaraco]

I'm having a very similar issue: When running pytproject-build, the resulting sdist has the tests folder included even though it is explicitly excluded using the exclude=tests,test* config. Interestingly, the .whl file does not include the tests.

This behavior is by design. The package inference happens during a build phase, but sdist happens earlier and is more abstract than the build phase. See this comment for a detailed explanation.

nowadays and I am still facing this same issue.

Same as the prior comment. The wheel looks fine and is excluding the packages from the wheel. You should expect source files (even tests) to be included in the sdist. To affect the sdist, you'll want to customize the MANIFEST.in.

The core of the evil is *.egg-info - it is not (properly?) updated on fresh build or installation.
So if there is *.egg-info from the previous build/installation without exclude-option, it makes you hair off.
Killing *.egg-info folder before build/install solves the problem.

This behavior is inconsistent with the reported issue. The reported issue is that users shouldn't have to specify both bug_demo.subpack and bug_demo.subpack.* to exclude it. The issue you encountered was with a stale egg-info, which seems to be a separate issue. #1347 seems to be related, though not precisely. I searched around and didn't find anything better, though.

I'm marking all of these comments as off-topic, as they're distracting from the primary report.

[jaraco]

I encountered this issue recently when working with distutils. I was trying to figure out why the distutils package was not being installed. It turns out it was because I'd included a find.exclude=['dist*'] directive in an attempt to exclude the dist directory, but that also excluded the distutils directory.

As it turns out, that directive wasn't necessary, so I removed it, but while troubleshooting, I noticed I was also excluding docs* and tests*, which are necessary. Unfortunately, these exclusions aren't having the intended effect (exclude the docs/ and tests/ directories and everything beneath them as packages. Instead, what's necessary is to exclude=['docs', 'docs.*', 'tests', 'tests.*'], which is rather clumsy. The user should be able to exclude a directory in one declaration and not two.

Like abravalheri, I'm trying to imagine in what scenario one would want to exclude the top-level package, but include the child package. That seems unlikely. My instinct is just to drop all nested packages and see if it causes any disruption and if so, roll it back and deprecate the behavior to identify affected use-cases.