setup.py + MANIFEST.in pulling from files in parent directory

[jamesbraza]

I have a feature request: supporting pulling from files in the parent directory of MANIFEST.in and setup.py.

---

Motivation

I have a Python 3.8 package whose unit tests are also package-able. Here is the file structure inside the repo:

.
├── README.rst
├── src
│   ├── my_pkg
│   ├── MANIFEST.in
│   ├── pyproject.toml
│   ├── requirements.txt
│   └── setup.py
├── tests
│   ├── my_pkg_tests
│   ├── MANIFEST.in
│   ├── pyproject.toml
│   ├── requirements.txt
│   └── setup.py
└── venv

The desired behavior I want in both package's setup.py + MANIFEST.in to be able to add something like this:

include ../README.rst

If there are any workaround I am not aware of, an explanation would be appreciated.

---

Research

Currently, I don't think it's possible to pull from files in the parent directory. This is based on:

1. I was getting errors saying README.rst couldn't be found
2. Stack Overflow question 1: Using Parent Directories in Manifest.in
3. Stack Overflow question 2: python setuptools include files from parent directory

[jaraco]

What you're trying to do, having two packages for one conceptual project, isn't something that's recommended or supported. Most projects have one package and either includes the tests as part of that package or excludes the tests except for source checkouts/distributions. I would recommend following the normative patterns. If you wish to do something very different, I'll have to ask that you delve into the source to find the answers to your questions.

After you've spent some time with the code and if you think there's a way to support the layout the way you would expect, consider filing an issue describing your proposed design and what value you think that brings to the project. If it's a compelling story, we can proceed with soliciting an implementation.

[excitoon]

@jaraco well, how to support this and all other possible cases: let setup take cwd as a parameter. Worst case we would just build file structure setuptools is okay with, by hand. It won't help pip but at least we would have a command which will prepare a package for pip.

[excitoon]

Generally speaking, you shall support lower level features than higher level "file structures", the latter is a broken approach, whereas the former will allow you to build universal and stable tool.
As an example, when you set package_dir to {'A.B.C.D': 'G/H/F'}, it shall have straightforward meaning, that files inside G/H/F must appear in A.B.C.D package. Anything else: 1) adds unneeded complexity; 2) allows to write code which works by a chance; 3) doesn't allow to check the input against simple rules. Right now, if G/H/F contains .., without such checks, setuptools corrupt user filesystem. So, ideally you set orthogonal lightweight rules and implement each of them in reliable enough way, this is how you get a good tool.

[jaraco]

@jaraco well, how to support this and all other possible cases: let setup take cwd as a parameter. Worst case we would just build file structure setuptools is okay with, by hand. It won't help pip but at least we would have a command which will prepare a package for pip.

If all that's needed is to change the working directory prior to executing setup(), could you not invoke that behavior yourself? e.g.

import os
import pathlib
import setuptools

os.chdir(pathlib.Path(__file__).parents[1])
setuptools.setup(...)

Setuptools is working to move users away from bespoke behavior in the setup script and reliance on the current working directory, so I'm not sure how much I'd recommend that approach, but from your comment, it seems to suggest that might work.

Generally speaking, you shall support lower level features than higher level "file structures", the latter is a broken approach, whereas the former will allow you to build universal and stable tool.

In principle, I agree. Having simple rules that are composed to provide higher level behavior tends to be more reliable and extensible.

As an example, when you set package_dir to {'A.B.C.D': 'G/H/F'}, it shall have straightforward meaning, that files inside G/H/F must appear in A.B.C.D package. Anything else: 1) adds unneeded complexity; 2) allows to write code which works by a chance; 3) doesn't allow to check the input against simple rules. Right now, if G/H/F contains .., without such checks, setuptools corrupt user filesystem. So, ideally you set orthogonal lightweight rules and implement each of them in reliable enough way, this is how you get a good tool.

I suspect the simple approach you describe exists or did exist at one point in distutils or setuptools, but in an attempt to satisfy a wide variety of use-cases, has developed complexity over time.

I'd be open to contributions to simplify the design or implementation, but any change is going to require some research and development. I've got my plate completely full just trying to reconcile Setuptools with distutils and deprecate the installer functionality.

If you wish to articulate an idea or issue, feel free to start the discussion in a new issue. Also, feel encouraged to study the source and devise a different design that achieves your goals.