Changes to the User Guide's Data Files page

[codeandfire]

Summary of Changes

• All code snippets were given for setup.py. Have added corresponding snippets for setup.cfg and pyproject.toml.
• To avoid incentivizing multiple top-level packages, have modified all the package trees and code snippets to include only a single package mypkg. Have added a separate example to illustrate the functionality of using the empty string "" / the asterisk * for capturing data files from multiple packages. Have also modified the setup.py code snippets and removed the find_packages("src") call since there is only a single package in each case (except one); have opted to explicitly name the package instead.
• Have added a package tree example for the first package_data snippet. Have also added a package tree / code snippet example to show how package_data patterns should include subdirectories, separating it from the example showing the empty string "" / asterisk * functionality.
• Tried to have consistent naming for all directories and data files used in the package trees and code snippets. All directories have been named mypkg and data files have been named data1.txt, data2.rst etc.
• Have reformatted package tree examples. Reformatting has been done by replacing the only-indentation based directory structure diagram with a line-based tree layout; I think this looks neater.
• Have added .. note:: blocks for paragraphs that would be more appropriately phrased as a Note. Other minor changes to text content have been made.

Pull Request Checklist

• News fragment added in changelog.d/.
  (See documentation for details)

[abravalheri]

Thank you very much @codeandfire, the improvements are very appreciated 👏 🎉

I would like to suggest some changes in the light of the recent discussions #3323:

I think we should encourage people to consider mypkg/data folder in one of your examples as the mypkg.data namespace package instead of a "data directory" (Python considers that anyway)... I might be able to submit a PR with suggestions for you later this week.

[codeandfire]

@abravalheri Sure, sounds interesting, will look forward to this PR since I'm considering implementing a similar layout in one of my own projects!

May I ask, though, what ramifications this will have for the advice given here? Especially since the linked importlib docs page says that namespace packages will be ignored.

EDIT: I think this has been tried in the discussion #3323 that you have linked. In one of the comments in the thread, one of the users has tried using importlib.resources.

[abravalheri]

May I ask, though, what ramifications this will have for the advice given here? Especially since the linked importlib docs page says that namespace packages will be ignored.

I did a quick test:

rm -rf /tmp/myproj
mkdir /tmp/myproj
cd /tmp/myproj
cat <<EOS > pyproject.toml
[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools", "setuptools_scm[toml]"]

[project]
name = "myproj"
dynamic = ["version"]

[tool.setuptools_scm]
EOS

mkdir -p src/myproj/data
touch src/myproj/__init__.py
echo "hello world" > src/myproj/data/data1.txt
echo 42 > src/myproj/data/data2.json
tree
# .
# ├── pyproject.toml
# └── src
#     └── myproj
#         ├── __init__.py
#         └── data
#             ├── data1.txt
#             └── data2.json

git init .
git add .
git commit -m "Initial commit"

virtualenv .venv
.venv/bin/python -m pip install -U pip importlib-resources
.venv/bin/python -m pip install .

ls .venv/lib/python*/site-packages/myproj/data  # making sure data files are installed

.venv/bin/python
>>> import importlib_resources, myproj.data
>>> importlib_resources.files(myproj.data).joinpath("data1.txt").read_text()
'hello world\n'

Here I used pyproject.toml configuration which is equivalent to include_package_data=True and packages = find_namespace:, etc... I also used setuptools-scm because I cannot be bothered to create a MANIFEST.in file...

importlib_resources seems to be fine with the approach...

I also tried:

>>> import importlib_resources, myproj
>>> importlib_resources.files(myproj).joinpath("data/data1.txt").read_text()
'hello world\n'

Maybe importlib_resources docs are outdated? Or maybe it does not support a "pure namespace-only" package?

[abravalheri]

I also did a second example:

rm -rf /tmp/myproj
mkdir /tmp/myproj
cd /tmp/myproj
cat <<EOS > pyproject.toml
[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools", "setuptools_scm[toml]"]

[project]
name = "myproj"
dynamic = ["version"]

[tool.setuptools_scm]
EOS

mkdir -p src/myproj/data
echo "hello world" > src/myproj/data/data1.txt
echo 42 > src/myproj/data/data2.json
tree
# .
# ├── pyproject.toml
# └── src
#     └── myproj
#         └── data
#             ├── data1.txt
#             └── data2.json

git init .
git add .
git commit -m "Initial commit"

virtualenv .venv
.venv/bin/python -m pip install -U pip importlib-resources
.venv/bin/python -m pip install .

>>> import importlib_resources, myproj.data
>>> importlib_resources.files(myproj).joinpath("data/data1.txt").read_text()
'hello world\n'

Everything still seems to work fine even with a namespace only package.

[abravalheri]

The support for namespace packages seems to have been added to recent versions of importlib-resources (October last year): python/importlib_resources#196.

[codeandfire]

You're right, it's the docs that are outdated. I have raised a PR with them python/importlib_resources#251.

[codeandfire]

I think we should encourage people to consider mypkg/data folder in one of your examples as the mypkg.data namespace package instead of a "data directory"

@abravalheri from this, should I take it to mean that we are encouraging a scheme in which we set include_package_data=True and only specify files to exclude using the exclude_package_data argument, over a scheme in which we explicitly specify all the data files using the package_data argument?

[abravalheri]

My original idea is not exactly about using include_package_data instead of package_data.
Both are fine.

It is more about not writing:

package_data={"mypkg": ["*.txt", "data/*.rst"]}

but instead:

package_data={
    "mypkg": ["*.txt"],
    "mypkg.data": ["*.rst"],
}

[codeandfire]

I see. I'll go ahead and amend my PR then, to include these changes, and maybe then you can review them and let me know of any further changes.

[codeandfire]

@abravalheri sorry for the delay, but have amended my PR to incorporate the changes suggested by you along with a bunch of other changes. Let me know your feedback. Thanks!

[abravalheri]

Thank you very much @codeandfire, those are really very good improvements.
(Sorry for the delay in providing a review/suggestions)

I added some small comments, but most of them refer mainly to existing content that was already present in the previous version. I am very happy to accept this PR as it is and do small adjustments later if necessary.

[abravalheri]

Thank you very much @codeandfire, I think I will go ahead and merge the PR, unless you would like to work in other changes.

[codeandfire]

@abravalheri no other changes from my side, you can go ahead and merge. Thanks again for your review!