Build sdist with setuptools_scm and subfolders
#322
Comments
|
You can modify the SDist command to collect the missing parts of the repo when it is building the SDist. Then you'll need to make sure the remaining accesses to the files work with the files in either location, since you can't be sure that sdist is always called first. Maybe there's a better way to do that, but that's what I've done in the past. A bit more is happening in pybind11's setup.py (including writing a new setup.py for each of two output packages), but that's part of it (in pybind11's case, it's not because it's in a subfolder, but IIRC the process is the same). |
|
The reason I don't have a good example with the project fully in a subfolder is because I usually put the key bits of the project folder in the main directory, and then everything else related to the bindings in a "python" subfolder or similar. This is much easier, but does mean that three files (pyproject.toml, setup.cfg, and setup.py) have to sit in the top level of your project, which isn't always possible. Maybe someone else has a good example, I know it's a common design pattern. |
|
|
|
@henryiii thanks a lot for the prompt reply!
I'm used to keep the files in the top level of the repo when possible. This is not the case, we're trying to store them in a subfolder since PyPI support is community maintained. Funny enough, building wheels with the current structure works flawlessly, only the sdist are not fully working. I checked the code of pypa/build and pypa/setuptools_scm, but I couldn't find the right entry point to understand how files are collected from the repository right before populating the tar archive. I'm not sure if I have to look for it either in build or setuptools_scm. It seems to me that the wrong folder is picked when creating the tar gz file and also the wrong content is written into the What I'm not 100% sure, is whether an sdist containing the whole repository (having the setup.py in a subfolder) could even work with |
|
That should be handled by |
|
Is this about using files from a top level folder in a subfolder python package? |
|
I think the package is in |
|
@layday Thanks for the explanation, we replied almost together :)
Thanks for this detail. I was suspecting the same in my previous comment. Therefore, I think what I'm trying to do is not that easy as I initially assumed.
I'd love to hear other opinions or suggestions about it. I'll give some more thought about how this can be done without much effort.
I suspected this 😕 I've already tried to move the logic to the setup.py, without any luck (due to what written above): setup(
use_scm_version=dict(
root="../..",
fallback_root="../..",
relative_to=str(Path(__file__).absolute()),
local_scheme="dirty-tag",
),
# [...]
Thanks, I'll give a look to
Yes, the aim is to move the python-specific files in a subfolder. Wheels seem working fine, instead creating the sdist seems more problematic. |
AFAIK, this is necessary to prevent PEP517 to create a temp folder in /tmp with just the subfolder. In fact, in this case, there would not be any .git folder (since it's in the top-level) and setuptools_scm cannot detect the version. |
|
Sdists have no concept of folders outside of their control My understanding is that a setuptools extension would be necessary to support the mechanism It may be easier to make a new git repo that uses a subrepo of the main libraries to control the structure |
|
Today I managed to give a deeper look to @henryiii's suggestion in #322 (comment), particularly the custom sdist command used in pybind11. Under the assumption that there is a git repo and the sdist package should contain all the content of the repo plus the egg metadata, the following custom sdist command provides the expected behavior. In short, I use GitPython to get all the repo files and I copy them in the tmp folder of the sdist before creating the archive. import os
from pathlib import Path
from shutil import copy2
from typing import Generator, List
import git
import setuptools.command.sdist
import setuptools_scm.integration
from cmake_build_extension import BuildExtension, CMakeExtension
from setuptools import setup
# Use a custom sdist command to handle the storage of setup.cfg in a subfolder.
# It only supports creating the sdist from git repositories.
class SDist(setuptools.command.sdist.sdist):
def make_release_tree(self, base_dir, files) -> None:
# Build the setuptools_scm configuration, containing useful info for the sdist
config: setuptools_scm.integration.Configuration = (
setuptools_scm.integration.Configuration.from_file(
dist_name=self.distribution.metadata.name
)
)
# Get the root of the git repository
repo_root = config.absolute_root
# Prepare the release tree by calling the original method
super(SDist, self).make_release_tree(base_dir=base_dir, files=files)
# Copy the entire git repo in the subfolder containing setup.cfg
for file in SDist.get_all_git_repo_files(repo_root=repo_root):
src = Path(file)
dst = Path(base_dir) / Path(file).relative_to(repo_root)
# Make sure that the parent directory of the destination exists
dst.absolute().parent.mkdir(parents=True, exist_ok=True)
print(f"{Path(file)} -> {dst}")
copy2(src=src, dst=dst)
# Create the updated list of files included in the sdist
all_files_gen = Path(base_dir).glob(pattern="**/*")
all_files = [str(f.relative_to(base_dir)) for f in all_files_gen]
# Find the SOURCES.txt file
sources_txt_list = list(Path(base_dir).glob(pattern=f"*.egg-info/SOURCES.txt"))
assert len(sources_txt_list) == 1
# Update the SOURCES.txt files with the real content of the sdist
os.unlink(sources_txt_list[0])
with open(file=sources_txt_list[0], mode="w") as f:
f.write("\n".join([str(f) for f in all_files]))
@staticmethod
def get_all_git_repo_files(repo_root: str, commit: str = None) -> List[Path]:
# Create the git Repo object
git_repo = git.Repo(path=repo_root)
# Allow passing a custom commit
commit = git_repo.commit() if commit is None else commit
# Get all the files of the git repo recursively
def list_paths(
tree: git.Tree, path: Path = Path(".")
) -> Generator[Path, None, None]:
for blob in tree.blobs:
yield path / blob.name
for tree in tree.trees:
yield from list_paths(tree=tree, path=path / tree.name)
# Return the list of absolute paths to all the git repo files
return list(list_paths(tree=commit.tree, path=Path(repo_root)))
if (Path(".") / "CMakeLists.txt").exists():
# Install from sdist
source_dir = str(Path(".").absolute())
else:
# Install from sources or build wheel
source_dir = str(Path(".").absolute().parent.parent)
if "CIBUILDWHEEL" in os.environ and os.environ["CIBUILDWHEEL"] == "1":
CIBW_CMAKE_OPTIONS = ["-DCMAKE_INSTALL_LIBDIR=lib"]
else:
CIBW_CMAKE_OPTIONS = []
setup(
cmdclass=dict(build_ext=BuildExtension, sdist=SDist),
ext_modules=[
CMakeExtension(
name="CMakeProject",
install_prefix="ycm_build_modules",
disable_editable=True,
write_top_level_init="",
source_dir=source_dir,
cmake_configure_options=[
"-DBUILD_TESTING:BOOL=OFF",
"-DYCM_NO_DEPRECATED:BOOL=ON",
]
+ CIBW_CMAKE_OPTIONS,
),
],
)The only caveat is if build extensions like those used in this example require to know the location of files like Do you have any comment of this solution? It seems quite easy and it covers all the use-cases I can think right now. |
|
The custom sdist command seems working fine. Thanks to everyone for the prompt support and suggestions 🚀 Closing. |
I'm trying to build a source distribution for a modern project that:
setup.cfgThe current behaviour is that the sdist will only contain the files of the subfolder instead of the files of the whole repository. Any
pip install <sdist>.tar.gzwould fail since the archive is not the complete repo.Please check this repo as a reference, particularly the tools/pip subfolder.
In order to reproduce the problem, execute the following commands:
Instead, using calling directly the setup.py file, creates an archive with the whole repo but with missing metadata.
Direct setup.py call
The text was updated successfully, but these errors were encountered: