Accidentally passing package_data a str value triggers error on Windows but not Linux

[dhimmel]

As part of manubot/manubot#51, I specified the package_data argument of setuptools.setup() in our setup.py:

    # Specify additional patterns to match files
    package_data={
        'manubot': 'cite/*.lua',
    },

The Linux Travis CI build passes, but the Windows AppVeyer builds error when attempting pip install .. Here is the error:

%CMD_IN_ENV% pip install .
Processing c:\projects\manubot
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info\manubot.egg-info
    writing pip-egg-info\manubot.egg-info\PKG-INFO
    writing dependency_links to pip-egg-info\manubot.egg-info\dependency_links.txt
    writing entry points to pip-egg-info\manubot.egg-info\entry_points.txt
    writing requirements to pip-egg-info\manubot.egg-info\requires.txt
    writing top-level names to pip-egg-info\manubot.egg-info\top_level.txt
    writing manifest file 'pip-egg-info\manubot.egg-info\SOURCES.txt'
    c:\python36\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type'
      warnings.warn(msg)
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-akczu3gv\setup.py", line 75, in <module>
        'manubot': 'cite/*.lua',
      File "c:\python36\lib\site-packages\setuptools\__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "c:\python36\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "c:\python36\lib\distutils\dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "c:\python36\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "c:\python36\lib\site-packages\setuptools\command\egg_info.py", line 278, in run
        self.find_sources()
      File "c:\python36\lib\site-packages\setuptools\command\egg_info.py", line 293, in find_sources
        mm.run()
      File "c:\python36\lib\site-packages\setuptools\command\egg_info.py", line 524, in run
        self.add_defaults()
      File "c:\python36\lib\site-packages\setuptools\command\egg_info.py", line 560, in add_defaults
        sdist.add_defaults(self)
      File "c:\python36\lib\site-packages\setuptools\command\py36compat.py", line 34, in add_defaults
        self._add_defaults_python()
      File "c:\python36\lib\site-packages\setuptools\command\sdist.py", line 134, in _add_defaults_python
        for _, src_dir, _, filenames in build_py.data_files:
      File "c:\python36\lib\site-packages\setuptools\command\build_py.py", line 66, in __getattr__
        self.data_files = self._get_data_files()
      File "c:\python36\lib\site-packages\setuptools\command\build_py.py", line 83, in _get_data_files
        return list(map(self._get_pkg_data_files, self.packages or ()))
      File "c:\python36\lib\site-packages\setuptools\command\build_py.py", line 95, in _get_pkg_data_files
        for file in self.find_data_files(package, src_dir)
      File "c:\python36\lib\site-packages\setuptools\command\build_py.py", line 114, in find_data_files
        return self.exclude_data_files(package, src_dir, files)
      File "c:\python36\lib\site-packages\setuptools\command\build_py.py", line 198, in exclude_data_files
        files = list(files)
      File "c:\python36\lib\site-packages\setuptools\command\build_py.py", line 234, in <genexpr>
        for pattern in raw_patterns
      File "c:\python36\lib\distutils\util.py", line 125, in convert_path
        raise ValueError("path '%s' cannot be absolute" % pathname)
    ValueError: path '/' cannot be absolute

In summary, this line in distutils.util.convert_path is throwing the following error:

ValueError: path '/' cannot be absolute

So at some point, it seems that the package_data relative path is converted to absolute triggering this error. Any advice on how to resolve this issue would be greatly appreciated! This is occurring with pip-18.0 in Python 3.6 & 3.7.

[dhimmel]

As @agitter discovered, the issue was with the forward-slash path separator. In other words, pip install does not throw an error if we switch to:

    # Specify additional patterns to match files
    package_data={
        'manubot': 'cite\\*.lua',
    },

In the case, of a cross-platform setup.py, presumably we could use os.path.join('cite', '*.lua') or pathlib. However, after making this adjustment, the specified file did not end up getting included in the package distribution. We gave up and switched to using MANIFEST.in.

So I guess this issue could be updated to be "package_data does not accept forward-slash separated paths on Windows". I'm not sure whether this is intentional or a bug.

[axelv]

I'm not sure whether this is intentional or a bug.

Indeed, I would be happy to know how to resolve/work around this issue.

[pganssle]

This is very puzzling, but it does not sound like deliberate behavior. Possibly an over-zealous regex somewhere?

[placidchat]

I recently met with this problem as well where a module wasn't installing, and i had to download the module to manually install it.

As a stopgap measure to change the absolute paths being defined, what i did was to alter the _manifest_normalize function in egg_info.py to null replace all paths to produce the relative paths instead, and assign that to self.filelist.files. On the other hand this might not be a good thing if self.filelist.files is not be altered.

It appears that, at least in my case, while .egg-info/SOURCES.txt is written, unfortunately setuptools relies on the in memory set of files to carry on with the setup instead of rereading the .egg-info/SOURCES.txt.

[aschmied]

@dhimmel according to the docs here the values in the package_data dict should be arrays. Try

package_data={
    'manubot': ['cite/*.lua'],
}

[dhimmel]

@aschmied looks like passing a list as a value rather than a raw string fixed the issue on Windows: see manubot/manubot#111.

package_data ends up being passed to the _get_platform_patterns as the spec argument:

setuptools/setuptools/command/build_py.py

Lines 220 to 235 in a94ccbf

	def _get_platform_patterns(spec, package, src_dir):
	"""
	yield platform-specific path patterns (suitable for glob
	or fn_match) from a glob-based spec (such as
	self.package_data or self.exclude_package_data)
	matching package in src_dir.
	"""
	raw_patterns = itertools.chain(
	spec.get('', []),
	spec.get(package, []),
	)
	return (
	# Each pattern has to be converted to a platform-specific path
	os.path.join(src_dir, convert_path(pattern))
	for pattern in raw_patterns
	)

Therefore, what I think was happening is that each character of cite/*.lua was being iterated through and treated as it's own glob. It seems that the Windows setup choked when in encountered the glob / and threw ValueError: path '/' cannot be absolute.

But now I'm confused as to why Linux works. In fact, it seems that Linux may include the package data file regardless of whether package_data or include_package_data is specified in setup.py. So there is something weird potentially happening on Linux.

For setuptools, I think the best course of action would be to add a warning if the values for package_data or exclude_package_data are strings rather than a list. I'm happy to make a PR for this. Would appreciate any guidance on how / where this package usually put these sanity checks.

[dhimmel]

Looks like the validation of the package_data and exclude_package_data fields occurs here:

setuptools/setuptools/dist.py

Lines 307 to 322 in 3adda96

	def check_package_data(dist, attr, value):
	"""Verify that value is a dictionary of package names to glob lists"""
	if isinstance(value, dict):
	for k, v in value.items():
	if not isinstance(k, str):
	break
	try:
	iter(v)
	except TypeError:
	break
	else:
	return
	raise DistutilsSetupError(
	attr + " must be a dictionary mapping package names to lists of "
	"wildcard patterns"
	)

[beniroquai]

This is an old error, but was there any solution suggested eventually? Having the same issue on Mac apparently.