Metadata and options are now could be set in configuration files

[idlesign]

Hi,

It is useful to be able to have metadata and other options declared in configuration files (e.g. setup.cfg). That not only allows some automation, but also reduces bolerplate code in some cases.
See also #394.

This chage allows us to have something like this in our config files:

[metadata]
name = some_name
version = attr: some_package.VERSION
description = Some description
long_description = file: README
keywords = one, two
license = BSD 3-Clause License

[metadata.classifiers]
Framework :: Django
Programming Language :: Python :: 3.5

[options]
zip_safe = False
include_package_data = True
packages = find:
scripts = 
  bin/one.py
  bin/two.py

[options.package_data]
* = *.txt, *.rst
hello = *.msg

[options.extras_require]
pdf = ReportLab>=1.2; RXP
rest = docutils>=0.3; pack ==1.1, ==1.3

Currently every metadata and other options are supported (or I'd like to believe that). More examples in test_config.py.
What I'm concerned with though is whether we need to sandbox file: directive which is supported by some options for security considerations (shouldn't it be impossible to read data from outside).

If overall design and implementation is fine I could update the documentation.

Below are supported ways of value definition for various options.

Terms used

• list-comma: dangling list or comma-separated values string
• list-semi: dangling list or semicolon-separated values string
• bool: True is 1, yes, true
• dict: list-comma where keys from values are separated by =
• attr: value could be read from module attibute
• file: value could be read from a file
• section: values could be read from a dedicated section

Metadata

• version: string or attr (iterables, callables are supported; others - through str())
• platforms (alias: platform): list-comma
• keywords: list-comma
• provides: list-comma
• requires: list-comma
• obsoletes: list-comma
• classifiers (alias: classifier): file, section, or list-comma
• license: file or string
• description (alias: summary): file or string
• long_description: file or string
• author_email (alias: author-email): string
• url (alias: home-page): string

Options

• zip_safe: bool
• setup_requires: list-semi
• install_requires: list-semi
• extras_require: section
• entry_points: file, section
• use_2to3: bool
• use_2to3_fixers: list-comma
• use_2to3_exclude_fixers: list-comma
• convert_2to3_doctests: list-comma
• scripts: list-comma
• eager_resources: list-comma
• dependency_links: list-comma
• tests_require: list-semi
• include_package_data: bool
• packages: list-comma or find:
• package_dir: dict
• package_data: section
• exclude_package_data: section
• namespace_packages: list-comma

[idlesign]

Fails on unrelated test_easy_install.py, and test_sandbox.py.

[idlesign]

@jaraco What I can do for this issue to be reviewed?

[jaraco]

I'm trying to re-start the Travis build now that the test failures have been fixed in master, but it's not restarting readily. I restarted the appveyor tests, and the Python 2.7 build is now passing, but not the Python 3.

[jaraco]

The Travis builds are running now and are showing the same - test failures on Python 3.

[jaraco]

I like this a lot. The implementation is well-factored and well-thought-out.

I'd like @ncoghlan's thoughts - does this feature fit well into the plans for packaging metadata? Does it conflict or align with the goals of PEP 426 or similar PEPs?

A few aspects I'll want to consider before acceptance:

• Fix test failures on Python 3.
• Update docs (as you've offered to do), probably following additional concurrence on the implementation. An important question to answer in the docs: What is the precedence when a value is defined both in setup.cfg and setup.py imperatively? I'd expect setup.py values to take precedence.
• Use from __future__ import absolute_import, unicode_literals and maybe others in the new modules for better future compatibility.
• Assess this behavior in light of pbr. IIRC, pbr also uses setup.cfg to store declarative metadata. Does this feature interfere with that functionality?
• Run the integration tests. There's a tests/test_pypi.py which can be run manually. Let's run those tests as well (which may answer the question about pbr).

[idlesign]

Py3 failures seem to be connected with pbr indeed, since those 'author_email': 'openstack-dev@lists.openstack.org must have come from there.

I'll peep into pbr to deside how to handle those cases. Yet most probably the only option is not to raise Unknown distribution option exception and to skip unknown keys silenlty.

I'd expect setup.py values to take precedence.

Yes, that's how it's currently done.

[idlesign]

Skimmed through pbr docs. There seems to be another option: we can handle pbr notation of metadata section. And it's easy enough.

Yet I'm not sure whether it'll the right thing to do, since keys used there do not fully correspond with setup() argument names (classifier vs classifiers; author-email vs author_email; etc.) that we are all used to.

So I propose that we skip unknown keys, and handle both forms for those clashing (it seems to be keywords only).

What do you think?

[idlesign]

Items from #862 (comment) seem to be covered except docs update, which I'll do after @ncoghlan review.

There's one more thing I'm concerned with: it's seems that it'll the the right thing to expose even minimal API, so that automation tools could easily get already parsed and preprocessed data from configs without ado.

[ncoghlan]

@jaraco When we decided to go in the pyproject.toml direction for the build system independent toolchain bootstrapping, that left setup.cfg free for setuptools/d2to1/pbr to use as you wish.

[idlesign]

Now with PEP-518, pbr and d2to1 in mind, I think it will be wise not only to handle metadata keys introduced by the latter two, but also to keep closer to .toml. It should simplify the task of transition for users.

[idlesign]

@jaraco Seems to be quite complete. Docs require proof reading, I guess.

[anthrotype]

Thanks for working on this! 👍