Moving from setup.py to pyproject.toml

[terriko]

Setuptools is apparently going away, so we'll need to move to probably pyproject.toml. I haven't done the research on what this will take yet; it may be very simple or there may be weirdness with the way we handle checker entry points.

[XDRAGON2002]

Thoughts on "poetry"?
As we are intending to move away from setup.cfg it might be worth looking into poetry as well.

[Molkree]

Setuptools is apparently going away

Hm, I'm not aware of this, can you link to the statement from them?

Replace setup.cfg

Our setup.cfg only has configs for tools we use, I suppose you meant setup.py?

Moving to pyproject.toml is cool, moving to poetry is unnecessary IMO.

You can read a bit more on what changed with pyproject.toml here. And setuptools' site has pretty good documentation on how to package overall.

[terriko]

Hm, I'm not aware of this, can you link to the statement from them?

Honestly, I got it out of gossip from a fellow developer and haven't verified it yet. I only opened this because we were discussing it on a pull request and I didn't want to forget that we need to investigate!

[anthonyharrison]

@terriko @Molkree pip version 22.3 is now issuing the following

DEPRECATION: xxxx is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559

According to pypa, version 23.1 of pip is scheduled for April 2023.

[Rexbeast2]

@terriko we still haven't moved to pyproject.toml file. I would like to work on this.

[Rexbeast2]

I have created a PR regarding this, but the only issue that I faced was with the entry points, especially the checkers. The solution I was able to come up with was to include every checker by itself, but its really inefficient way so I didn't use it. @terriko Do let me know what improvement can be made.

[terriko]

If I had to guess, part of the move to pyproject.toml over setup.py is to avoid having pip execute arbitrary code. So... it's possible that there's a way to enumerate a directory but it's also possible that that's explicitly not included to avoid potential malicious use.

So... I'd say it's worth investigating to see if there's something similar we can use for checkers, but it's possible that we'd need to explicitly add the checkers ourselves as part of the github action that adds them to the docs and whatnot. Since we already have that script, I don't think it'll be a huge problem for us.

[anthonyharrison]

This might help

[terriko]

@Rexbeast2 did some experimentation with this in #2654 but it looks like we have a ways to go yet.

For the record, here's the the pyproject.toml from that pull request as of today (when I intend to close the pull request):

[tool.isort] 
profile = "black"

[tool.flake8]
exclude = "build"
max_line_length = 88
extend_ignore = "E203, E501"

[tool.pytest.ini_options]
asyncio_mode = "strict"

[build-system]
requires = ["setuptools >= 61.2"]
build_backend = "setuptools.build_meta"

[project]
name = "cve-bin-tool"
version = "v3.2"
description = "CVE Binary Checker Tool"
readme = "README.md"
authors = [{ name = "Terri Oda", email = "terri.oda@intel.com" }]
license = { text = "GPL-3.0-or-later" }
keywords = ["CVE", "tools", "security"]
requires-python = ">=3.7"
classifiers = [
    "Development Status :: 5 - Production/Stable",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: GNU General Public License (GPL)",
    "Natural Language :: English",
    "Operating System :: OS Independent",
    "Programming Language :: Python :: 3.7",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: Implementation :: CPython",
    "Programming Language :: Python :: Implementation :: PyPy",
]

[project.urls]
homepage = "https://github.com/intel/cve-bin-tool"

[project.optional-dependencies]
PDF = ["ReportLab"]

[tool.setuptools.packages.find]
exclude=["locales", "presentation"]

[tool.setuptools.package-data]
cve_bin_tool = [
            "schemas/*.xsd",
            ]
"cve_bin_tool.output_engine" = [
            "html_reports/templates/*.html",
            "html_reports/css/*.css",
            "html_reports/js/*.js",
            "print_mode/templates/*.html",
        ]

And here's the errors I was getting at the time when I decided to give up and close it for now.

      /tmp/pip-build-env-86geumey/overlay/lib/python3.8/site-packages/setuptools/config/_apply_pyprojecttoml.py:73: _WouldIgnoreField: `scripts` defined outside of `pyproject.toml` would be ignored.
      !!
      
              ********************************************************************************
              ##########################################################################
              # configuration would be ignored/result in error due to `pyproject.toml` #
              ##########################################################################
      
              The following seems to be defined outside of `pyproject.toml`:
      
              `scripts = ['cve-bin-tool = cve_bin_tool.cli:main', 'csv2cve = cve_bin_tool.csv2cve:main']`
      
              According to the spec (see the link below), however, setuptools CANNOT
              consider this value unless `scripts` is listed as `dynamic`.
      
              https://packaging.python.org/en/latest/specifications/declaring-project-metadata/
      
              For the time being, `setuptools` will still consider the given value (as a
              **transitional** measure), but please note that future releases of setuptools will
              follow strictly the standard.
      
              To prevent this warning, you can list `scripts` under `dynamic` or alternatively
              remove the `[project]` table from your file and rely entirely on other means of
              configuration.
      
              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
              ********************************************************************************
      
      !!
        _handle_missing_dynamic(dist, project_table)
      /tmp/pip-build-env-86geumey/overlay/lib/python3.8/site-packages/setuptools/config/_apply_pyprojecttoml.py:73: _WouldIgnoreField: `dependencies` defined outside of `pyproject.toml` would be ignored.
      !!
      
              ********************************************************************************
              ##########################################################################
              # configuration would be ignored/result in error due to `pyproject.toml` #
              ##########################################################################
      
              The following seems to be defined outside of `pyproject.toml`:
      
              `dependencies = ['aiohttp[speedups]>=3.7.4', 'beautifulsoup4', 'cvss', 'defusedxml', 'distro', 'gsutil', 'importlib_metadata>=3.6; python_version < "3.10"', 'importlib_resources; python_version < "3.9"', 'jinja2>=2.11.3', 'jsonschema>=3.0.2', 'lib4sbom>=0.3.0', 'python-gnupg', 'packaging<22.0', 'plotly', 'pyyaml>=5.4', 'requests', 'rich', 'rpmfile>=1.0.6', 'toml', 'urllib3>=1.26.5 # dependency of requests added explictly to avoid CVEs', 'xmlschema', 'zstandard; python_version >= "3.4"', '']`
      
              According to the spec (see the link below), however, setuptools CANNOT
              consider this value unless `dependencies` is listed as `dynamic`.
      
              https://packaging.python.org/en/latest/specifications/declaring-project-metadata/
      
              For the time being, `setuptools` will still consider the given value (as a
              **transitional** measure), but please note that future releases of setuptools will
              follow strictly the standard.
      
              To prevent this warning, you can list `dependencies` under `dynamic` or alternatively
              remove the `[project]` table from your file and rely entirely on other means of
              configuration.
      
              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
              ********************************************************************************
      
      !!
        _handle_missing_dynamic(dist, project_table)
      error: can't copy 'cache/gad/gemnasium-db-master-pypi/pypi/rpc.py': doesn't exist or not a regular file
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for cve-bin-tool
ERROR: Could not build wheels for cve-bin-tool, which is required to install pyproject.toml-based projects
Failed to build cve-bin-tool

Thanks to @Rexbeast2 for working on this, and sorry I don't know enough to drive this task to completion yet.

[terriko]

I'm still not sure about the correct path to take here if we want to get rid of setup.py, and I don't think it's urgent enough that I want to work on it further at this time, so I'm going to go ahead and close this now and we can re-open if it becomes more urgent in the future.