Unsafe requirements are no longer included in the requirements.txt generated by pip-compile

[VagueAndroid]

After v6.12.0 the unsafe packages pip and setuptools are no longer included in the requirements.txt when --allow-unsafe is used in pip-compile.

There was a change made in writer.py (#1766)
I think this line should be using unsafe_packages if the user has specified allow_unsafe rather than if they have not:

unsafe_packages = unsafe_packages if not self.allow_unsafe else set()
I think this should be:
unsafe_packages = unsafe_packages if self.allow_unsafe else set()

Environment Versions

1. Ubuntu 18
2. Python version: 3.7.11
3. pip version: 22.3.1
4. pip-tools version: 6.12.1

Steps to replicate

Ensure requirements.in file includes pip and then run:
pip-compile --verbose --allow-unsafe --output-file requirements.txt requirements.in

Expected result

requirements.txt should end with this:

The following packages are considered to be unsafe in a requirements file:

pip==22.3.1
# via -r requirements.in

Actual result

The unsafe packages are not listed in the requirements.txt at all

[webknjaz]

If you don't mind sending a PR with a failing test, it'd be highly appreciated.

[q0w]

requirements.in

pip

requirements.txt

#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --allow-unsafe requirements.in
#
pip==22.3.1
    # via -r requirements.in

[georgipopovhs]

I noticed that in previous versions of pip-tools (e.g. 6.6.2) the output would contain "unsafe packages" message at the end:

#
# This file is autogenerated by pip-compile with python 3.7
# To update, run:
#
#    pip-compile --allow-unsafe
#

# The following packages are considered to be unsafe in a requirements file:
setuptools==65.6.3
    # via -r requirements.in

What happened in a larger Python project we have at work, is that after upgrading pip-tools, the unsafe message at the bottom disappeared and setuptools moved "up" among the list of regular packages.
Was that an intended change @atugushev?

@VagueAndroid could it be that instead of not listing the package at all, it was still included but got moved elsewhere in the output?

[q0w]

@georgipopovhs could you share your minimal reproducer?

[georgipopovhs]

requirements.in

setuptools

Old version & format:

pip install pip-tools==6.11.0
pip-compile --allow-unsafe

requirements.txt

#
# This file is autogenerated by pip-compile with Python 3.7
# by the following command:
#
#    pip-compile --allow-unsafe
#

# The following packages are considered to be unsafe in a requirements file:
setuptools==65.6.3
    # via -r requirements.in

New version & format:

pip install pip-tools==6.12.0
pip-compile --allow-unsafe

requirements.txt

#
# This file is autogenerated by pip-compile with Python 3.7
# by the following command:
#
#    pip-compile --allow-unsafe
#
setuptools==65.6.3
    # via -r requirements.in

[q0w]

Proposed change fails in tests/test_cli_compile.py::test_allow_unsafe_option

[atugushev]

Was that an intended change?

@georgipopovhs No, that's a regression.

[q0w]

This test is wrong, is not it?

pip-tools/tests/test_cli_compile.py

Lines 1497 to 1507 in 59816bd

	pytest.param(
	"--allow-unsafe",
	dedent(
	"""\
	small-fake-a==0.1
	small-fake-b==0.3
	small-fake-with-deps==0.1
	"""
	),
	id="allow all packages",
	),

Considering UNSAFE_PACKAGES={"small-fake-with-deps"}
it should be

small-fake-a==0.1
small-fake-b==0.3

# The following packages are considered to be unsafe in a requirements file:
small-fake-with-deps==0.1

And why was it patching UNSAFE_PACKAGES only in resolver.py and not in writer.py too

monkeypatch.setattr("piptools.resolver.UNSAFE_PACKAGES", {"small-fake-with-deps"})

[atugushev]

And why was it patching UNSAFE_PACKAGES only in resolver.py and not in writer.py too

@q0w that was an oversight I guess.