Duplicates are not supported in requirements.txt when run with --disable-pip

[npalmius]

Bug description

Duplicate entries (with extras) in a requirements file cause an error when auditing if using the --disable-pip option.

This is closely related to #564 (but I would say that it is not a duplicate).

Reproduction steps

Use the following requirements.txt file (generated by poetry):

dash==2.7.1 ; python_full_version < "3.9.0" and python_version >= "3.8"
dash[compress]==2.7.1 ; python_full_version < "3.9.0" and python_version >= "3.8"

This gives the following outputs:

Without --disable-pip:

$ pip-audit -r requirements.txt
No known vulnerabilities found

With --disable-pip:

$ pip-audit -r requirements.txt --disable-pip --no-deps
WARNING:pip_audit._cli:The --no-deps flag is redundant when used with --disable-pip
WARNING:pip_audit._cli:--no-deps is supported, but users are encouraged to fully hash their pinned dependencies
WARNING:pip_audit._cli:Consider using a tool like `pip-compile`: https://pip-tools.readthedocs.io/en/latest/#using-hashes
ERROR:pip_audit._cli:package dash has duplicate requirements: dash[compress]==2.7.1 (from RequirementLine(line_number=2, line='dash[compress]==2.7.1 ; python_full_version < "3.9.0" and python_version >= "3.8"', filename=PosixPath('requirements.txt')))

For reference, the error is the same if the requirements.txt file is generated with hashes, so this is specifically related to --disable-pip, not --no-deps:

$ pip-audit -r requirements.txt --disable-pip
ERROR:pip_audit._cli:package dash has duplicate requirements: dash[compress]==2.7.1 (from RequirementLine(line_number=290, line='dash[compress]==2.7.1 ; python_full_version < "3.9.0" and python_version >= "3.8"     --hash=sha256:0935490e3dd2d2a046291989824732901437bc0336d592eadcb93fd9c866d4c9     --hash=sha256:13cd19d19288a8fb9c2a5cd069c72643758fc2fcda68cacc72474a4ec6b7667d', filename=PosixPath('requirements.txt')))

Expected behavior

pip-audit -r requirements.txt --disable-pip should perform the same as pip-audit -r requirements.txt

Platform information

• OS name and version:
• pip-audit version: pip-audit 2.6.1
• Python version: Python 3.8.17
• pip version: pip 23.1.2

[woodruffw]

Thanks for the report! I or someone else should have some time to triage this in the coming days.

[npalmius]

Thanks @woodruffw - it's not urgent as it runs fine without --disable-pip, but it's a nice optimisation as poetry is managing full dependency resolution so we know that the requirements file is complete.