Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python][pip-compile] Dependabot cannot resolve dependency version if the lockfile is used as constraints in other manifest files. #6550

Closed
1 task done
nwalsh1995 opened this issue Jan 30, 2023 · 11 comments · Fixed by #6757
Labels
L:python:pip-compile Python packages via pip-compile T: bug 🐞 Something isn't working

Comments

@nwalsh1995
Copy link

nwalsh1995 commented Jan 30, 2023

Is there an existing issue for this?

  • I have searched the existing issues

Package ecosystem

pip/pip-compile

Package manager version

pip 22.3.1 pip-compile 6.12.1

Language version

3.7.5

Manifest location and content before the Dependabot update

No response

dependabot.yml content

No response

Updated dependency

No response

What you expected to see, versus what you actually saw

I expected dependabot to strip the extras out of this file since it is being used as a constraint to another file which dependabot will attempt to generate.

Instead dependabot does not strip the extras out of the produced file which means that the process fails with

pip._internal.exceptions.InstallationError: Constraints cannot have extras

Here is the code where dependabot runs pip-compile to resolve the version.

filenames_to_compile.each do |filename|
# Shell out to pip-compile.
# This is slow, as pip-compile needs to do installs.
options = pip_compile_options(filename)
options_fingerprint = pip_compile_options_fingerprint(options)
run_pip_compile_command(
"pyenv exec pip-compile -v #{options} -P #{dependency.name} #{filename}",
fingerprint: "pyenv exec pip-compile -v #{options_fingerprint} -P <dependency_name> <filename>"
)
next if dependency.top_level?
# Run pip-compile a second time for transient dependencies
# to make sure we do not update dependencies that are
# superfluous. pip-compile does not detect these when
# updating a specific dependency with the -P option.
# Running pip-compile a second time will automatically remove
# superfluous dependencies. Dependabot then marks those with
# update_not_possible.
write_original_manifest_files
run_pip_compile_command(
"pyenv exec pip-compile #{options} #{filename}",
fingerprint: "pyenv exec pip-compile #{options_fingerprint} <filename>"
)
end

Given the manifest files below, notice how if we update a .in file that is being used as a -c to another file, then pip-compile will fail to run on that file.

Suggestion

There is a pip-tools discussion around making --strip-extras the default behavior: jazzband/pip-tools#1613. Therefore, pass --strip-extras by default in the two pip-compile calls here:

"pyenv exec pip-compile -v #{options} -P #{dependency.name} #{filename}",

PR here #6551

Alternative to suggestion

dependabot supports parsing --strip-extras in the file_updater, it should support it in the version resolver as well:

options << "--strip-extras" if requirements_file.content.include?("--strip-extras")

Native package manager behavior

No response

Images of the diff or a link to the PR, issue, or logs

updater | ERROR <job_592338820> Error processing click (Dependabot::SharedHelpers::HelperSubprocessFailed)
updater | ERROR <job_592338820> Using indexes:
updater | <job_592338820>   https://pypi.org/simple
updater | <job_592338820> 
updater | <job_592338820>                           ROUND 1                           
updater | <job_592338820> /usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/pip/_internal/req/req_install.py:877: PipDeprecationWarning: DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement. Discussion can be found at https://github.com/pypa/pip/issues/8210
updater | <job_592338820>   issue=8210,
updater | <job_592338820> Traceback (most recent call last):
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/bin/pip-compile", line 8, in <module>
updater | <job_592338820>     sys.exit(cli())
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
updater | <job_592338820>     return self.main(*args, **kwargs)
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/click/core.py", line 1055, in main
updater | <job_592338820>     rv = self.invoke(ctx)
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
updater | <job_592338820>     return ctx.invoke(self.callback, **ctx.params)
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/click/core.py", line 760, in invoke
updater | <job_592338820>     return __callback(*args, **kwargs)
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
updater | <job_592338820>     return f(get_current_context(), *args, **kwargs)
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/piptools/scripts/compile.py", line 580, in cli
updater | <job_592338820>     results = resolver.resolve(max_rounds=max_rounds)
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/piptools/resolver.py", line 592, in resolve
updater | <job_592338820>     compatible_existing_constraints=compatible_existing_constraints,
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/piptools/resolver.py", line 625, in _do_resolve
updater | <job_592338820>     check_supported_wheels=not self.options.target_dir,
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
updater | <job_592338820>     collected = self.factory.collect_root_requirements(root_reqs)
updater | <job_592338820>   File "/usr/local/.pyenv/versions/3.7.16/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 481, in collect_root_requirements
updater | <job_592338820>     raise InstallationError(problem)
updater | <job_592338820> pip._internal.exceptions.InstallationError: Constraints cannot have extras
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:224:in `run_command'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:282:in `run_pip_compile_command'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:82:in `block (3 levels) in fetch_latest_resolvable_version_string'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:76:in `each'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:76:in `block (2 levels) in fetch_latest_resolvable_version_string'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/shared_helpers.rb:168:in `with_git_configured'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:72:in `block in fetch_latest_resolvable_version_string'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `block in in_a_temporary_directory'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `chdir'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `in_a_temporary_directory'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:71:in `fetch_latest_resolvable_version_string'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:57:in `resolvable?'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker.rb:39:in `latest_resolvable_version'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:74:in `preferred_resolvable_version'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:260:in `preferred_version_resolvable_with_unlock?'
updater | ERROR <job_592338820> /home/dependabot/python/lib/dependabot/python/update_checker.rb:108:in `preferred_version_resolvable_with_unlock?'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:252:in `numeric_version_can_update?'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:202:in `version_can_update?'
updater | ERROR <job_592338820> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:44:in `can_update?'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:486:in `requirements_to_unlock'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:259:in `check_and_create_pull_request'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:109:in `check_and_create_pr_with_error_handling'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:82:in `block in run'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:82:in `each'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:82:in `run'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/update_files_job.rb:17:in `perform_job'
updater | ERROR <job_592338820> /home/dependabot/dependabot-updater/lib/dependabot/base_job.rb:50:in `run'
updater | ERROR <job_592338820> bin/update_files.rb:23:in `<main>'

Smallest manifest that reproduces the issue

requirements-shared.in

play-scraper

requirements-shared.txt. This is what dependabot will generate because it doesn't pass --strip-extras. Notice requests[security]

#
# This file is autogenerated by pip-compile with python 3.7
# To update, run:
#
#    pip-compile --output-file=requirements-shared.txt requirements-shared.in
#
beautifulsoup4==4.11.1
    # via play-scraper
certifi==2022.12.7
    # via requests
charset-normalizer==3.0.1
    # via requests
idna==3.4
    # via requests
lxml==4.9.2
    # via play-scraper
play-scraper==0.6.0
    # via -r requirements-shared.in
requests[security]==2.28.2
    # via
    #   play-scraper
    #   requests-futures
requests-futures==1.0.0
    # via play-scraper
soupsieve==2.3.2.post1
    # via beautifulsoup4
urllib3==1.26.14
    # via requests

requirements-uses.in

-c requirements-shared.txt

click

Note: command below is what dependabot will effectively be running.

$ pip-compile requirements-uses.in --dry-run --resolver=backtracking
/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/pip/_internal/req/req_install.py:877: PipDeprecationWarning: DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement. Discussion can be found at https://github.com/pypa/pip/issues/8210
  issue=8210,
Traceback (most recent call last):
  File "/home/discord/dependabot-script/ttt123123/venv/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/piptools/scripts/compile.py", line 580, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/piptools/resolver.py", line 592, in resolve
    compatible_existing_constraints=compatible_existing_constraints,
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/piptools/resolver.py", line 625, in _do_resolve
    check_supported_wheels=not self.options.target_dir,
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/home/discord/dependabot-script/ttt123123/venv/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 481, in collect_root_requirements
    raise InstallationError(problem)
pip._internal.exceptions.InstallationError: Constraints cannot have extras
@nwalsh1995 nwalsh1995 added the T: bug 🐞 Something isn't working label Jan 30, 2023
@nwalsh1995 nwalsh1995 changed the title [python][pip-compile] Dependabot cannot update dependencies if the lockfile is used as constraints in other files. [python][pip-compile] Dependabot cannot update dependencies if the lockfile is used as constraints in other manifest files. Jan 30, 2023
@nwalsh1995 nwalsh1995 changed the title [python][pip-compile] Dependabot cannot update dependencies if the lockfile is used as constraints in other manifest files. [python][pip-compile] Dependabot cannot resolve dependency version if the lockfile is used as constraints in other manifest files. Jan 31, 2023
@deivid-rodriguez
Copy link
Contributor

Hei @nwalsh1995, thanks for reporting this.

I just tried the latest command you present, but with the --strip-extras and it fails with the same error, so I don't think your patch is going to do anything? In general, when updating a file, Dependabot looks at pip-compile's header lines to learn which flags it needs to use to update that file correctly. But in this case, we're not looking at updating the file, but at reading its information to perform updates, so pip-compile flags probably won't help for that?

My guess is that user here is still on a pip version that worked with extras in constraint files, but Dependabot is using a more recent version of pip that does not work. SO we have several options:

I think we have two options here:

  • Detect the situation and use an older pip version to perform the update. But I'm not sure how to detect which pip version to use, we'd probably need to keep that information ourselves which we generally prefer not to.

  • Detect the situation, regenerate the constraints file with --strip-extras, and then perform the update using the new constraints file without extras. That should work but the final PR will include changes unrelated to the upgrade (the upgrade of the constraints file to not use extras), which is generally not desired.

It's tricky, I don't think the solution is as simple as adding the --strip-extras flag to all of our pip-compile commands :(

@deivid-rodriguez deivid-rodriguez added the L:python:pip-compile Python packages via pip-compile label Jan 31, 2023
@nwalsh1995
Copy link
Author

nwalsh1995 commented Jan 31, 2023

@deivid-rodriguez can you try with these files instead? Without the PR I expect it to fail, and with the PR its expected to pass. Also, if possible can you tell me how you're testing the changes so that I can reproduce. Our files have extras stripped from requirements.in and the corresponding .txt files (we use --strip-extras to generate the file), so our files are compliant with the newest version of pip, but when dependabot attempts to resolve versions it doesn't include the flag, so I'm surprised the PR doesn't work.

Here are the files, you might have to play with version numbers to force an update of something out of date. Notice all of these files have been generated with --strip-extras. When dependabot runs it will recreate the requirements-shared.txt with an extra (requests[security]) during version resolution because it doesn't include --strip-extras flag. When it moves on to the next filename_to_compile pip-compile will throw an error because there is an extra in -c file.

requirements-shared.in

play-scraper

requirements-shared.txt

#
# This file is autogenerated by pip-compile with Python 3.7
# by the following command:
#
#    pip-compile --resolver=backtracking --strip-extras requirements-shared.in
#
beautifulsoup4==4.11.1
    # via play-scraper
certifi==2022.12.7
    # via requests
charset-normalizer==3.0.1
    # via requests
idna==3.4
    # via requests
lxml==4.9.2
    # via play-scraper
play-scraper==0.6.0
    # via -r requirements-shared.in
requests==2.28.0
    # via
    #   play-scraper
    #   requests-futures
requests-futures==1.0.0
    # via play-scraper
soupsieve==2.3.2.post1
    # via beautifulsoup4
urllib3==1.26.14
    # via requests

requirements-uses.in

-c requirements-shared.txt

click

requirements-uses.txt

#
# This file is autogenerated by pip-compile with Python 3.7
# by the following command:
#
#    pip-compile --resolver=backtracking --strip-extras requirements-uses.in
#
click==8.1.3
    # via -r requirements-uses.in
importlib-metadata==6.0.0
    # via click
typing-extensions==4.4.0
    # via importlib-metadata
zipp==3.12.0
    # via importlib-metadata

@deivid-rodriguez
Copy link
Contributor

Sorry, I misunderstood your issue, I thought your constraint files had extras and Dependabot was not being able to deal with them. You're saying that dependabot is adding the extras itself during them update, and then crashing because of that.

Still I could not reproduce, I'm getting the same result, with or without your change. To test I pushed your manifests to a repo, downgrading click to 8.1.2 and play-scraper to 0.5.0 and I got proper updates for both.

I guess your issue happens when upgrading subdependencies? What's your configuration in .dependabot.yml?

@nwalsh1995
Copy link
Author

nwalsh1995 commented Jan 31, 2023

@deivid-rodriguez can you try from this repo? It's closer to our actual usage and dependabot is failing there with the same logs as in the OP issue.

https://github.com/nwalsh1995/depbot-example

@deivid-rodriguez
Copy link
Contributor

I see, you have extras in the constraints.in file not in the txt file.

I still get the same error with your patch though. I think what pip is telling us is that it can't resolve if a constraints file (included through -c) includes extras.

@nwalsh1995
Copy link
Author

@deivid-rodriguez Another example which is closer to my initial report, this one doesn't contain extras in any of the files, and they are pulled in by dependabots run: https://github.com/nwalsh1995/depbot-example2

How are you running with the patch? It will be helpful for me to reproduce locally to see what's going on.

@nwalsh1995
Copy link
Author

nwalsh1995 commented Feb 1, 2023

I think what pip is telling us is that it can't resolve if a constraints file (included through -c) includes extras.

Yes, but the files don't contain extras anywhere in the constraints file. Somehow extras are being generated.

  • In example 1 we have constraints in the .in file but the .txt is generated with --strip-extras. Note that extras in the .in file is okay since nothing declares a -c on the .in, only the .txt.
  • In example 2 we have no constraints but still get constraints error.

I have no explanation of why the suggested PR doesn't work though, will need more info on how to debug that for me to help.

@nwalsh1995
Copy link
Author

Okay, I managed to repro the constraint problem locally on my docker instance (dependabot-script's generic-update-script.rb) for example2. Re-ran with the patch and the run finished successfully.

I cannot repro the exception in depbot-example1 using dependabot-script, so I'm not sure what's different between that and what's running on GH.

@deivid-rodriguez
Copy link
Contributor

From your PR branch, run bin/docker-dev-shell python, and then from the devcontainer, bin/dry-run.rb pip nwalsh1995/depbot-example2. I haven't yet run that, so not sure if your patch works against the latest repo you posted.

@nwalsh1995
Copy link
Author

nwalsh1995 commented Feb 1, 2023

For depbot-example its reproducible by running pip-compile instead of the whole dependabot stack:

pip-compile -v --build-isolation --allow-unsafe --strip-extras --resolver backtracking --dry-run -P googleapis-common-protos\[grpc\] requirements.in

Internally pip-compile treats the things inside of the requirements.in file as constraints, and we have extras in there so it fails. I'm not sure whether dependabot is using pip-compile incorrectly, or pip-compile has a bug for this path.

#6551 fixes the case in depot-example2 where someone uses pip-compile and has no extras in their requirements.in and a dependency tries to pull it in.

There will have to be another issue/PR for handling manifest files with constraints which is reproducible in depbot-example.

@nwalsh1995
Copy link
Author

FYI I've created jazzband/pip-tools#1806 for pip-compile.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
L:python:pip-compile Python packages via pip-compile T: bug 🐞 Something isn't working
Projects
None yet
2 participants