Support options in requirements.txt in pip-sync #824

atugushev · 2019-05-23T18:35:36Z

Adds support of the following options in requirements.txt in pip-sync:

--index-url
--extra-index-url
--no-index
--find-links
--no-binary
--only-binary
--trusted-host

Closes #637
Closes #638
Inspired by #703 #464.

Changelog-friendly one-liner: Support requirements.txt options in pip-sync

Contributor checklist

Provided the tests for the changes.
Requested a review from another contributor.
Gave a clear one-line description in the PR (that the maintainers can add to CHANGELOG.md on release).
Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

codecov · 2019-05-23T18:38:25Z

Codecov Report

Merging #824 into master will increase coverage by 0.21%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #824      +/-   ##
==========================================
+ Coverage   99.26%   99.47%   +0.21%     
==========================================
  Files          34       34              
  Lines        2446     2494      +48     
  Branches      306      312       +6     
==========================================
+ Hits         2428     2481      +53     
+ Misses         10        6       -4     
+ Partials        8        7       -1

Impacted Files	Coverage Δ
tests/conftest.py	`100.00% <0.00%> (ø)`	⬆️
tests/test_cache.py	`100.00% <0.00%> (ø)`	⬆️
piptools/resolver.py	`100.00% <0.00%> (ø)`	⬆️
piptools/scripts/compile.py	`100.00% <0.00%> (ø)`	⬆️
tests/test_repository_pypi.py	`100.00% <0.00%> (ø)`	⬆️
piptools/repositories/pypi.py	`95.16% <0.00%> (+2.48%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d405bf9...e927939. Read the comment docs.

jnns · 2019-08-22T10:45:11Z

Thank you for taking care of this. 👍

We hit that problem in one of our projects. Can we help in any way to get this fix in with the next version?

blueyed · 2019-08-22T15:55:41Z

@jnns
Review and test it? :)

atugushev · 2019-08-22T18:33:50Z

Hi, @jnns! Thanks for being interesting in this PR. I'll rebase this branch to the master then (for there was a lot of changes since this PR has been created), let's see whether the tests pass.

atugushev · 2019-08-26T23:35:02Z

@jnns

We hit that problem in one of our projects. Can we help in any way to get this fix in with the next version?

FIY PR is ready to test and review. Take a look at it please if you have time.

jnns · 2019-08-27T06:22:52Z

@atugushev Thank you! Your branch fixes the problem we have with --extra-index-url. We can now sync our requirements without pip-sync bailing out.

atugushev · 2019-08-28T11:27:42Z

@jnns great! Let's wait a review from contributors then.

webknjaz · 2019-10-19T21:58:43Z

tests/test_cli_sync.py

+        ["--only-binary", ":all:"],
+    ],
+)
+@mock.patch("piptools.sync.check_call")


FTR the preferred way of patching things in pytest is to use a built-in fixture called monkeypatch.

Are there any links to the discussion about monkeypatch vs mock? I see only pytest-dev/pytest#4576 and a few others where arguments are mostly based on personal preferences. Personally, I do like @mock.patch decorator.

Would you like to open an issue so that we can discuss it and decide whether it's worth to refactor codebase form mock to monkeypatch?

I just think that it's cleaner because it's natively integrated + fewer deps.
Also, it automatically, unpatches things on exit.

fewer deps

MagicMock would require pytest-mock btw.

I think it's vice versa

Sure.. but that's not monkeypatch.

Wait, the patching method may still be monkeypatch while the value would be MagicMock().

@webknjaz

I just think that it's cleaner because it's natively integrated + fewer deps.

You are right, it could be the mock.MagicMock. Instead of using mock and patch things with mock, the suggestion is to use monkeypatch with mock.MagicMock. I don't get it, how it can be "fewer deps"?

It's fewer deps under py3. Also, I'm not sure I'd actually endorse magic mock, better define a proper fake object which would fit expectations instead of implicitly hiding possibly malicious behaviors.

I see. Pretend might be an alternative to the MagickMock, btw.

piptools/scripts/sync.py

Reduce the nesting. Co-Authored-By: Sviatoslav Sydorenko <wk@sydorenko.org.ua>

Rename _build_install_flags() to _compose_install_flags() and update the docstring. Co-Authored-By: Sviatoslav Sydorenko <wk@sydorenko.org.ua>

atugushev · 2020-02-10T19:30:55Z

@Nebukadneza I've rebased the PR against current master. Thanks for pinging!

amonsch · 2020-02-15T07:16:55Z

bump

codingjoe

I'd probably go for --binary and --source and drop the "only" or "no" prefixes.

altendky · 2020-02-16T17:46:29Z

How would that work given that pip uses the prefixes? I didn't think we were creating new options, just supporting existing pip ones.

$ venv/bin/pip --version
pip 20.0.2 from /home/altendky/venv/lib/python3.8/site-packages/pip (python 3.8)

$ venv/bin/pip install --help
<snip>
  --no-binary <format_control>
                              Do not use binary packages. Can be supplied multiple times, and each time adds to the
                              existing value. Accepts either :all: to disable all binary packages, :none: to empty the
                              set, or one or more package names with commas between them (no colons). Note that some
                              packages are tricky to compile and may fail to install when this option is used on them.
  --only-binary <format_control>
                              Do not use source packages. Can be supplied multiple times, and each time adds to the
                              existing value. Accepts either :all: to disable all source packages, :none: to empty the
                              set, or one or more package names with commas between them. Packages without binary
                              distributions will fail to install when this option is used on them.
  --prefer-binary             Prefer older binary packages over newer source packages.
<snip>

Nebukadneza · 2020-02-16T18:05:54Z

ACK with @altendky, I think it’s common convention and supported behavior pip to put the standard options of pip in requirements.txt. I think this is what users would expect to get in requirement.in too. Another downside of inventing custom options would be that users need a custom translation layer between the 2 worlds …

amonsch · 2020-02-24T08:54:35Z

bump

JensPfeifle · 2020-03-03T08:24:36Z

Would be interested in getting this merged as well. Any help needed?

atugushev · 2020-03-04T04:59:16Z

Hello @JensPfeifle,

This PR needs an approvement form one of the members.

auvipy · 2020-03-04T05:37:03Z

thanks all for your great work!

atugushev · 2020-03-04T13:10:07Z

@auvipy thanks for reviewing and merging. Looks like this PR should be rebased onto master before merging. Master's CI is broken now.

webknjaz · 2020-03-04T13:21:46Z

@atugushev that could be solved by https://bors.tech/

altendky · 2020-03-04T14:07:00Z

GitHub can also just require that PRs be up to date.

atugushev · 2020-03-04T14:22:37Z

@altendky

GitHub can also just require that PRs be up to date.

Sounds good to me!

atugushev · 2020-03-04T14:51:44Z

Addressed in #1085.

atugushev · 2020-03-05T01:29:12Z

I made a post-merge conflict fix in #1086. Please, review.

atugushev force-pushed the pip-sync-infile-options branch from 8e7e9c4 to 8b55247 Compare May 23, 2019 18:38

atugushev added the enhancement Improvements to functionality label May 23, 2019

atugushev added this to the 3.8.0 milestone May 23, 2019

atugushev requested review from blueyed and suutari-ai May 23, 2019 19:56

jazzband deleted a comment from codecov bot May 23, 2019

atugushev removed this from the 3.8.0 milestone Jun 6, 2019

atugushev requested a review from vphilippon June 6, 2019 18:46

atugushev force-pushed the pip-sync-infile-options branch 2 times, most recently from a3c7c1f to 890d1b5 Compare August 25, 2019 00:32

atugushev closed this Aug 26, 2019

atugushev reopened this Aug 26, 2019

atugushev mentioned this pull request Sep 19, 2019

Example or documentation on --index-url #545

Closed

atugushev added needs review needs rebase Need to rebase or merge and removed needs review labels Sep 26, 2019

atugushev mentioned this pull request Oct 19, 2019

[WIP] Add support for file based --extra-index-url to pip-sync #951

Closed

3 tasks

atugushev force-pushed the pip-sync-infile-options branch from 134664f to 22d2c7c Compare October 19, 2019 19:20

This was referenced Oct 19, 2019

pip-sync ignores option --no-binary #637

Closed

pip-sync ignores options in requirements.txt #638

Closed

pip-sync doesn't respect --extra-index-url in requirements.txt #952

Closed

webknjaz reviewed Oct 19, 2019

View reviewed changes

piptools/scripts/sync.py Outdated Show resolved Hide resolved

atugushev and others added 4 commits February 11, 2020 01:01

Add support options from requirements.txt in pip-sync

36db515

Address review comments

9ec51b0

Reduce the nesting. Co-Authored-By: Sviatoslav Sydorenko <wk@sydorenko.org.ua>

Address review comments

2955625

Rename _build_install_flags() to _compose_install_flags() and update the docstring. Co-Authored-By: Sviatoslav Sydorenko <wk@sydorenko.org.ua>

Refactor after rebase against master

61a7ee5

atugushev force-pushed the pip-sync-infile-options branch from 3d8db95 to 61a7ee5 Compare February 10, 2020 19:28

Use get_trusted_hosts()

e927939

codingjoe reviewed Feb 16, 2020

View reviewed changes

auvipy approved these changes Mar 4, 2020

View reviewed changes

auvipy merged commit c4dcaf7 into jazzband:master Mar 4, 2020

atugushev mentioned this pull request Mar 4, 2020

Require branches to be up to date before merging #1085

Closed

atugushev mentioned this pull request Mar 5, 2020

Pass arbitrary args to 'pip' from pip-compile and pip-sync, with '--pip-args' #1080

Merged

3 tasks

atugushev deleted the pip-sync-infile-options branch March 5, 2020 00:21

atugushev mentioned this pull request Mar 5, 2020

Fix post-merge conflicts #1086

Merged

This was referenced Mar 5, 2020

pip-sync: Support install flags in requirements.txt #703

Closed

Add support for file based --extra-index-urls to pip-sync #674

Closed

sync: Use options from the txt file #464

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support options in requirements.txt in pip-sync #824

Support options in requirements.txt in pip-sync #824

atugushev commented May 23, 2019 •

edited

Loading

codecov bot commented May 23, 2019 •

edited

Loading

jnns commented Aug 22, 2019

blueyed commented Aug 22, 2019

atugushev commented Aug 22, 2019

atugushev commented Aug 26, 2019 •

edited

Loading

jnns commented Aug 27, 2019

atugushev commented Aug 28, 2019

webknjaz Oct 19, 2019

atugushev Oct 20, 2019

webknjaz Oct 20, 2019

atugushev Oct 21, 2019 •

edited

Loading

webknjaz Oct 21, 2019

blueyed Oct 22, 2019

webknjaz Oct 23, 2019

atugushev Oct 27, 2019 •

edited

Loading

webknjaz Oct 27, 2019

atugushev Oct 27, 2019 •

edited

Loading

atugushev commented Feb 10, 2020

amonsch commented Feb 15, 2020

codingjoe left a comment

altendky commented Feb 16, 2020

Nebukadneza commented Feb 16, 2020

amonsch commented Feb 24, 2020

JensPfeifle commented Mar 3, 2020

atugushev commented Mar 4, 2020

auvipy commented Mar 4, 2020

atugushev commented Mar 4, 2020

webknjaz commented Mar 4, 2020

altendky commented Mar 4, 2020

atugushev commented Mar 4, 2020

atugushev commented Mar 4, 2020

atugushev commented Mar 5, 2020

Support options in requirements.txt in pip-sync #824

Support options in requirements.txt in pip-sync #824

Conversation

atugushev commented May 23, 2019 • edited Loading

Contributor checklist

codecov bot commented May 23, 2019 • edited Loading

Codecov Report

jnns commented Aug 22, 2019

blueyed commented Aug 22, 2019

atugushev commented Aug 22, 2019

atugushev commented Aug 26, 2019 • edited Loading

jnns commented Aug 27, 2019

atugushev commented Aug 28, 2019

webknjaz Oct 19, 2019

Choose a reason for hiding this comment

atugushev Oct 20, 2019

Choose a reason for hiding this comment

webknjaz Oct 20, 2019

Choose a reason for hiding this comment

atugushev Oct 21, 2019 • edited Loading

Choose a reason for hiding this comment

webknjaz Oct 21, 2019

Choose a reason for hiding this comment

blueyed Oct 22, 2019

Choose a reason for hiding this comment

webknjaz Oct 23, 2019

Choose a reason for hiding this comment

atugushev Oct 27, 2019 • edited Loading

Choose a reason for hiding this comment

webknjaz Oct 27, 2019

Choose a reason for hiding this comment

atugushev Oct 27, 2019 • edited Loading

Choose a reason for hiding this comment

atugushev commented Feb 10, 2020

amonsch commented Feb 15, 2020

codingjoe left a comment

Choose a reason for hiding this comment

altendky commented Feb 16, 2020

Nebukadneza commented Feb 16, 2020

amonsch commented Feb 24, 2020

JensPfeifle commented Mar 3, 2020

atugushev commented Mar 4, 2020

auvipy commented Mar 4, 2020

atugushev commented Mar 4, 2020

webknjaz commented Mar 4, 2020

altendky commented Mar 4, 2020

atugushev commented Mar 4, 2020

atugushev commented Mar 4, 2020

atugushev commented Mar 5, 2020

atugushev commented May 23, 2019 •

edited

Loading

codecov bot commented May 23, 2019 •

edited

Loading

atugushev commented Aug 26, 2019 •

edited

Loading

atugushev Oct 21, 2019 •

edited

Loading

atugushev Oct 27, 2019 •

edited

Loading

atugushev Oct 27, 2019 •

edited

Loading