Allow excluding certain packages from pip-sync

[tuukkamustonen]

What's the problem this feature will solve?

When running pip-sync, it will uninstall also editable package(s) and you need to pip install -e . them again. This adds unnecessary steps. Adding support for pip-sync --ignore <packages...> or similar would allow to avoid that.

This topic has been discussed in #686. However, that one was closed with the assumption that if non-declared packages don't need to be touched, vanilla pip install -r requirements.txt will do.

The difference here is that while I do want to uninstall all the non-declared packages, I also want to retain the editable packages.

Describe the solution you'd like

Maybe:

pip-sync --ignore "(my-package1|my-package2)"  # regex
pip-sync --ignore my-package1 --ignore my-package2

My problem is specifically with editable packages. So it also could be:

pip-sync --ignore-editable

...but I'm not sure how difficult it would be find out which of the packages are actually editable and which are not. Also, former suggestion is more generic, should someone want to exclude actual, non-editable, packages.

Alternative Solutions

My workaround currently is to simply run pip install -e . always after pip-sync.

[tomasaschan]

Even with that workaround, thouhg, pip-sync effectively breaks if you want to run it a second time.

In other words, this workflow doesn't actually work:

$ # clone repo
$ pip-compile requirements.in
$ pip-sync requirements.txt
$ pip install -e .
# do some development
# add a new dependency to requirements.in
$ pip-compile requirements.in
$ pip-sync requirements.txt
...
AttributeError: 'NoneType' object has no attribute 'specifier'

So there's a hole to be filled here...

[AndydeCleyre]

Missing from this discussion right now is the option of putting your editable requirements in requirements.in with all the others.

Disadvantages:

• The absolute path will end up in requirements.txt if using e.g. -e . (pip-compile absolute links when given relative path #204) [Fixed by unmerged Fixup relative and absolute path handling #1329]
  • You can use -e file:. or -e file:.#egg=pip-tools, and the path will stay relative through compilation, but this is not a reliable standard (URI parsing pypa/packaging#120)
• You must compile from the same directory as the in-file (not usually a problem) [Fixed by unmerged Fixup relative and absolute path handling #1329]
• The relpath is relative to the in-file, not the out-file (not usually a problem) [Fixed by unmerged Fixup relative and absolute path handling #1329]
• Technically it will get removed and reinstalled on each sync, but that should be near-instant since it's mostly linking around (right?)
• In the big picture, "editable"/-e package installation is not yet well defined for the new pyproject.toml era, AFAIK. For example, flit projects don't support -e installation. When it's the project you're working on, you can use flit install -s, but I don't know about installing dependencies this way in a pip-tools friendly way.

See my last comment from #204 for a more thorough summary of the relpath problems and links to relevant issues further upstream.

---

All that said, for now, it will probably "just work" if you put your editable reqs in the in-file, with a form like

-e file:RELATIVEPATH#egg=PACKAGENAME

or just

-e file:RELATIVEPATH

[tomasaschan]

@AndydeCleyre I thought I had tried that, but I was doing -e ., not -e file:.. That makes all the difference! Thanks!

[AndydeCleyre]

@tomasaschan If I can get #1329 merged in, the -e . syntax should work as well.

@tuukkamustonen Do you think putting your editable requirements in your in-file adequately serves your purpose?

[tuukkamustonen]

@AndydeCleyre I have yet to try it, but if it works I think it's a feasible workaround.

I mean, better would be to allow ignoring certain packages as suggested, because people's workflows differ. Someone might not want to initialize all sub-projects always, and avoiding that would require splitting editable packages to several different files. Or maybe people don't use pip install -e . but python setup.py develop (if that's still supported) or some other way.

But yes indeed, your proposal (-e file:...) looks viable 👍🏻.

[tuukkamustonen]

Oops, accidentally clicked the wrong button.

Was about to comment that I finally tried -e file: and indeed it works nicely.

However, like already mentioned there are few drawback in packages getting uninstalled/re-installed with pip-sync. Something else is that pip-sync now always does that, even if I want to manage my editable packages via other tooling/commands. pip-sync --ignore-editable would nicely tackle that.

[AndydeCleyre]

I'll be more keen on --ignore-editable or similar if/when upstream standardizes what it is to be "editable."

Meanwhile, let's iron out the essentials.

• Does anyone here want to be able to ignore some editable packages and not others, in the same project?
• Does anyone here want to ignore non-editable packages?

Another workaround, kind of a "hook":

psync () {  # [<pip-sync-arg>...]
  pip-sync $@
  if [[ -r post_pip_sync.sh ]]; then
    ./post_pip_sync.sh
  fi
}

And you put your custom commands for installing editables and whatever in that script.

[sbaack]

I would prefer pip-sync to ignore editable installs by default and offers a --include-editable rather than a --ignore-editable flag.

This way, if I add -e file:. to requirements.in, I would use the --include-editable flag only once when I set up my environment, but afterwards I can use pip-sync normally without having to add --ignore-editable each time.

Alternatively, I would also be fine if there is no extra flag and pip-sync ignores editable installs. I don't mind typing pip install -e . once to set up the environment.

BTW, does PEP660 help with this? The aforementioned flit supports it as well.