Upgrade pip to 20.2+

[kba]

https://pip.pypa.io/en/stable/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020

The most significant changes to the resolver are:

It will reduce inconsistency: it will no longer install a combination of packages that is mutually inconsistent. In older versions of pip, it is possible for pip to install a package which does not satisfy the declared requirements of another installed package. For example, in pip 20.0, pip install "six<1.12" "virtualenv==20.0.2" does the wrong thing, “successfully” installing six==1.11, even though virtualenv==20.0.2 requires six>=1.12.0,<2 (defined here). The new resolver, instead, outright rejects installing anything if it gets that input.

It will be stricter - if you ask pip to install two packages with incompatible requirements, it will refuse (rather than installing a broken combination, like it did in previous versions).

So, if you have been using workarounds to force pip to deal with incompatible or inconsistent requirements combinations, now’s a good time to fix the underlying problem in the packages, because pip will be stricter from here on out.

We should explore that new dependency algorithm and consider upgrading.

[bertsky]

True, this must be examined ASAP. I'd expect things will not improve for us unambiguously, but stir up some unresolved problems (which might necessitate downgrading pip for the time being). For example, I recall we could not use pip check because of some stupid Tensorflow packaging bugs like its scipy dependency and others.

[stweil]

pip is upgraded automatically when make all creates a new virtual Python environment. So all we have to do is suggest users who create the venv manually or use an older venv to run pip install --upgrade pip from time to time.

[bertsky]

Another example of unresolved problems that the new pip behaviour will make us stumble over: Our core installation procedure!

There we have a couple of sub-packages which mutually depend on each other. But pip cannot install all of them at once (at least to my knowledge), so our make install is a loop over each of them. Each iteration updates a single sub-package but leaves the others in their previous (probably inconsistent) state:

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

ocrd 2.11.0 requires ocrd-modelfactory==2.11.0, but you'll have ocrd-modelfactory 2.13.0 which is incompatible.
ocrd 2.11.0 requires ocrd-models==2.11.0, but you'll have ocrd-models 2.13.0 which is incompatible.
ocrd 2.11.0 requires ocrd-utils==2.11.0, but you'll have ocrd-utils 2.13.0 which is incompatible.
ocrd 2.11.0 requires ocrd-validators==2.11.0, but you'll have ocrd-validators 2.13.0 which is incompatible.

This can currently be seen in our Docker builds. I guess that 2.11.0 comes from FROM ocrd/core, but the same goes for native installation under various scenarios.

[bertsky]

There we have a couple of sub-packages which mutually depend on each other. But pip cannot install all of them at once (at least to my knowledge), so our make install is a loop over each of them. Each iteration updates a single sub-package but leaves the others in their previous (probably inconsistent) state

Strange enough, but this does not seem to be a problem when using --use-feature=2020-resolver. (I get no warnings with a make ocrd -W core PIP_OPTIONS=--use-feature=2020-resolver.)

I suggest starting a PR or feature branch using --use-feature=2020-resolver by default to see the full picture.

[stweil]

@bertsky, some minutes ago I tried make all PIP_OPTIONS=--use-feature=2020-resolver and noticed that pip gets updated too late in the sub venvs, so the build failed early.

[stweil]

Looking closer at that problem, I think that the pip upgrade should ignore PIP_OPTIONS and just use pip install --upgrade pip, because that command fails when an old version of pip gets the new option.

[bertsky]

Looking closer at that problem, I think that the pip upgrade should ignore PIP_OPTIONS and just use pip install --upgrade pip, because that command fails when an old version of pip gets the new option.

Yes, agreed.

[stweil]

@kba, I think we can close this issue. We already use pip 20.2.4, and I expect it will work after October, too.

[kba]

@stweil Should we make PIP_OPTIONS=--use-feature=2020-resolver the default?

[stweil]

I don't think that's needed, because it will be the default in a few days anyway, and we would have to remove that option later again.

[kba]

Okay then :)