Upgrade to newer Python pip>=21.0

[frenck]

Proposed change

We currently lock pip to <20.3.
This will work for a while and doesn't cause problems right now. But in the long run, will become problematic. This PR serves as a draft to monitor progress towards getting it upgraded.

ℹ️ We now have a pip-check script in CI in place to protect us against creating more conflicts.
We are currently down to 11 conflicts (some have a PR open, others will probably end up being disabled as an integration).

⚠️ I've adjusted this PR to use the legacy resolver, this gives us an upgrade path for now, and will help with Python 3.10 handling as well (#59729).

Todo:

• Fix aiohttp dependency range on crownstone-sse==2.0.2
  • aiohttp is pinned to strict crownstone/crownstone-lib-python-sse#1
  • Bump crownstone-sse to 2.0.3 #60428
• Fix aiohttp dependency range on pyatmo==6.1.0
  • Release v6.2.0? jabesq-org/pyatmo#163
  • Upgrade pyatmo to 6.2.0 #59975
• Fix aiohttp dependency range on python-smarttub==0.0.27
  • Bump python-smarttub dependency to 0.0.28 #60391
• Fix click dependency range on python-miio==0.5.8
  • Bump xiaomi_miio dependency #60650
• Fix croniter dependency range on python-miio==0.5.8
  • Bump xiaomi_miio dependency #60650
• Fix cryptography dependency range on python-miio==0.5.8
  • Bump xiaomi_miio dependency #60650
• Fix PyYAML dependency range on python-miio==0.5.8
  • Bump xiaomi_miio dependency #60650
• Fix httpx dependency range devolo-plc-api==0.6.2
  • Bump devolo_plc_api to 0.6.3 #59991
• Fix async_timeout range on hangups<=0.4.16
  • Allow async-timeout v4 tdryer/hangups#524
• Fix cryptography range on hass-nabucasa<=0.51.0
  • maybe this? Update cryptography requirement from <36.0,>=2.8 to >=2.8,<37.0 NabuCasa/hass-nabucasa#297
• Fix async_timeout range on enturclient==0.2.2
  • Allow async-timeout 4.x too hfurubotten/enturclient#39
• Disable Discord integration
  • Not fixable dependency issues, upstream = archived
• Fix async_timeout on hole==0.6.0
  • Bump to 0.7.0
  • Upgrade hole to 0.7.0 #60779
• Fix async_timeout & httpx on lufdaten==0.6.5
  • Bump to 0.7.0 (for removal of loop and async_timeout)
  • Bump to 0.7.1 (for httpx)
  • 0.7.1 is missing on PyPi home-assistant-ecosystem/python-luftdaten#11
• Disable mycroft integration
  • mycroftapi 2.0 depends on websocket-client==0.44.0, repository no longer exists
• plugwise 0.8.5 depends on async-timeout<4.0
  • Remove code/functions that belong in HA Core plugwise/python-plugwise#102, plugwise/python-plugwise@988c234, fix available in >= 0.15.6
• py17track 3.2.1 depends on attrs<21.0 and >=19.3
  • Bump py17track to 2021.12.1 #60762
• pydelijn 0.6.1-1.0.0 depends on async_timeout<4.0 and >=3.0.1
  • Update async_timeout and drop loop argument bollewolle/pydelijn#12
• pyicloud 0.10.2 depends on click<=7.1.1 and >=6.0
  • https://github.com/picklepete/pyicloud/blob/master/requirements.txt
• pysmarty 0.8 depends on pymodbus==1.5.2
  • Requirement is pinned for pysmarty 0.8: pymodbus==1.5.2 z0mbieprocess/pysmarty#1
• surepy 0.7.2 depends on async-timeout<4.0.0 and >=3.0.1
  • Allow async-timeout 4.x too benleb/surepy#103

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: closes Support pip 20.3+ #45444
• This PR is related to issue:
• Link to documentation pull request:

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• The code has been formatted using Black (black --fast homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
• Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

• No score or internal
• 🥈 Silver
• 🥇 Gold
• 🏆 Platinum

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[scop]

Updated missing statuses in the list, filed a few PR's. The only fixable one without a PR or issue report at the moment is pyicloud (click>=6.0,<=7.1.1, might need some knowledge and a bit of work to fix).

[balloob]

Since we can still use the legacy resolver with the latest pip, can we update to this PR, add command line flag for legacy resolver and make sure we block any 22+ versions?

[cdce8p]

Since we can still use the legacy resolver with the latest pip, can we update to this PR, add command line flag for legacy resolver and make sure we block any 22+ versions?

That would work. According to the documentation, the old resolver hasn't yet been removed. Although there are no guarantees that it will continue to work for each new release. It might be removed even in the next minor release.

I do think however that the original intent for this PR was to track the issues which need to be addressed for the resolver change. Adding the legacy flag will only push that down the line.

There is another reason to use --use-deprecated=legacy-resolver though. The current pip version has a bug which prevents the install of Python 3.10 wheels even if they have been uploaded to PyPI. The fix was included in 21.3.2. This would definitely and allow some users to start using 3.10 even if not officially supported yet (#59729). Especially if the resolver change still needs some more time. I would suggest pinning 21.3.4 in that case.

[frenck]

do think however that the original intent for this PR was to track the issues which need to be addressed for the resolver change.

Yeah regular rebase and see what it spits out and try to resolve.

The current pip version has a bug which prevents the install of Python 3.10 wheels even if they have been uploaded to PyPI. The fix was included in 21.3.2. This would definitely and allow some users to start using 3.10 even if not officially supported yet

That is not blocking at this point.
People have been using Python 3.10 and it's not blocking our upgrade path.

That said, yeah using the legacy flag will do for now. I Will index all the locations that need updating (wheels, Dev containers, CI and a couple of spots more).

[cdce8p]

That is not blocking at this point. People have been using Python 3.10 and it's not blocking our upgrade path.

True. Although it's much more convenient to use prebuild wheels.

That said, yeah using the legacy flag will do for now. I Will index all the locations that need updating (wheels, Dev containers, CI and a couple of spots more).

👍🏻 One issue we should be aware of. Using the flag on <20.3 seems to be an error. It might be necessary to check the version before adding it to the install command, since we support a pretty large dependency range for pip atm.

option --use-deprecated: invalid choice: 'legacy-resolver' (choose from )

[cdce8p]

Something more for the Todo list. Pip 21.3 supports editable installs with just build-backends. Thus we could remove setup.py completely. #65154 (comment)

https://pip.pypa.io/en/stable/news/#v21-3

[cdce8p]

Github doesn't work, I can't leave review comments it seems 😞

Some notes here instead:

• I would suggest setting an upper limit. According to the deprecation timeline, the legacy-resolver option should have been removed already. It isn't yet, but that doesn't mean it will still be there in the next release.

pip>=21.0,<21.1

• Should the subprocess commands be updated too? Probably enough to search for "pip".
  

  core/homeassistant/util/package.py

  Line 76 in 486c068

  args = [sys.executable, "-m", "pip", "install", "--quiet", package]

[frenck]

I would suggest setting an upper limit.

Agreed but let's make it pip>=21.0,<22.1 😉

Should the subprocess commands be updated too?

I skipped that one, as that prevents e.g., custom integrations to make a mess?
For our full distribution, it matters, as it installs "everything". But I don't think it should be a problem after that. In my point of view: I rather add that one if we learn it is problematic,

[cdce8p]

Should the subprocess commands be updated too?

I skipped that one, as that prevents e.g., custom integrations to make a mess? For our full distribution, it matters, as it installs "everything". But I don't think it should be a problem after that. In my point of view: I rather add that one if we learn it is problematic,

I wasn't sure about that myself. AFAIK and if it hasn't changed since, the dependency resolver doesn't take the installed packages into account. So it would not even change much.? To have a fully deterministic build, we would probably need to add every requirement to the constraints file, but that will surely impact custom integrations.

For the full distribution, I agree. There it's necessary.

--
As I'm writing this, could we add the constraints option to the subprocess calls? Without it, custom integrations might be able to overwrite core dependencies.

[frenck]

As I'm writing this, could we add the constraints option to the subprocess calls?

Yeah, that has crossed my mind a couple of times as well, but that is kinda "meh".
Ideally, you would constrain it to all packages currently installed/in use (e.g., requirements_all.txt`) or create a pip freeze on build and use that one (which would only apply to container distributed versions).

Nevertheless, it doesn't have to be problematic if a custom integration replaces a dependency of an integration that isn't used by the end-user either...

So a choice between evils probably? 🙈 🙉 🙊

PS: Let's not put these things in this upgrade path 😉

[cdce8p]

Nevertheless, it doesn't have to be problematic if a custom integration replaces a dependency of an integration that isn't used by the end-user either...

So a choice between evils probably? 🙈 🙉 🙊

True. We could also argue that if a custom integration breaks Home Assistant, it's the responsibility of the integration author to fix it. So maybe we don't need to change anything for the time being.

PS: Let's not put these things in this upgrade path 😉

😄

[cdce8p]

LGTM!