Release 22.3

[pradyunsg]

The next pip release is due this month!

[pradyunsg]

@pypa/pip-committers Anyone got bandwidth to pick this up?

[pfmoore]

I can have a go. I may need someone to help with the get-pip release, as the new automation requires signing the commit, and I'm not sure I have working git signing set up.

I'll aim for next weekend, 15/16 Oct - does anyone have any ongoing work that they need to get in before that date? As per our standard policy, I'll pick up whatever is in main at the time of release, so if it doesn't get merged, it won't get into the release 🙂

I'm going to be pretty strict on bugfix releases. So my aim will be ideally no bugfix release, or at worst one if we have serious regressions. As we're so close to the Python 3.11 release, I don't want to put a version of pip into ensurepip, with a bugfix released days later. (I haven't yet decided if we should try to get 22.3 into Python 3.11 - it may be rejected by the RM in any case). Also, multiple bugfixes means more work for virtualenv, more wheels in users' virtualenv seed data directories, etc. Plus it's more work for me 😉 So please, make sure anything that gets merged is as solid as we can make it.

[uranusjr]

Given the concerns of this being close to the 3.11 release, would this be a good time to push back the schedule to line up releases with CPython? We talked about this a bit in the past but there was no timeline proposed, and this is as good a time as any.

[pfmoore]

I'm happy enough to just do 22.3 as per our current schedule, and ask the RMs if it can go into 3.11. If it can't, 3.11 will ship with 22.2.2 and there's no harm done. Any change to our release schedule now will have no effect on what goes into 3.11, other than removing any possibility of 22.3 being in there.

We can change our release schedule for 2023 onward. I think that makes more sense, as long as we start the discussion straight after 22.3 is done¹. Trying to have such a discussion just before a release feels like it just means we'll rush things.

Footnotes

1. As per my point about being strict on bugfixes, I'd be happy to assume that 22.3 will be done by the end of October. ↩

[sbidoul]

does anyone have any ongoing work that they need to get in before that date

#11359 is ready to merge if everyone is happy with the wording of the various warnings now triggered by --install-option.

[pfmoore]

Fine with me, but be aware that if users complain that the deprecation causes issues for them, I won't consider that as a bug, so they'll have to deal with it until 23.0.

[sbidoul]

Fine with me, but be aware that if users complain that the deprecation causes issues for them, I won't consider that as a bug, so they'll have to deal with it until 23.0.

Of course, that is also my approach to this. My question is more about whether pip maintainers think the wording is ok, from a UX perspective.

[pradyunsg]

I'm not sure I have working git signing set up.

Try https://garrytrinder.github.io/2021/09/get-verified-configure-signed-git-commit-signing-on-windows?

---

There's a somewhat major release of packaging due: https://github.com/pypa/packaging/blob/main/CHANGELOG.rst#unreleased

I wanted to get it into the last release but that slipped due to, well, availability of people. It contains a few potentially-disruptive changes though, so... I don't know how compatible that might be with a strict bugfix policy. :)

[pfmoore]

Try https://garrytrinder.github.io/2021/09/get-verified-configure-signed-git-commit-signing-on-windows?

Nope, I don't want anything to do with GPG or having to manage GPG keys. I think I have something set up in 1Password that might work, but if not, someone else will have to sign the release, sorry.

There's a somewhat major release of packaging due: https://github.com/pypa/packaging/blob/main/CHANGELOG.rst#unreleased

Will it be released by the end of next week?

It contains a few potentially-disruptive changes though, so... I don't know how compatible that might be with a strict bugfix policy. :)

If you have concerns about its stability, I'm happy to leave it out of this pip release, but if it goes in then I'm going to stick with the policy. If it goes in and it caused major issues, I'd just revert packaging to the current version - I definitely don't want to get into a cycle of bugfixes with pip trying to patch around packaging changes.

[pradyunsg]

Sounds reasonable. Ping me when you've uploaded to PyPI, and I'll put the get-pip.py changes out. :)

Update: Looks like you can use 1password to sign commits! https://blog.1password.com/git-commit-signing/

[pfmoore]

Update: Looks like you can use 1password to sign commits

Yes, that's what I now have set up. So hopefully it'll work - we'll see 😉 Although I do think the automation should cater for the fact that someone might not have signing working - either check and fail with a message saying what to do in that case, or allow unsigned uploads with a warning. Just blowing up after committing seems suboptimal...

[pradyunsg]

I agree -- I do think that would need someone who doesn't have everything set up to sign by default to contribute that. 🤪

[pfmoore]

I guess so, maybe. Or maybe just remove the -s, as people who do have everything set up to sign by default wouldn't be affected and the rest of us would no longer get an error 😉

More seriously, I'm way too scared of the get-pip release automation to be willing to mess with it at this sort of level. I'd be willing to modify it, but only if people were OK with a major overhaul, so that the automation never changes the main branch at all, and releases involve the same sort of branch/PR/merge workflow that pip's release uses. And I'd probably want to make "sign" an (optional) independent action in that case as well. OTOH, that level of major overhaul likely needs more time than I have free right now, so it'll end up on the "maybe someday" stack.

I suspect at some point, we'll need to formalise what being a RM for pip involves. But I'm not in a hurry to do that, as it's quite possible that in doing so, we'll reduce the pool of people we have who can (or are willing to) be RMs...

Or we go the other way, and fully automate the release process so that everything happens with no human intervention as soon as someone pushes a tag to pip's main branch. But achieving that level of automation will be a lot of work, and it's work that's extremely hard to test. So I don't see that happening anytime soon.

Anyway, this is a digression. Let's drop it for now and not pollute the release issue with unrelated topics.

[pradyunsg]

get-pip.py's release automation creates a PR, and doesn't modify main directly.

[pradyunsg]

@pfmoore Wanna self-assign yourself to reflect that you're the RM for this cycle? :)

[pfmoore]

I just noticed that certifi appears to have switched to using the new as_file API from importlib.resources, which is documented as only available from Python 3.9+. Certifi doesn't have a dependency on the importlib_resources backport, and as far as I can see we don't vendor it for any other reason. I haven't tested the new certifi with Python 3.7, but I'm not willing to risk breaking pip over this issue.

Therefore, I'm not going to upgrade our vendored copy of certifi for 22.3, as I don't have the time to deal with this issue properly. But if anyone else wants to sort out a fix, that's fine with me.

Which leads onto another question - how do I tell nox -s vendoring -- --upgrade to not upgrade a particular dependency?

[pradyunsg]

how do I tell nox -s vendoring -- --upgrade to not upgrade a particular dependency?

You can't. You'll have to run an interactive git rebase (git rebase -i main) and delete the line that represents the commit that upgrades that package. (if you use VS Code, you might want to configure it as the git editor, FWIW)

[pfmoore]

Rats. The problem is that the upgrade fails on that package, because our patch no longer applies cleanly.

I'll see what I can do about it. Maybe delete our patch, let it do the upgrade, then hack it to remove the certifi upgrade.

[pradyunsg]

Yea, try that (make a commit for removing the patch too).

[pfmoore]

Hmm, looks like our urllib3-disable-brotli patch no longer applies cleanly either 🙁

Can someone either explain to me, or point me to documentation, how to update our vendoring in a situation where the patch we carry no longer applies cleanly to the new upstream code? I'm not clear how the patches were generated in the first place, so I don't really understand how to modify them.

Alternatively, if someone else wants to handle the revendoring for the release, that would be fine with me (but I'd still suggest documenting the process, so we don't get into the same situation in the future).

Worst case scenario, I'll just not upgrade our vendored dependencies in this release. I might be able to cherry-pick the distlib and pep517 upgrades, as they did apply cleanly, but those are the only ones that happen before things start failing. I'll take the decision mid-week (probably on Thursday) whether we do this.

[pradyunsg]

I'll pick up the revendoring.

The workflow is that the patching failures result in the source code being left as-is and you're supposed to stage that, make changes manually and regenerate the patch by running git diff.

Definitely worth adding a bullet list to the docs of vendoring.

[pradyunsg]

FWIW, worst case, we could also manually do what the automation does. :)

I'll think about whether it makes sense to move the upgrading logic to vendoring while I'm at it.

[pfmoore]

The workflow is that the patching failures result in the source code being left as-is and you're supposed to stage that, make changes manually and regenerate the patch by running git diff.

Ah! I see what's confusing me here. I have a lot of files showing as changed, but when I do git diff, it looks like a lot of them are "LF will be replaced by CRLF the next time Git touches it".

But even with that, it looks like what I need to do is stage the changes, re-patch and regenerate the diff, then throw away the staged and modified files, and re-do the vendoring. Is that correct?

Oh, and just to make things even more of a PITA, when I try to regenerate the patch on Windows, I get "corrupt patch" errors from git. I'm not sure why, but the lack of an -o option on git diff probably means it's down to something about CRLF or something like that with redirection on Windows 🙁

I'll leave it to you at this point, but as a note for future RMs, if you use Windows, be prepared to either get someone on Unix to look at vendoring, or spend a chunk of time getting the vendoring automation Windows-compatible... (I don't have time to look at that myself, at the moment).

[pradyunsg]

Is that correct?

Yup.

---

I swear I remember that we'd fixed these line ending issues on Windows...

Let's move the vendoring discussions to the dedicated issue. :)

[pfmoore]

I will note here that I disagree with the practice we seem to have fallen into of doing vendoring updates as part of the release process. It leaves us with a short timescale to address any issues that might arise, as well as putting pressure on the RM to sort out any issues.

I think we should do vendoring upgrades whenever we want to, as part of the normal cycle of making PRs. Ideally, whenever a dependency gets upgraded, we could re-vendor as soon as we notice¹. And conversely, I think we should not do a vendoring update at release time - we should apply our normal "release what's on main" policy, and if nobody did a vendoring update in advance, then fine, we release what we have.

Footnotes

1. I seem to recall that in the past, we've even rejected vendoring PRs from the community, on the basis that we'll do them ourselves at release time. That seems particularly unnecessary when someone's already done the work for us... ↩

[pfmoore]

There's a somewhat major release of packaging due: https://github.com/pypa/packaging/blob/main/CHANGELOG.rst#unreleased

@pradyunsg - I'm guessing this won't make it in time for 22.3 now?

[pfmoore]

Release has now been done, and is on PyPI.

I'll work on get-pip and the announcement later today. I'm still considering how to do the CPython PR, I can obviously do one for 3.12, but I'm hesitant to backport until we've got some assurance there won't be a bugfix release.

[pfmoore]

OK, get-pip and announcement done. 🎉 🚀

/me hides under a blanket

[pfmoore]

(PS @pradyunsg git signing with 1password works, so I could use the get-pip automation and it signed the tag 🙂)

[pfmoore]

OK, so release 22.3 has been out for 2 weeks now. There have been a couple of reported bugs (which don't have merged fixes yet), but no-one screaming that anything is critical.

I'm inclined to say that we're done with 22.3, and at this point any further changes/fixes should target 23.0. Does anyone have any objections (and if so, can you please provide an issue number and a rough idea of likely timeline for anything we're waiting for)?

I guess the most likely candidate is #11527 (and the fix #11538) but there seems to be quite a bit of uncertainty as to what's affected there and how deep the problem goes. My inclination is to say that we shouldn't rush a fix, but should make sure we apply the right fix, and if that means waiting till 23.0, then there's no indication yet that doing so will be a serious issue (only one person has reported the problem so far, and that was for a niche case around copying the cache).

[uranusjr]

I would really want #11547 in so CPython can fix ensurepip in 3.11.1. Very few people are complaining because a new Python version is generally slow to be picked up, but it’d be too late if people do loudly complain because the fix will be picked up very slowly by CPython and inherently everybody using Python 3.11.

[pfmoore]

OK. I'm not sure I'd have much sympathy for people "loudly" complaining that an unnecessary pip3.10.exe is created - it seems like it would be pretty rare for a user to be running pip3.10 as their command, having both Python 3.11 and Python 3.10 on PATH with 3.11 before 3.10, and not being able to just delete the unwanted wrapper. But I agree it's annoying.

Is there anything holding up getting this merged¹?

Any others from anyone?

Footnotes

1. It's worth noting that I just upgraded my PC to Python 3.11, so it's entirely possible that even without this change, if I did a new release now the problem would disappear. That's not to say I think we should ignore the PR, just that we shouldn't treat "the new release had the right wrappers" as an indicator that the PR worked 🙂 ↩

[uranusjr]

It's worth noting that I just upgraded my PC to Python 3.11, so it's entirely possible that even without this change, if I did a new release now the problem would disappear

Without the PR it’d introduce a new problem: people upgrading pip to 22.3.1 on Python 3.10 would have pip3.11 point to Python 3.10. Much less a problem since that’s not in CPython though.

[pfmoore]

@uranusjr Gentle ping here. I really don't want to leave 22.3 hanging indefinitely. So I'm going to say that 22.3 is closed 2 weekends from now (the 12/13 Nov, 1 month after the release). If #11547 isn't merged by then, it will have to wait for 23.0. If it does get merged, I will do a 22.3.1 the weekend after it gets merged, and close the 22.3 cycle once that is done.

[uranusjr]

Feel free to merge it if you want, I'm on mobile now. Otherwise I'll merge it tomorrow.

[pfmoore]

Thanks, done. I'll do 22.3.1 this weekend, then release 22 will be done.

[pfmoore]

I'm starting the release process now, but the instructions say to "Cherry pick the fixed commits off of the main branch, fixing any conflicts"¹. I'm struggling to work out how to cherry pick #11547 - if I use git cherry-pick on the merge commit, I get "error: commit 6d13113 is a merge but no -m option was given", but if I add -m, I get "error: option `mainline' expects a number greater than zero". I don't know what the "parent number" is supposed to be.

@pradyunsg I'm pretty sure you've used this process before - how should I cherry pick a merge?

Also, just to be clear, if I'm cherry picking, #11547 is the only change I'll be adding to 22.3.1. Nothing else seems to be significant enough to be worth including.

Footnotes

1. Is this a changed process? Was it discussed anywhere? I don't recall doing cherry picks before, I thought our process was to always release from main. ↩

[pradyunsg]

You need a -m 1

I think you're remembering #8990 (comment)? It's worth adding this info to the release process, if it isn't there already.

I've not looked at the release process, but also... make sure to delete the news fragment associated with that change. I'd write a shorter letter if I had time, but I have to hop onto a flight so here's the long form.

The cherry-pick + release workflow means that we pick the changelog fragment file and then delete it within the commits in the PR. So, the overall diff of those changes is that... well, only the changelog file has changed and no changes are made within the news/ directory. But, we don't want that -- we also want to delete the news fragments that have been committed into there. So, my workflow in the past has been to merge main into the release branch after the release commit and deleting the changelog fragment file either as a part of the merge commit or, more easily, as a follow up commit.

[pradyunsg]

Is this a changed process? Was it discussed anywhere?

This has been there for multiple years -- we just typically avoided that since we didn't merge any other changes into main other than the bug fix ones. :)

[pfmoore]

we just typically avoided that since we didn't merge any other changes into main other than the bug fix ones.

That would be it, I'm good at avoiding things that make my life harder 🙂

We could probably do with documenting how to do a patch release from main. It's likely nothing more than using the "full release" process with a different version number, but I don't feel like experimenting right now, so I'll leave that for later.

[pfmoore]

CI failures on the 22.3.1 branch are because the test fixes weren't backported. The release PR didn't fail because main was merged into it, so I'm not going to worry about it.

[pradyunsg]

It's likely nothing more than using the "full release" process with a different version number, but I don't feel like experimenting right now, so I'll leave that for later.

It's exactly that, FWIW.

[pfmoore]

Cool, I'll add a note to the bugfix release section saying that you can simply use the normal version if you want to release what's on main. If only because I'll forget by the next time I need to do this 🙂

[pradyunsg]

@pfmoore If you reckon this release cycle is done, let's close this out and unpin this issue?

[pfmoore]

Sounds good to me!

[pfmoore]

There are two vendoring updates on the 22.3 milestone that didn't get completed. I'm not sure how much there is still to do on #11500 - but I guess we can close it as we'll pick up any that didn't get done in the next round of vendoring. For #11515 (rich) do you want to keep that open, or shall we close it and start again on rich?

BTW, I strongly recommend that we re-vendor as we go along from now on, rather than waiting till the release. To that end, I'll probably have a go at a vendoring update sometime in the next few weeks, just to keep on top of things.

[pradyunsg]

I closed one. I'll keep the other open -- but I won't be spending time on the vendoring story this week or the next.

I've been chipping away at the venv-based build environments change for a few weeks now -- I'd like to get that over the line and filing pull request.