Update containers and actions to Python 3.11

[dekkers]

We should update all containers images (for both production and development), actions and all other places where we specify a Python version to Python 3.11.

[praseodym]

We will have to update dependencies first, for example Django only has Python 3.11 compatibility since Django 4.1 (we're at Django 3.2.18).

[ammar92]

I'm fine with this, but does that mean we're dropping support for older versions?

[Darwinkel]

I'm fine with this, but does that mean we're dropping support for older versions?

Our unofficial policy is to support Debian Stable and the last two Ubuntu LTS releases - these contain 3.8, 3.9, and 3.10 by default. I'm all for upgrading the Docker containers, but we should be very careful not to use backwards-incompatible features. The CI currently runs the unit tests against these versions which should provide a basic safeguard, but we should probably spend some time to make it more robust.

Alternatively, we could investigate the possibility of providing a universal Python 3.11 environment for Debian and Ubuntu, but that will probably increase our maintenance burden.

[praseodym]

Created #958 for the Django 4 upgrade.

[Darwinkel]

We should probably have a meeting about this sometime soon. Upgrading the containers and dependencies is relatively straightforward, but guaranteed backwards compatibility with older Python interpreters is not. That's a maintenance burden, and it would be nice if we could drop it. But I have a feeling that beheer and our .deb users will not appreciate that unless we can provide an easy to install 3.11 interpreter.

[praseodym]

Django 4.2 still supports Python 3.8, but I agree that supporting several Python versions is a maintenance burden. It'd be easier if all end users could upgrade to Python 3.11. However, recent Debian and Ubuntu versions don't have a clean way to install Python 3.11: it's not in Debian Backports, and I didn't find any official repositories.

The current Ubuntu 22.04 and upcoming Debian 12 Bookworm do have Python 3.11 packages, however. But requiring an upgrade might be a big ask.

[Darwinkel]

What about pyenv? It can build and manage Python interpreter versions on a per-project basis. We could look into a workflow or some instructions for combining it with our beheer and deb packages.

I did some testing with compiling Python & pip from scratch as well. It's reasonably easy to ship and use self-built binaries as long as we use our own virtual environment (which we currently do).

[praseodym]

I wouldn't be against it personally, but it's an entirely different direction than what is being worked on in #401 (packaging all Python requirements in separate Debian packages).

[Darwinkel]

We've also discussed pyinstaller internally, but it'll probably lead to issues with e.g. latex, django, and OpenTelemetry.

[dekkers]

I think compiling and shipping our own Python would be worse than what we do now because it would result that we also have to support that. Just compiling Python once is different from continuously supporting a self-build Python and spending time on fixing all the things that can go wrong. I also wouldn't know how to start packaging such a thing for Debian.

Supporting such a thing would at least mean making sure it works correctly (a lot of things can go wrong if you compile something yourself, that's why we have distributions and container images), make sure that upgrading to new minor and major versions of our own maintained Python would work correctly, make sure that it is updated including doing security updates for which we would not be notified beforehand (Debian, Ubuntu, etc. usually get a notifications of upcoming security releases so they can prepare). And probably more things I can't think of right now.

With regards to which distribution versions we support with our Debian packages: when we discussed it I suggested that we support the latest Ubuntu LTS and Debian releases and provide support for the previous release for 6 months so people have time to upgrade. I don't know how we ended up with the latest two Ubuntu releases in the documentation because we have never had Ubuntu 20.04 packages. With the policy I suggested we can drop support for Debian 11 / Python 3.9 in December 2023 and drop support for Ubuntu 22.04 / Python 3.10 in October 2024. I think that is pretty reasonable.

I also fail to see how supporting multiple Python versions is a big burden. It's just a matter of running the unit tests for all the Python versions we support which we already do for a big part and testing the Debian packages which we need to do anyway. Python packages support different Python versions all the time without much problems, and for Django packages it is even common that they also support multiple Django versions. I do think it is good to drop support for previous Debian/Ubuntu version after six months, but that is mostly because I don't want to support too many different versions of the Debian packages and iIt would also be nice to be able to use things introduced in newer Python versions at some point.

[Darwinkel]

I think compiling and shipping our own Python would be worse than what we do now

Yeah, It's not a clean solution for sure, and comes with a responsibility to test and patch the interpreter.

I also fail to see how supporting multiple Python versions is a big burden. It's just a matter of running the unit tests for all the Python versions we support

Is that sufficient though? If we state that we support a certain version, then that must come with (some) promise of stability. Imho just relying on the unit tests passing in the CI is fragile at best to detect potential bugs.

I suggested that we support the latest Ubuntu LTS and Debian releases and provide support for the previous release for 6 months so people have time to upgrade.

It's a decent compromise, but still limits our ability to use newer features such as data classes with slots or the free performance upgrades with 3.11 on short notice. I suppose we could use decorators or some other alternate code path to enable 3.11+ functionality in our container images instead, for users which have high-performance requirements.

[Darwinkel]

Meeting notes (23-05-2023):

• We will keep supporting 3.8 (and by extension 3.9 and 3.10) at least as long as our clients need it. Update containers and actions to Python 3.11 #627
• In the future, we will upgrade our codebase to the highest possible version (e.g. pyupgrade) to eliminate tech debt.
• We will upgrade to Django 4.2, and then upgrade the containers to 3.11. Update containers and actions to Python 3.11 #627
• We will leave the data classes optimization for later, as we have easier performance improvements we can make. Pydantic should not be used as internal data container #738
• We will implement pip-tools instead of Poetry. Unified approach to dependency management (requirements.txt) #735