Reconsider approach to supported dependency versions

[callahad]

Right now Synapse maintains very lenient lower bounds for versions of its dependencies (e.g., Twisted>=18.9.0, which was released more than three years ago, or pillow>=4.3.0 from 2017.)

We do this to ease packaging Synapse for Linux / BSD distributions, which maintain their own snapshots of the Python ecosystem.

This locks us several years behind the state of the art as we can only avail of libraries once they've made their way into the oldest of the distributions we support.

Take Debian: Version 11 (Bullseye) was released in August of 2021. Under current practices, we will drop support for it in October 2025 when Python 3.9 reaches EOL. This means we could be stuck waiting four years to use libraries released in the past three months (excepting backports, etc.)

We're ceding competitiveness and velocity for audience reach. But how big is that audience, really? And are we receiving value commensurate with that cost, or do we need to rethink our support baselines?

This does not need a decision now, but I'd like to see us come up with an intentional strategy around distro / dependency support some time in 2022 so we can get away from needing to discuss every proposed version bump. We should be able to define our baselines and sweep forward accordingly.

[callahad]

The status quo is strictly for the benefit of folks who want to install packages that use their distribution's snapshot / fork of the Python package ecosystem for dependencies, rather than going to upstream PyPI.

For contrast, imagine Synapse as a proprietary, SaaS offering: we'd be pinning the bejesus out of our dependencies, updating versions without a second thought, etc. There would be virtually zero overhead or heartache associated with any of this.

So those are the two extremes. We're already at one end of the spectrum. The right balance is likely closer to the middle.

[callahad]

To wit: anyone installing from source into their own virtualenv can pull in whatever dependencies they need, at any version, straight from PyPI. Our Debian packages and docker images also bundle their own dependencies, independent of the host distribution.

Is that enough? How far back from that do we need to walk from that to get most of the value while minimizing lag and overhead? What does other software do?

[callahad]

A point of comparison: Zulip is open source, written in Python, competitive with Matrix, and not packaged by any distribution. They offer a Docker image or assume they can totally take over a recent Debian or Ubuntu system. They use pip-tools to completely lock their Python dependencies.

And not that GitHub stats necessarily mean anything, but their server implementation has 5x the forks, 2x the stars, and 2x the contributors compared to Synapse.

[callahad]

Notable: Fedora 34 (April 2021) adopted the Stop Shipping Individual Nodejs Library Packages proposal:

This change is to make it default to bundle the nodejs libraries with the package that needs them, and retire the vast majority of nodejs library packages.

Something similar occurring for Python packages would solve most of our challenges, assuming adoption throughout major distributions.

[richvdh]

Take Debian: Version 11 (Bullseye) was released in August of 2021. Under current practices, we will drop support for it in October 2025 when Python 3.9 reaches EOL. This means we're stuck waiting at least four years to use any libraries released in the past three months.

This is not correct. Synapse is not packaged in stock bullseye; rather it is in bullseye-backports. What that means is that we are limited to library versions which can be relatively easily backported to bullseye, which is a much lower bar than "is in bullseye itself"; and even that latter bar isn't quite as immovable as you imply (python libraries are updated in Debian stable all the time).

Your point is valid in that there is a cost to making Synapse packageable for distributions - but when you say things like "we're stuck waiting at least four years" it makes me think that you are wildly overestimating that cost. I am just keen that we don't end up making a decision based on a misunderstanding about how much of a problem this really is for us.

[callahad]

we are limited to library versions which can be relatively easily backported to bullseye, which is a much lower bar than "is in bullseye itself"; and even that latter bar isn't quite as immovable as you imply (python libraries are updated in Debian stable all the time). [...] you are wildly overestimating that cost.

Point well taken. Slightly softened language above. It's been nearly 15 years since I looked into becoming a Debian Developer, and 4 years since I actively used Debian on any of my boxes. So my sense of what's acceptable into each archive may be quite out of date.

Useful test cases might be:

• When could we use psycopg3 (initial release: October 2021) in Synapse?
• What's the minimum version of Twisted that Synapse needs to support right now?

It would be good if we got to a place where we could answer those quickly and objectively.

[callahad]

Another note on the Debian lifecycle: The Debian LTS project means each stable release gets at least five years of support... but stable releases tend to come out every two years and nominally go EOL one year after their successor releases (the LTS project then picks them up to fill out the rest of the five year lifespan)

[callahad]

As a strawman, we could draw a line in the sand at: "We support dependency versions sourced from the Debian stable channel as a lower bound." Possibly add a grace period for the first month or two of a stable to oldstable transition... but anyone sourcing from Debian is probably also using the backports channel, so this could be a relatively brief transition.

[richvdh]

"We support dependency versions sourced from the Debian stable channel as a lower bound."

Perhaps I misunderstand you, but I think this is going to be too tight a bound. As you said yourself, that would mean we would have to wait four years to be able to use new library features.

Edit: that's not quite correct. But it is typically over two years between Debian stable releases, so even so that's an alarmingly long time in the python ecosystem.

[callahad]

it is typically over two years between Debian stable releases, so even so that's an alarmingly long time in the python ecosystem.

I would love to have an even shorter time horizon. How close can we get to zero delay between upstream release of a dependency and setting that version as a lower bound?

I agree that waiting two years for Debian Stable is alarmingly long... but our current bounds exceed even that. We set Twisted>=18.9.0 and pillow>=4.3.0 while Debian has 20.3.0 and 8.1.2 respectively. Ubuntu Focal (20.04 LTS) is slightly older than Debian Stable, and is still on 18.9.0 and 7.0.0. Is that a problem?

[richvdh]

We set Twisted>=18.9.0 and pillow>=4.3.0 while Debian has 20.3.0 and 8.1.2 respectively.

Sounds like we'd probably be fine bumping these dependencies then - but if there's nothing that's particularly useful to us in them, we shouldn't do so just for the sake of it.

Ubuntu Focal (20.04 LTS) is slightly older than Debian Stable, and is still on 18.9.0 and 7.0.0. Is that a problem?

Not really. We don't have anyone attempting to maintain packages for Ubuntu (other than our own, which as you know vendor the dependencies).

[richvdh]

Related: #13315

[richvdh]

Rereading this issue today, I think the TLDR of the actual problem is: "by supporting older library versions, we are inhibited from using the latest features in those libraries".

Whilst there is some truth to that statement, I think it overstates the situation: if there is a sufficiently attractive feature in a new version of a library, we can and should update to it and leave the packagers to unpick it.

My inclination is to close this issue. We may wish to open others with a clearer problem statement (of which I consider #13315 am example).

[richvdh]

closing, per the above