CI, Repo, and Versioning change proposal for 2024 and beyond #65
Replies: 9 comments 27 replies
-
|
Re: bots. New bot. 100%. The stuff I've written is very limited in functionality. It's designed to do exactly one thing, so adding release stuff to it would get messy. This is probably a rabbit hole of it's own, but honestly we need either one bot that does everything, or an army of small purpose built bots. I personally lean towards the small purpose built bots, because I think the monolith bot makes everything far more complicated than it needs to be and stifles new features. If we do go the monolith route, imo it needs to be in a language more people actually want to work on. |
Beta Was this translation helpful? Give feedback.
-
|
What do you think about small purpose built bots in a monorepo? I definitely agree with small bits that do 1/a few things great. All bots should be treated as the same codebase I think to still keep them "monolith" in nature with regards to the codebase and the same developers being able to work on any of the bots because they are familiar with the general idea of how they are made. |
Beta Was this translation helpful? Give feedback.
-
|
I have no objections to that. IMO anything we write that's for the project like this should end up under the org eventually, so it makes sense for them to just live in one repo instead of half a dozen spread around who knows where. |
Beta Was this translation helpful? Give feedback.
-
|
I really love the ideas expressed here. For awhile now I have thought that the concept of the fabled “v11” release has (mentally?) held us back in some ways and our current major release cycles drag on forever.
They absolutely can. We already have some actions running on a cron schedule (unstable sdk updates for example). We will have to figure out some of the areas in our releases where dependencies between builds exist (we need the server to build to have an updated openapi spec to build an updated sdk to bundle in web), but I think this is manageable. |
Beta Was this translation helpful? Give feedback.
-
|
I actually at this point think the API spec is the only cross-dependency, aside from the code itself of course. In every other are the current builds are totally independent, then whenever the "second" one fires (usually server on releases) it runs all the combining stuff. |
Beta Was this translation helpful? Give feedback.
-
The proposed changes to the release process/API rewrite process is where I have the most thoughts. I'm strongly against the plan for the API rewrite to be a separate repository. As we've seen multiple times before, This is partially unrelated, but there's been some discussions on EF Core in relation to the new API, and I have a few How the API rewrite can happenUltimately, the reason the API rewrite hasn't happened is, in my opinion, because of these two restrictions:
If one of those restrictions are lifted, I think the API rewrite is perfectly doable, in one of two ways: Rewrite the API in partsIf we can break the API in parts, either by treating 10.x as semver 0.x, or releasing consecutive major versions, Once 10.9 is released, immediately start the 11.0 release cycle.The 10.8 release cycle has lasted for over 2 years. I think it's completely possible to do the API rewrite in a |
Beta Was this translation helpful? Give feedback.
-
|
I know I've suggested option 1 ("Rewrite the API in parts") several times but people don't like the idea of adding wrapper functions to handle both while that happens, so we can't really do deprecation with that option. Which can be an issue if we move fast while also doing this in master. Option 2 is good, but it requires us to commit to a much longer timescale, and I kinda hoped we'd speed that up. But if that's what we want to go with, that's doable. The only issue I'd have with it is the intervening branch handling but we can work that out. We'll definitely want to transition away from semver eventually. TBH I think it just holds us back too much, as every little change has to be considered on that axis needlessly, and then we get the whole "stable release branch" thing going on and people mis-targeting fixes and having to deal with backports/forwardports. So at the very least 11 will actually be IDK "2025.1.0" or something, we're just using "11" as a placeholder I think. Or it will go 11 -> 12 -> 13 etc. with a much faster cadence, either way. I just want to be sure we actually move towards "new API" at a reasonable pace and not have it be stuck in limbo. My idea with a separate repo was more to help enable tracking of that separately and more dedicated focus on it in its own space, rather than having to think too hard about what each change will do to the current stable/master/unstable releases. After all, unstables are functionally useless if all the clients keep breaking every release and no one can reasonably run it. With a dedicated repo, you don't have to think about that or the intervening release schedule, then when it's ready we can merge the whole thing back in. It's a one-time pain event instead of tons of little pain events in between. But I'm open to either. |
Beta Was this translation helpful? Give feedback.
-
|
Last though I missed is the local Docker builds. I think that can remain, though I'd rather not because then we're back to having to change things in two places to keep both repos in sync. But that's an open Q because just about everyone does local dev differently. |
Beta Was this translation helpful? Give feedback.
-
From what I remember, I was the one that objected to this, because the suggestion as I understood it was that we could write the new API in parts, but we would have to maintain the entire old API until we finished. I don't think it's feasible to have that level of backwards compatibility, but doing best effort of keeping old endpoints around (at least one point release, for every 10.x release) makes sense to me. Whenever we come across something that can't be done in a backwards compatible way, we can push that specific change to the final 11.0 (or 2025.x, I'm in favor of getting off of semver as soon as we're done breaking things) release. |
Beta Was this translation helpful? Give feedback.
-
|
Will the local docker builds moving break the github codespaces thing? |
Beta Was this translation helpful? Give feedback.
-
|
no i dont think so, you can run everything on codespaces it must not be docker ready |
Beta Was this translation helpful? Give feedback.
-
|
I have always thought of v11 as the release when we remove the old API. I think that (smaller) performance regressions while both APIs need to coexist is an acceptable tradeoff. I don't think we should break the old API until we're ready to throw it out. This way we can implement the new API piece by piece, field by field, while also not breaking clients with every minor release. Release semver v11 is the release when the old API is disabled. No more, no less. That is the only deciding factor of what constitutes the v11.0.0 release. I'm strongly against switching away from semver, as it is important to signal when things are likely to break (in a big way) for dependents – in our case that is mainly the client API spec, but we should also think of plugins in the future, tho those are more likely to break even with patch releases currently. There is nothing preventing us from following the proposed release cadence with the current semver versioning, except for just deciding to do it. We're, as I see it, currently held back by "this and this and this feature needs to get included in the release – oh, and this one too!". We need to become better at simply "if it ain't good enough for inclusion withing this timeframe, it goes in next release." Handling what is a major vs minor vs patch release is easily done with PR tags, then the version becomes whatever the version needs to be. Here, releasedrafter decided that the version needed to be 1.0.0, due to the python min version bump being tagged with I think the packaging change would be fine, even for people using docker for dev, as they'd just clone the packaging repo (with submodules) and build it that way – we should keep this usecase in mind. With a central packaging repo, we could set up CI to push release whenever we merge a specific PR, or when a release draft is promoted to full release status. TLDR; I think we're over-thinking the versioning, and just holding off on bumping minor for too long, we can do the proposed cadence of every couple of months with the current semver version schema, we just won't bump major version only because most of the earth launched some gunpowder into the air. |
Beta Was this translation helpful? Give feedback.
-
|
If we are being honest though we have had breaking api changes in every minor and even some patch releases… so we already aren’t complying with semver really |
Beta Was this translation helpful? Give feedback.
-
|
Yep, that's exactly my thinking. We're already barely respecting semver, because in practice such a large project just can't. It's too finicky and arbitrary. |
Beta Was this translation helpful? Give feedback.
-
|
Well, I would say there is a difference between (accidentally) changing the case of enums, and whole api calls disappearing and getting replaced with something that might not do things in a similar enough way, forcing rewrites of larger blocks of logic – sanitizing input is easier (still annoying, but easier). If we as a team decide it's easier to do it in stages, and break compat / bump major version multiple times, that is also fine I suppose (API rewrite doesn't fully happen for v11, but happens over v11-v18 for example) |
Beta Was this translation helpful? Give feedback.
-
|
I mean that's just a policy decision. Semver "enforces" that only in as much as the team respects it, which would be the case with a policy-based plan as well. "X releases for deprecation of an endpoint" or something like that. |
Beta Was this translation helpful? Give feedback.
-
|
My main concern here is testing. I don't run unstable as my primary instance, and there's periods where I don't have any dev time for weeks at a time. None of us found the enum case issue, that was a user report. And then it took another month to actually get fixed in the server. That's a long time for half of our clients to be broken, or a lot of extra burden on client devs to push out updates in a hurry. The only way I see this being viable is to have multiple tiers of releases. I'm mostly speaking for docker tags here, but it could likely apply to other packages as well.
I'm sure there's still going to be changes that slip through that will piss client devs off, but at least this gives us some chance of catching them before they go mainstream. |
Beta Was this translation helpful? Give feedback.
-
|
I like the packaging changes but do have some concerns regarding the proposed move away from semver and release branches. |
Beta Was this translation helpful? Give feedback.
-
|
I don't necessarily think it would, at least not with some strict policies on it:
This would enforce a "don't break master" which is something we already generally try to do, and has been enabled more with the tests and PR builds, but if things do slip through the cracks we can always skip a week, that's my thinking. Really, the release branches is what I want to get away from. They're just cumbersome and will become more so if we move faster. |
Beta Was this translation helpful? Give feedback.
-
|
The issue here is that this forces all backend devs to either do work on the clients or force.client devs to update their code within days when breaking changes are introduced. And this is not a trivial additional burden, I'd argue. |
Beta Was this translation helpful? Give feedback.
-
|
I strongly disagree that the packages can't be splitted. In my opinion, every repo should have a The schema on top of my head is: CI builds on every repo Docker and deb images and dispatches another job in the metapackages repo when it's finished. That other jub simply pulls the pushed Docker image (or workflow artifact) and builds the metapackage. In fact, workflows recently improved support for shared artifacts between workflows Back in the day I remember the major blocker was all the NuGet publishing stuff, which I couldn't figure out properly. This time, I'm going to try going more piece by piece, so when all user-packages are in CI I can move the rest with the help of server people. As a side note, I had planned to work on Christmas again in CI, but found out I'm not an owner of sandbox. Can I get those perms given so I can do a PoC without restrictions @joshuaboniface? |
Beta Was this translation helpful? Give feedback.
-
|
What I'm worried about @ferferga is that this is what we have now, and it requires the 3 step builds (build server, build web, combine in metapackages) which is fragile. I don't see why we shouldn't just build server and web together in the other repo (whatever it's called). |
Beta Was this translation helpful? Give feedback.
-
|
Would it make sense to have the basic build infra to live in their respective repositories and symlink or otherwise use them in the meta repo to allow them to be built separately or together? |
Beta Was this translation helpful? Give feedback.
-
|
I think it would make sense to have the scripts mostly in their respective repos, but to have it executed in the packaging repo's CI. This way, PR CI can utilize the build scripts without much extra, and we'd be able to make the release pipeline less fragile. |
Beta Was this translation helpful? Give feedback.
-
|
@joshuaboniface That's the problem of the current infraestructure, that it's fragile, but that doesn't mean that "splitting builds" is a fragile architecture. They're not correlated. The problem it's in the implementation and that all the build process of server and web is purposefully made to make our build process "happy" instead of adapting the build process to the particularities (which funny enough, is part of why this transition is being so difficult). Last time I checked I recall there were some "hacks" in the building process of web which were not necessary. There are a lot of differences in how JavaScript is checked To add more salt into the wound, although all the scripts are open source, most of the process happen in our servers, and there's no way for anyone beside Core to audit the process in the real environment. There's also no safeguard against race conditions or concurrency iirc (for example, if build A starts before build B, but build B finishes before build A, the "latest" published would be build B). I think you guys understimate how powerful GitHub Actions is for things like that (I even amaze myself sometimes!) Most open-source repos who have a feature-complete and scalable CI pipeline with multiple packages (lsio it's the nearest example) have all their build process completely separated but their deployment is completely automated, we're not reinventing the wheel.
With keeping them tied together, we're shooting ourselves at the foot again, since in the future we're not going to scale well and we're going to be in the same situation! (Of course not every possible combination will be possible rightaway from the beginning, but what we want to achieve is that with the minimum set of changes needed) @barronpm That's the idea but without the symlink: I'm still thinking about some details, but the idea is for every repo send an dispatch to the meta repo, with the meta repo pulling the artifacts and joining them together. But no hard links between those, every repo will still be able build and push packages themselves but ofc, if the symlink with "meta" is removed from the equation (because we manually stop it, or something else), there won't be combined packages. But for example, if the tokens for meta repo are missing, server repo will still be able to push packages. |
Beta Was this translation helpful? Give feedback.
-
I'd advise over-engineering the ci to try to accommodate anything and everything we might want to do at some point, in favor of making it as simple and reliable as possible. That being said, I think it's safe to say we've come to an agreement here: pull meta packaging out of server into a |
Beta Was this translation helpful? Give feedback.
-
|
There are a couple additional points that have been on my mind on this topic:
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
OK this is my first main official update. Thank you everyone for the feedback, I have taken as much as possible into account. First, the things that is definitely off the table: non-Semver. That's fine. For everything, else, I've been a busy bee over the last week and a bit building the new CI infrastructure. Sorry @ferferga I just sort of started from scratch :-) The actual building part, including all the combined package builds and GitHub Actions CI is all under here: https://github.com/jellyfin-sandbox/jellyfin-packaging and the README explains most (I'm sure I forgot a few...) of the design decisions I made and a quick "why". I've also created a fresh new repository master VM to send these to, both to avoid messing up the other one and to give a clean slate for the new CI. The next step is a bot I have planned for this, to automate the remaining release drudgery. In short the bot will handle changelog generation, release tagging on the server and web repos, triggering the build CI, and publishing a release blog/forum/reddit(maybe) post and links to Matrix, Discord, and anywhere else we may want. Thus a release will be as simple as I plan to promote/transfer that repo to the main system and shut down the existing server/web CI in a couple weeks (after I complete the bot), and then we can immediately activate the new CI pushing to the new repo server and update everything else to push there as well. This will require some coordination PRs to update CI in various other repos with a planed archive layout that I'm still formulating, but will post here when I do. Part of this will also be the move as I originally proposed to a "weekly" unstable model. "Every-PR" and even, I think, "daily", just moves far too fast for users to keep up with, while weekly provides a great balance between keeping it moving, but also being stable (in a changing sense) enough for people to actually reasonably test with it and report bugs. There's one thing I want to figure out about this (getting the build ID in the UI somehow for bug reporting purposes) but otherwise, this is on track. Once that's all in place, I'll put out a blog post detailing all the changes in prep for 10.9.0 and basically make that the call to start testing with the nightlies for 10.9.0's actual release. I'm planning on us using these "weeklies" as the beta/RCs for 10.9.0. Basically we do a roughly month-long (3-6 "weeklies") feature freeze where it's just bugfixing + translations, and once it seems good, we tag the release. Trying to do "pre-release" tagged versions over-complicates everything, so I don't want to do it; the goal with the weeklies is to be as close as possible to valid "betas" and then just go from there. That is currently the plan I've laid out, so it's what I plan to proceed with. If there are any major objections not covered below, I would like to hear them, but know that most of this is already on-track and not just theoretical. In terms of the implementation of all of this, both @nielsvanvelzen and @thornbill have brought up interesting complications regarding the SDKs they build from server. Bill's is especially cumbersome since it introduces a catch-22 cross dependency between server and web builds. Our proposed solution is building only the weekly "unstable" versions of the Typescript SDK, and once we tag the release, the version it's using is the version for the stable release. ZERO API CHANGES in server for point releases going forward; if we need an API change after that point, it becomes a new major (i.e. 10.10) full stop. I think we're both on board with this; @nielsvanvelzen would something like that work for the other SDK(s)? This whole thing will also enable significantly faster release speed which I know everyone wants. The weeklies will be acting as rolling testing, our triage team is hot on top of the issues, so from that point on, it's really just a matter of "how good does master look? good enough to cut a release? go." for future "major" versions. I think that makes something like quarterly releases totally doable, and maybe even more frequently as time goes on. Over time minor versions will become reserved pretty much only for the most serious bugs or security vulnerabilities that wouldn't justify a major release. That about wraps it up. I'll update my progress on the Bot, the repo switchover, and everything else over the next couple weeks. |
Beta Was this translation helpful? Give feedback.
-
|
After looking over my workflows and how the SDK works with versioning I believe it can all work with just unstable builds. It would be max 1 per week because updating and publishing the SDK is a manual task which I don't have time for to do everyday. And there is no way to automate it because there is no difference between unstable builds and "prerelease" unstable builds. However, I still think we should have named pre-releases (as in "10.9.0-rc.1" etc.). This is much clearer in communication:
Additionally, we could use those pre-releases as stability indication. For example: "RC1: No guarantees for plugin ABI & HTTP API", "RC2: Stable HTTP API", "RC3: Stable plugin ABI/HTTP API". I've done this myself previously, once rc1 is releases I'll update my own Jellyfin instance and go through all API changes to find any breaking changes and fix/report them so the second release candidate is "API stable" I think some kind of hybrid form would be best, keep the unstable builds running every PR/daily and then manually tag one of them as RC. Those release candidates are then our recommended pre-release and those will be listed on the downloads page of the website / announced on matrix / built with the tray apps / etc. This effectively creates 3 release channels: unstable, prerelease and stable. |
Beta Was this translation helpful? Give feedback.
-
|
See this sort of immense complexity is precisely what I want to avoid. Keeping track of all those tags, what is in which tag, what bugs are fixed where, what is backported and not, etc. is an absolute nightmare, and to put it as bluntly as possible... I'm not doing it again like we did for 10.8.0. I don't care how "convincing" it is, our weeklies are the testing releases, no tags, no branches, until the actual release. That's my hard line in the sand on this. Yes, it needs a lot of testing I agree, but if someone isn't going to test the prerelease stuff because "unstable 2024031305" is a scarier name than "10.9.0-rc1" or "10.9.0-beta1", then, is anything of value really lost? BTW the whole release/prerelease/unstable is what we had for 10.8.0 and, yea, that burned me out of wanting to do a release for nearly two years (and completely under the old CI) so, yea. Plus, the idea here is, longer term, to completely eliminate the need for this sort of pre-release stuff altogether. If people are just regularly running the unstable train, then they're acting as bugtesters for everything that happens going forward, and cutting releases is then not a long scary endeavour with a large "testing window" required beforehand, because the people running the weeklies will have already been testing these things. Sure, 10.9.0 is a bit of an exception to that given how long it's been, but I truly don't think that few tags are worth the complexity that adds to everything around the build and such. Really, all that is to say, that I just.. don't see the value in such complexity because all it did was make my life hell in 10.8.0 and then we still had to implement API-breaking changes in 10.8.1 to fix bugs. So, if I come off as very sour about it, it's because I am, not you personally Niels. I'm just trying to come up with a better way to do it, and keep it scalable. |
Beta Was this translation helpful? Give feedback.
-
As we draw towards 10.9.0 I do want to resume our CI planning.
While I'm currently seeing no lingering major issues with the current CI (most if not all of the bugs have been worked out), it's still a convoluted mess, so I still see benefits in switching it all over to GH Actions.
First, I want to see if we can recap everything that @ferferga (and K3rnelPanic, but he left the org) did in repo, what works and what doesn't work. I don't know if we ever got the cross-repo stuff working, did we?
Next I want to get a few "realities" out there.
I don't think decoupling server and web versions is going to happen, ever. They're just too dependent on each other. It was a nice idea that's been floated many times, and this was part of why some of the CI works how it does, but really I don't think it's going to happen or be valuable anytime soon and thus my ideas below ignore and in some ways prevent that from happening.
A massive API rewrite isn't really happening. I know everyone wants it, but I just don't see it happening anytime soon at our current pace of development. And if it did, it would really have to be a separate
jellyfin-nextrepo that can move at its own pace until its ready.Our current CI continues to "work". Though it's cumbersome, if the consensus is indeed that no one wants to change anything, I think that's still viable. The benefits of an updated system however are that we could speed up releases dramatically.
I don't think Semver is really serving us. Too much stuff takes too long to land in a full release, and our bugfixes break things anyways. So really, why bother much longer?
From those, I've had some showerthoughts about ways to simplify the packaging, but I want to get the dev community's feedback on the proposed changes because some of them majorly affect how testing Jellyfin will work.
Separate all packaging into a single repo. First I don't think there's much benefit to having separate repos for Debian, Fedora, Docker, etc. aside from added confusion. This repo can then use submodules to track release code, which I think is a relatively good solution to keeping everything coordinated and providing the code there.
This is the one that's going to likely be the most controversial. IF we do this, there should be nothing in the main repos that does actual building - they'd be
dotnet publish/npm buildonly to get artifacts. But I think this would simplify the CI/packaging a lot. For instance these repos would not need to track tags or branches the same way as the main repos, they could be a lot more dynamic, and you just tell them to build a version of the code and it does. See below for a more detailed idea of how this particular item would work.Remove "unstable" builds and move towards a nightly or, ideally, weekly unstable build. This will reduce a lot of CI burden and complexity as these can just be timed. The question is if GH Actions can do time-based builds, but we can see.
Moving away from semver "early" (i.e. before 11). At this point I've personally accepted that a massive API rewrite is not going to happen anytime soon (see above). We might be waiting for 11 for another 5 years. I'd like to propose then a move away from semver sooner rather than later; maybe not with 10.9.0 but definitely the next major release. My main reason for wanting this is the massive complexity of how we handle release branches now, and the fact that I don't believe there are any good solutions to this beyond simply ditching semver and making master the only release branch. This will let us move faster and get features, translations, etc. in much quicker while still giving us room for quick bugfix releases if desired.
I don't know for sure what versioning structure we'd use for this necessarily, but coupled with the next point something date-based wouldn't be bad. I do like HomeAssistant's
<Year>.<Month>.<Point>non-semver versioning scheme personally.With 2 and 3, move towards a "promotion" model for stable releases. By that I mean, we build our weekly unstables. Then, if we find a particular weekly (maybe one every month or two months) is good for general release, we simply "promote" it to a tagged version and rebuild it as a stable release. This gives us both more extensive testing of releases (as anyone using the unstable will have tested it) but also lets us keep moving fast on both features, bugfixes, and translations. If big bugs come up, we just release point versions after that, which would indeed include features as well, but they'd also be promotions after at least a week of testing.
More automation for releases using @mcarlton00's, @sparky8251's, or a new bot, to automate the promotion and release creation stuff.
More details about how #1 will work in my vision:
jellyfinandjellyfin-web.jellyfin-packaging.jellyfinandjellyfin-webadded as submodules tojellyfin-packaging.preparescript injellyfin-packagingtakes a desired version/tag, commit ref, etc. for both repos and automatically checks out the proper versions.jellyfin-packagingrepo has Debian, Fedora, Docker, and standalone (archive) build elements designed to build in the submodules and collect the resulting artifacts together into a single package, without the intervening "combining" aspects we currently do. So a build injellyfin-packagingfor Debian would simply create all of thejellyfin-server,jellyfin-web,jellyfin-vue(eventually), etc. packages in one shot..Anyways feedback definitely requested as this will shape how this moves forward.
Beta Was this translation helpful? Give feedback.
All reactions