Introduce epoch support on packages

[Digitalone1]

Have you checked for existing feature requests?

• Completed

Summary

In the current state a publisher is able to notify a new update of their package publishing a new release with a semver higher than the previous version.

This is because it's the usual behavior when you publish a new version, but also because Pulsar won't install the new version if the reported semver is not higher than the current one installed on the system. Indeed we designed the backend to strictly allow a new update only when an higher semver is provided.

Even if this works and it's coherent, there might be some edge cases where an author makes a mistake and they want to rollback to a previous version as the "latest". We had an author publishing a 1.0.0 version that regret and wanted to have a 0.1.3 as the latest version.

How can this mistake be repaired?

• A first way is to delete the latest version, but this is not working in all the cases and I'll explain why further ahead. The case of this author was very tricky because all the versions were higher than the one they wanted, so they couldn't delete all the versions.
• So another solution is to delete the whole package, but this is too harsh and maybe not what we want. We are not even sure we want a package to be republished with the same name after the deletion.

Anyway in the current state there are some cases where it's not possible to correct this mistake. So I though of a solution and have been inspired by the package manager in Arch Linux. It works somehow in the same way. The database is updated, then it's checked to look the installed packages and if some of those have an higher semver, then they are downloaded and installed.

But also on Arch mistakes can happens (they made once here). So how do they make a workaround? With epochs.

In the default state a package has a 0 epoch and the progress of the versions is the usual

0.0.1 < 0.1.0 < 1.0.0 < 1.2.0

But when the epoch is increased, all their versions are higher then the previous:

0:0.0.1 < 0:0.1.0 < 0:1.0.0 < 0:1.2.0 < 1:0.1.0 < 1:0.2.0 < 1:1.2.0

So if we allow an author to increase the epoch, they can repair the mistake. But why am I talking here since this is mainly a topic related to the beckend? Because the support for this should be enabled in Pulsar to be effective even when it will be implemented on the backend.

You might wonder: Is this really necessary? Couldn't it just be resolved with a single version deletion or the total package deletion and republication. Well, not really. As I said before, we are not sure we want a package to be republished with the same name (@Spiker985 talker on discord about avoiding "supply chain attacks"), but there's also another issue with the single version deletion. Let's make an example:

I make a new version of a package which has a big security issue, but I don't know. It was on 1.4.0 and I now published the 2.0.0 as the latest. Someone luckily discover the security issue after one week and tell my on the GH repo. I want to return to the previous version which does not have all the new features, but at least it's safe. Unfortunately the issue would take time to fix, so I'd like everyone to return to use the previous one, so I'm reassured they are safe. How can I do that?

I could delete the 2.0.0. As we designed the beckend, when I delete it, the previous 1.4.0 becomes the latest version. So did I resolve the issue? Nope, Pulsar won't make the update because it sees 1.4.0 which is compared to 2.0.0 and that's not higher, so it's not installed.

Another way could be updating to 2.0.1 which yes, it works and pushes the update to everyone, but this is misleading because I'm not shipping all the new features that I wanted in the 2.0 version of the package (basically I'm doing a downgrade even if I increased the semver).

That's why the epoch is a good solution here. I publish a new version specifying the 1.4.1 semver, but with the epoch incremented. This translates in going from 0:2.0.0 to 1:1.4.1. If Pulsar has support to this, it checks for the epoch and the semver:

• if the epoch of the current installed version is lower than the latest version reported by the API response, just upgrade;
• otherwise, if the epochs are equals, just do the comparison as it has always been done, so upgrade only if the API response is reporting an higher semver.

This way I could let them upgrade to the 1:1.4.1 safe version, then when I fixed the issue, I publish the new 1:2.0.0.

What benefits does this feature provide?

As explained above. Be aware that if we introduce this system, we won't let to increase the epoch on higher semver, so nobody will abuse of this feature.

Any alternatives?

I'm open to alternatives. Share you ideas. @confused-Techie @mauricioszabo

If we go on this route, we should do some refactoring on the beckend and the database, but it's first necessary that Pulsar implements the support for it.

Other examples:

No response

[Digitalone1]

Okay, only now I realized this should have gone in ppm...

[mauricioszabo]

Honestly, I'm not sure this is a good idea. In fact, I'm strongly against it.

One thing I feel we can do is to add some status to a version. One could tag a version as "broken" or "insecure" and then Pulsar could suggest to downgrade (if a newer version doesn't exist) or upgrade (if a newer version exists).

Support for epoch can break lots of assumptions, and can make package publishing buggy - like, one publishes a newer version but that never gets promoted because it's a lower "epoch". It also is misleading to the user that installed the package (why my 2.0 package is upgrading to 1.2?) and basically makes all comparisons more complicated.

We also will need to add APIs to update "epochs" and to query for them; maybe some UI so the package author can check the versioning order; maybe even a change in our current UI to show epochs; it will need also a change in settings view to somehow sort this, etc...

[Digitalone1]

@mauricioszabo If we don't want to break things, the only way is to remain at the current state, but the problem is not solved. Also tagging the version needs to update pulsar to show that broken/insecure status.

And that system does not solve the issue because if an author wants all users to downgrade to a specific version, only suggesting it to the user is not a solution, not everyone will follow the recommendation.

I don't understand the point of making the comparison complicated. What is complicate it's the semver comparison which we always are managing with a specific algorithm on the backend and generated columns on the database. Adding support for epoch is not so complicate as we have to make a regex like we already did.

Last point, publishing with a lower epoch cannot happen because we would have a specific endpoint to increase the epoch (which is a positive number starting from zero) and checks could be done to avoid the increase of the epoch if the version is higher than the latest (so we won't have people abusing of this system).

I understand this will need changes, but which solution does not imply changes?

[Digitalone1]

This is the regex to support epochs:

https://regex101.com/r/YzIh85/2

And for how we manage the semver on the backend

https://github.com/pulsar-edit/package-backend/blob/main/src/utils.js#L369

We only need to increase the array indexes and consider the 0 epoch if the index 1 is not set.

if (array.length !== 5) {
  return null;
} 

if (!array[1]) {
  array[1] = 0;
} 

return array.slice(1, 4);

[mauricioszabo]

And that system does not solve the issue because if an author wants all users to downgrade to a specific version, only suggesting it to the user is not a solution, not everyone will follow the recommendation.

Which is totally fine. Pulsar is a hackable text editor, meaning that the user should have full control of their editor. No package author should have power to "force" someone to downgrade or update a package.

Sure, adding a status will need changes on settings-view, but honestly I feel we want to change package installation anyway - currently, there are a lot of things that we can do in ppm that we can't do in settings view, and there are a lot of deprecated and old things that we may want to remove.

As for my other points - adding epoch generates a confusion on which version is considered the "latest". I prefer something that says we have a "recommended" version - latest by default, but we can add more things with this status - beta or alpha channels for packages (the user can decide that latest with status beta is the recommended), detection of buggy or insecure versions (the user can say "this security issue is not a problem for me because I don't use this specific part of the package, but there are newer features that I need so I'll keep it), and lots of other interesting things.

It can also explain better why a package is asking for downgrade - like, we can force that, to tag something as "buggy" or "insecure" the author needs an additional field explaining the issue.

And finally, it doesn't affect future packages. One can tag version 2.0 as "insecure" then publish a 2.0.1, and we basically don't need to do anything else, whereas if we add support for epoch, one would need to tag 1.9.1 as "epoch 1" then 2.0.1 will also need to have "epoch 1". We can either ask the author to manually tag the epoch, which is bad because the author can forget what is the latest number and make the new version an older one, or it can be automatic which mean that the backend will need to keep track of the latest "epoch"...

[Digitalone1]

Okay, it's fine with the "tagging".

The solution with epochs is more an easy way to handle the correction of the versions by the author if they make a mistake.

If I am on 0.1.0 and publish a 1.0.0 by mistake, how can I go back? But this is not a big issue. To me staying on 0 or 1 changes nothing as long as the packages works.

No package author should have power to "force" someone to downgrade or update a package.

Nice, but the package manager is automatically upgrading to new upper versions, right? At least if it's set to do so, because newer versions are supposed to be better in most of the cases.

And if I set it this way, I don't see anything wrong in going back automatically if the author thinks so in the case there's a serious issue, especially about the security.

In fact this is most related to security issues, but I don't even know if packages in pulsar can harm the system in a bad way. Maybe it's an overstate of mine saying a package on pulsar can break the overall system.

And, in any case, the issue can still be solved upgrading to an upper semver. So if the epochs are too complex to implement, feel free to close.

[mauricioszabo]

There's another idea that came to me while discussing this - we can, for now, change the way Pulsar handles unpublished packages. Meaning that if an user have a specific version installed, and that version is unpublished, we can trigger a downgrade. This is probably the least disruptive for now, I'll check how to implement this.

Nice, but the package manager is automatically upgrading to new upper versions, right?

I don't think so. I never changed any config regarding this, and Pulsar/Atom never upgrades automatically - it always asks.

I don't even know if packages in pulsar can harm the system in a bad way.

Oh yeah, they can... packages on Pulsar can access anything that node.js can. It's one of the downsides to have a "hackable text editor" - they behave the same as VIM and Emacs, too, where it's up to the user to install packages that will not harm the system in any way.

feel free to close.

I'll actually first create a new ticket to track how to fix this issue the best we can. You actually helped me brainstorm this idea of "tagging" - I always wanted a "beta channel" for packages in Atom, but could not imagine how to make one - so, if more people agree with the idea it'll solve both problems at once 👍

[confused-Techie]

Yeah @Digitalone1 and I were the ones to originally talk about possibly using epoch support, but I'd be happy to work with this tagging system to allow packages to communicate more clearly with Pulsar about what version is best.

And allowing notifications to say a downgrade is recommended for a package for whatever reason is awesome to me, and if the user has enabled any automatic updates, then a downgrade automatically would make sense.

But do agree that it shouldn't be able to be done by only the package maintainer unless it's enabled in a users settings.

So feel free to brainstorm about this further, and if needed we can determine what new endpoint is best

[Digitalone1]

This won't be introduced, so I'd say it can be closed. 👍