-
Notifications
You must be signed in to change notification settings - Fork 240
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFC: registry:
dependency specifiers
#314
Conversation
We've already discussed this pretty much to death in the context of #217, but added the |
A new dependency specifier is added: | ||
|
||
``` | ||
registry:<registry url>#<package name>[@<specifier>] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not leveraging the current syntax? The <source>:
syntax which is already used for github and some other sources.
if there are multiple packages from the same registry, the URL will be duplicated? Maybe it should be moved out to a separate field.
For instance:
"dependencies": {
"foo": "corp:^1.0.0",
"bar": "corp:^1.0.0"
},
"registries": {
"corp": "https://registry.corp.com"
}
also, if such a package will be published to the public registry, are all the third-party registries trusted by default?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Defining a spec name to point to a registry is a good idea, and there's a number of ways we could go with various different trade-offs. Worth doing as a subsequent RFC that builds on this one, since corp:foo@1.x
would presumably desugar to registry:https://registry.corp.com#foo@1.x
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not only a matter of syntactic sugar - if we support / implement this, then we need to be sure the syntax meet our standards. If fully qualified URLs lead to subpar developer experience, then it'll be very confusing to later say "our bad, now you can use this other syntax that work better but isn't as well supported".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't really understand the objection here. If we want to support a custom registry url in the package manifest as suggested here, how is that made any more difficult by also supporting a full registry url as part of the dependency spec? In fact, it seems like it would be somewhat easier implementation-wise, because most of the code that consumes specs would be able to remain agnostic as to whether the alias spec was corp:foo@1.x
or npm:foo@1.x
or registry:https://registry.npmjs.org/#foo@1.x
, and we would have a verbose canonical way to save it that doesn't rely on having the rest of the package.json file in order to parse it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we want to support a custom registry url in the package manifest as suggested here, how is that made any more difficult by also supporting a full registry url as part of the dependency spec?
I think our point is that we're not convinced we want to support both, since that has a cost in terms of documentation, and has the potential to be confusing for our users. Should they use an url or a name? In which context? Etc. I'd much prefer having a single consistent syntax, and that makes it important to discuss what this syntax would be.
we would have a verbose canonical way to save it that doesn't rely on having the rest of the package.json file in order to parse it
Imo registry names shouldn't be mapped to urls via the package.json, but rather by our respective configuration files - just like scope urls.
I don't see a lot of benefits from this versus using direct tarball URLs. |
Users know their registry URL but don't easily know the tarball URL. |
The main advantage is that you can specify a dist-tag or SemVer range.
That too :) |
Suggestion from @wesleytodd: make sure we're crystal clear in the rfc and docs that this will break if users are on npm versions prior to its inclusion, and anyone who doesn't have access to the registry defined in the spec. |
Perhaps when npm notices a registry specifier in the current package.json, but it lacks an appropriate |
Would the official npm registry reject packages with registry specifiers? Or would it check if they are available? Or would it be worth requiring an additional confirmation aka |
The registry doesn't do anything with dependency specifiers. It would just let them on through, like it does with unrecognized specifiers yarn has introduced that npm doesn't know how to handle. |
Maybe it makes sense for the CLI to warn people then? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm a proponent of making package managers less reliant on default registries (we even discussed a fairly similar approach back whe the GitHub registry was released), so I'm happy to see npm championing a proposal in this direction 👍🌟
With that being said, I believe this RFC is missing information in the "Rationale and Alternatives" section. In particular, I'd like to see an objective rundown of the problems we may hit with this approach rather than another, why they don't matter in the grand scheme of things, and why other approaches are worse. This is important since we should not only be convinced that this move is a good thing (I already agree with that), but that there isn't a better approach.
- `<registry>` is a fully qualified URL to an npm registry, which may not | ||
contain a `hash` portion, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As @zkochan mentioned, the RFC doesn't make it clear why fully qualified URLs are the right choice. They have various drawbacks (such as hard to reconfigure; strongly vulnerable to hosts going down; syntactically ambiguous), so I think it's important to have the discussion about this before the fact, not after.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, the only reason mentioned in the RFC for using URLs seem to be this, but it could be expanded (as is, I'm not very convinced the pros outweigh the cons).
contain a `hash` portion, | ||
- `<package name>` is the (scoped or unscoped) name of the package to | ||
resolve on the registry, and | ||
- `<specifier>` is an optional dist-tag, version, or range. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- `<specifier>` is an optional dist-tag, version, or range. | |
- `<specifier>` is an optional dist-tag or semver range. |
Versions are valid ranges.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, they are. And also, every time we rely on that fact in our documentation, I get someone asking whether it has to be a range, or if a single version is allowed, so I just got in the habit of being a little extra verbose about it.
When a package is installed using a registry specifier, it *must* be saved | ||
using a registry specifier. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is another argument in favor of not making them URLs. A common complaint back in Yarn 1 was that storing the registry urls within the lockfile was causing annoying issues when switching from a registry to another - common use cases for China users, for instance, which frequently need to use Taobao.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I run npm install foo@registry:https://registry.foo.com/#foo@1.x
, and it is saved in such a way that future npm install
invocations do not fetch from https://registry.foo.com/
, then that is a clear violation of expectation and intent.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Precisely; and if you run npm install foo@registry:openjs#foo@1.x
then there's no such expectation, and users will instead assume they can swap the openjs
registry url to another if they need - which is a reasonable feature, considering that it's already an established use case (cf the Taobao example).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What this is saying is that we have to save it to the package.json
so that if you run this:
npm install foo@registry:https://registry.foo.com/#foo
rm -rf node_modules
npm install
then the second one gets it from the https://registry.foo.com
registry, and not the default configured registry.
If you agree, then I don't understand the objection.
A new dependency specifier is added: | ||
|
||
``` | ||
registry:<registry url>#<package name>[@<specifier>] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not only a matter of syntactic sugar - if we support / implement this, then we need to be sure the syntax meet our standards. If fully qualified URLs lead to subpar developer experience, then it'll be very confusing to later say "our bad, now you can use this other syntax that work better but isn't as well supported".
Tarball URLs require an exact version, don't support deprecation notices, harder to authenticate, etc. Plus, I think that semantically the very idea of fetching packages that belong to different sets, to treat the npm registry as only one of those sets, make sense. With that we could imagine separate organizations providing their own registries / sets. |
I understand, but IMHO, hardcoding the URLs in the specs makes it a bit inflexible. I think it should be decoupled somehow. The mapping of the package source to the package base URLs may be provided in a separate configuration file. So as mentioned in the RFC, when a package is in a staging environment, the mapping will provide one registry URL, in prod, it would provide a different one. Isn't this how the registries in some Linux package managers are specified? Like a list of mirrors is provided. This way you can change the registries without republishing the packages. |
Yep, on that I fully agree. |
One thing we didn't discuss in the last openRFC call, but which probably needs to be addressed, is how the dependencies of packages fetched via registry specifiers would be resolved. For example, say that I am using a package {
"dependencies": {
"foo": "registry:https://registry.a.com/#foo@1.x",
"bar": "registry:https://registry.b.com/#bar@1.x"
}
}
Two options I can think of off the top of my head (there may be more):
|
This is a good idea. It's also a different idea from the one being proposed here, and should have its own RFC so we can examine it fully ;) |
I personally don't agree it's a different RFC. It's an alternative to one part of this one, and thus should at the very least be discussed in the "Rationale and Alternatives" section, along with an explanation why you still think the original design has more value considering the problems we raised against it.
Indeed 🤔 At the moment (at least in Yarn), all packages exist in the same vaccum, so there's no "context aware ranges". This would likely change that, as dependencies of a packages would semantically make more sense to resolve against the same registry by default (but then how would it work if, say, I download the package tarball and reference it directly using the |
Updated the RFC with a fixup commit to add a bit saying that the dependencies of a package fetched via a I was tempted to say that it should fall back to the main configured registry if it's not found, but this opens the door to name hijacking attacks (including non-malicious "attacks" where something just gets unpublished, and then the wrong thing gets installed). And yeah, I also find the "context aware range" idea somewhat appealing.
It doesn't, lol. It's a huge pain and kind of makes file and tarball URLs annoying and useless, unless they expect to use the main public registry. That's part of the problem we're addressing here :) |
I don't prefer to eat the elephant in one bite, if I can help it. User-customizable registry shorthands is a pretty big feature that touches different parts of the system than this does. If we can get this well understood and implemented, and then express the registry shorthands feature in terms of registry specifiers, that's a lot easier. On the other hand, if we add it to this RFC, we're going to get bogged down in more issues. The main disadvantage raised, which that would address, is "urls are clunky". And ok, I agree. But they're also explicit and it's easy to see what's going on. npm works by layering portable UX affordances on top of clunky explicit non-portable implementations. When you The power that this RFC adds is a branch point in that process. So rather than only having one registry to resolve deps against, you can explicitly state that some deps come from specific registries. I'd rather not also add the second UX affordance of saying that some registries are named, especially since "it is annoying to have to assign names to all my registries explicitly" is part of the pushback against scope-specific registries that led us here, here in this one RFC, because that bloats the feature and makes iterative implementation more difficult. An RFC should be one thing. We add the clunky explicit thing, and then we add the UX affordance that makes it less explicit, more portable, and easier for humans. I agree that this is a valuable feature you are suggesting, and I want to implement it. That is why I am advocating steps to get there efficiently and carefully. |
I understand the desire to add the least powerful tool to solve some of the issues users are facing. The issue is though, that once npm adds some feature, it is impossible to revert that feature. So I would not rush with this RFC until the bigger picture is discussed. It can be discussed in a different RFC, no problem. I think if this was considered at an earlier stage, even scopes would have been a good fit. |
From the RFC call today:
It sounds like you aren't opposed to this RFC as written, but you'd like to see a discussion of how this might function with the sugar. Is this a fair summation? It seems to me that if folks feel an RFC for that would be premature that an RRFC might fill that gap? Is your concern solely that |
I don't mind the esthetic side at all. My worry is about urls (just like @zkochan). Specifically, what's the story we want to sell our users? Urls vs identifiers are only a "sugar" as an implementation detail. From a user perspective, what should they use, and why? This feature has a cognitive cost, and by experience multiple ways of doing the same thing tend to confuse people more than help them.
Kind of; I could get behind that (I think the feature is generally nice to have), but I'm still not convinced supporting urls on the first iteration is needed. I'm worried this complexity will be unneeded (those who would use it could just as well specify a tarball url as dependency; it would be mostly the same, except for semver ranges). That being said, I won't block on that if symbolic links are supported.
Indeed. Given that there aren't much public registries that would want this feature right now it doesn't seem too bad - and should it sprout competing registries it won't be much different from a system like apt, where you register the locations providing your packages. If well documented I don't think that would be a problem. |
This seems like a really critical question to answer.
Agreed, it seems like a feature to me. |
Perhaps a way to move forward and be sure to all be aligned would be to write the user documentation as part of the rfc (top down design), from the perspective of a reader with no prior context ? Given that documentation will be needed anyway, time taken on this won't take away implementation time. |
Yes, I think it would answer the question of whether this RFC is needed at all. Would we think about these problems 5 years ago, I think scopes could've solved these issues. |
Does the "Motivation" section not cover that? It has scenarios with problems that current tooling has no way to resolve. (If you don't feel the problems there are worth solving then that's a discussion worth having, but if they are then SOME sort of RFC is warranted.) |
I understand the problems but I see downsides to using semi-URLs. I would look into alternative solutions first. I am talking not about "sugar" but about alternative ideas. URLs are too unflexible IMO. And as a result, their usage will be limited to a very small subset of problems. |
c151191
to
4df8c0b
Compare
Moving this to accepted status, as decided in our OpenRFC meeting. No point discussing the next steps here, apart from the "Future Work" section of this RFC. It's time to move this to the next step. If we get a bit further and find a better way there, we'll take that instead. An RFC isn't a binding contract or anything. If pnpm or yarn wants to wait to see step 2 and if there's a better way to get there than the path npm takes, then that's fine. |
I think the thing being glossed over here is that pnpm and yarn want to see a different solution entirely, not a "step two" or sugar on this proposal. There's nothing about the proposed alternatives that necessitates a raw registry-url type specifier, and so there's no reason they would be dependent on this. It sounds like the next step forward, if cross-package-manager support is valued(?), would be to open an RFC that replaces this one with an alternative to discuss that. |
The objection I've seen here is "urls are inflexible, we shouldn't use urls or anything that looks like them". This objection was addressed on the following basis:
The other objection about URLs is that they confuse users. This is an assertion that requires more evidence to accept. If there's any population of people that are comfortable with URLs, it's web developers. Unless we were making software for browser makers or members of the IETF committees defining web standards, it would be hard to find a population of users more able to handle URLs without confusion. If there's an alternative that aims to address the use cases presented here, a rival RFC is indeed the way to go. It may render this one irrelevant! Or it may be additive to this, or fully satisfied by this one in a better way, and thus not optimal overall (even if it is better in some ways). If there are new objections to be brought up, which have not already been considered, by all means do so! But the "rough" in "rough consensus" means we sometimes decide to do something, even when not everyone agrees, once all the objections are explored and decided on.
There is a value in mimicking one another's features and being able to interoperate, for sure. There is also a value in us sometimes exploring in different directions. Yarn and pnpm are their own projects, and it's up to their maintainers whether they wish to copy npm's features or go a different way. npm features do not depend on yarn or pnpm implementation, any more than yarn and pnpm features depend on npm implementing them. |
I'm quite curious how you envision this feature to be used. From my perspective, it sounds like it can only end two ways:
I have my idea on which one it's gonna be, but I'm a bit saddened to see how much weight you put into constructive feedback from your peers. It's the second time you basically decide to ignore us even though we clearly try to find compromise, and I don't understand how you expect to build trust in this context (and to be clear, the very fact that @zkochan and me are here is because we each time want to give you a chance to show us you care about cooperation).
I already said it in the peer dependency PR: I absolutely don't care what you do in npm that is purely "client-side" (ie, that isn't expected to have a practical functional effect once the package is published). We have different philosophies, and that's good. But once the package is published, it ceases to be about just our package managers because it affects us all. In this regard, this rfc isn't an "npm feature", it's a "common trunk feature". It should at least require rough consensus from the stewards of all the relevant projects. And if you don't manage to have it, then perhaps it's a sign that should give you pause. Of course, perhaps you think that "this is npm, we have most of the marketshare, we'll do as we please". If that's the case, you probably should be upfront about this so that at least everyone is on the same page. Or you can lock the thread... |
We discussed this at length in our OpenRFC meetings. I will summarize here. A common requested use case for this is for users who have a private registry, and want another way to avoid name hijacking attacks without having to migrate their legacy applications to use package scopes. Either it will end up being used by public published packages, or it will not.
It is patently incorrect, and frankly insulting, to say that we have ignored your concerns. They have been discussed and evaluated, and very carefully considered, in these pull requests and in our Open RFC meetings that are open to the public, and which we encourage you to attend. It is not our responsibility to ensure that you see or agree with the resolutions to the concerns that you bring up. It is your responsibility to stay informed and be a constructive part of the decision making process if you wish to have a part in it.
I think that you misunderstand the purpose of this repository, and I encourage you to review the README.md file. This is the npm RFC process, not the "all JavaScript package managers" RFC process. npm is the final authority as to which features npm implements. If you wish to set up a space in which the maintainers of yarn, pnpm, and npm agree to block one another from implementing features until all of us agree on them, then that is a much larger conversation. But this is not that forum, and never has been. That forum does not exist. "Rough consensus" is not "full agreement". It is "all objections and issues have been thoroughly considered, and a decision has been made on each of them". If yarn decides not to implement a feature, that is of course a relevant piece of information, which we do pause to consider (often for months), but once considered and decided upon, we move on with the decision. Sometimes that decision is "do a different thing", and sometimes it is "do the thing that yarn won't do". Continuing to bring up previously addressed concerns is disruptive to this process. If you have new issues to consider, we welcome them. If not, then please behave appropriately or find another venue to complain. You have been warned before, please do not let it happen again. I am locking this thread for now, because it seems that there are no issues to bring up which have not been addressed. If anyone has new concerns to bring up, message me and I will gladly unlock it. If you have an alternative proposal that you believe would better meet the use cases that this one targets, then please start a new RFC for it. If you would like to suggest constructive changes to this RFC, please open a new PR with the proposed edits. |
Fix: #275
Close: #217
References