Remove hostnames from the lockfiles #5892

arcanis · 2018-05-26T10:59:21Z

We want to remove hostnames from the lockfiles:

wrappy@1:
  version "1.0.2"
  resolved "https://registry.yarnpkg.com/wrappy/-/wrappy-1.0.2.tgz#b5243d8f3ec1aa35f1364605bc0d1036e30ab69f"

Would become:

wrappy@1:
  version "1.0.2"
  resolved "/wrappy/-/wrappy-1.0.2.tgz#b5243d8f3ec1aa35f1364605bc0d1036e30ab69f"

This will allow us to switch the default registry more easily if we need to (for example if we want to deprecate our mirror, or if npm suddenly disappear from the surface of the earth), and will also make it easier for the users to switch from a registry to another.

The text was updated successfully, but these errors were encountered:

hutson · 2018-05-26T14:32:34Z

@arcanis this idea is being tracked in the following RFC - yarnpkg/rfcs#64

Daniel15 · 2018-05-26T19:11:28Z

What about custom registries/repositories? If a project has a mix of different registries (eg. some from regular public npm, some from a private internal server), how would it know which is which? Particularly since there could be overlap in package names.

arcanis · 2018-05-26T19:15:40Z

@Daniel15 I think in this case it is recommended to keep the packages coming from a private internal server behind a specific scope, so that you can use scope-specific registry urls.

dballance · 2018-08-21T19:37:52Z

Adding a datapoint here for @arcanis.

At least for VSTS/TFS, this pattern seems to work.

My yarn.lock matches the pattern for upstream packages:

typescript@~2.8.3:
  version "2.8.3"
  resolved "<INTERNAL_REGISTRY>/typescript/-/typescript-2.8.3.tgz#5d817f9b6f31bb871835f4edf0089f21abe6c170"

Be sure to keep scoped packages in mind - there is an extra segment:

"@types/estree@0.0.38":
  version "0.0.38"
  resolved "<INTERNAL_REGISTRY>/@types/estree/-/estree-0.0.38.tgz#c1be40aa933723c608820a99a373a16d215a1ca2"

liamjones · 2018-08-22T08:07:17Z

Looks like it'd work for Nexus Repository too - you have a repo section after the server root but the tail of the URL is still the same:

type-check@~0.3.2:
  version "0.3.2"
  resolved "https://<server>/repository/<repo-identifier>/type-check/-/type-check-0.3.2.tgz#5884cab512cf1d355e3fb784f30804b2b520db72"
  dependencies:
    prelude-ls "~1.1.2"

besh · 2018-08-31T19:30:51Z

Confirming that it looks like it would work for artifactory too.

https://<country>.artifactory.<company>.com/artifactory/api/npm/<repo>/<module>.tgz

juanpicado · 2018-09-01T10:19:21Z

Verdaccio follows the same tarball url pattern as main registry using the default configuration, unless a user decides to use a prefix for reverse proxy.

https://<domain:port>/<prefix?>/-/<module>/<module>.tgz

Examples from yarn.lock.

resolved "http://localhost:4873/md5.js/-/md5.js-1.3.4.tgz#e9bdbde94a20a5ac18b04340fc5764d5b09d901d"
# scoped
resolved "http://localhost:4873/@babel%2fparser/-/parser-7.0.0-beta.51.tgz#27cec2df409df60af58270ed8f6aa55409ea86f6"

arcanis · 2018-09-01T14:38:11Z

Unfortunately we discovered that some registries follow entirely different rules (a surprising one being ... NPM Enterprise ...), so we won't be able to just trim the hostname 😞

Maybe we'll end up having a whitelist of domain names that can work this way ... but it's annoying.

swojit · 2018-09-05T16:29:45Z

What are their rules?

arcanis · 2018-09-05T16:33:15Z

Currently, it's something similar to this (cf pnpm/pnpm#867):

https://npm-registry.compass.com/p/pnpm/_attachments/pnpm-1.9.0.tgz

No idea why they chose to break their convention.

Swivelgames · 2018-11-20T21:50:40Z

What about a compromise to this?

Allowing for the --registry CLI argument when running both the yarn and yarn add command might be a helpful compromise to resolve this issue.

e.g.,

yarn --registry=http://registry.npm.taobao.org

This would allow the user to specify the registry to be used during the installation.

Another might be to trim the hostname if it matches the standard path structure, and leave it in if the URL differs. This would prevent the need for a whitelist, and instead only a simple regexp or simple parser, but allow for using the registry in the user's configuration as necessary for conforming registries.

The interim solution will be to use sed to point to a different registry during our CI builds, but this is obviously not a desired solution.

sed -i 's#http://private-registry/repository/npm#https://registry.yarnpkg.com#' yarn.lock

(from @victornoel in #2566 )

(cc @arcanis @BYK @doxiaodong @victornoel )

…ssues/5892)

npm is agnostic about which registry you used to generate the package-lock.json since 5.0.0 shipped with Node.js 8.0 [0]. And LeanEngine only uses `package-lock.json` for Node.js 8.0+. But yarn still depends on the registry host in yarn.lock [1]. [0]: https://stackoverflow.com/questions/53814967/what-is-the-point-of-having-resolved-url-in-package-lock-json [1]: yarnpkg/yarn#5892

…680) #680 Currently, when a package is installing its dependencies and those dependencies are in different subtrees of the monorepo with different yarn registries, the update and sync methods in jazelle are not aware of the difference in registries when it stitches the lockfile back together. During the stitching process, jazelle will index all of the lockfiles and then as it builds the lockfiles back up, it will take the highest versioned package across all deps and use that which is correct. However, if one of the projects was resolved with a different package registry, jazelle will potentially overwrite it indiscriminately since it has no concept of ensuring that the registry is correct for that package. This fix uses liberal use of yarn config get registry before installing missing and transitive deps so that the correct registry for the package will be used. In the case of lockfile syncing, this fix will convert all lockfile resolved paths to a relative path instead to be used as the index. Then when lockfiles are rebuilt, the registry that is local to the package (again using yarn config get registry) will be prepended to the relative path to reconstruct the lockfile. There is an open issue in the yarn repo that discusses changing the paths for good to relative paths but there seems to be no recent movement on this: yarnpkg/yarn#5892 The end result of this PR is that: multiple projects in the monorepo can have their own .yarnrc files that point to different registries jazelle will ensure all dependencies are synced across all projects it controls while respecting the configured registry for each project Co-authored-by: Derek Ju <derek.ju@gmail.com> Co-authored-by: UberOpenSourceBot <UberOpenSourceBot@users.noreply.github.com>

Ginxo · 2019-11-07T14:46:36Z

I (Redhat productization team) recently created a new npm library called lock-treatment-tool https://www.npmjs.com/package/lock-treatment-tool in order to treat lock files before initialising or installing the npm project. It will treat either package-lock.json/yarn.lock files, remove/replace the registries and integrity hash and then you can use your own registries.
I hope this helps you.

PMExtra · 2020-05-19T16:42:16Z

I can't believe this issue has existed for 3 years. (#3330)
Is there any schedule or roadmap for 2.0?

Daniel15 · 2020-05-19T17:34:59Z

@PMExtra This is the wrong repo for 2.0. The 2.0 repo is here: https://github.com/yarnpkg/berry

Ginxo · 2020-05-20T06:03:17Z

@PMExtra anyway you have the chance to use this lock-treatment-tool library https://www.npmjs.com/package/lock-treatment-tool
locktt --registry=https://npmregistryurl.com
You can even integrate it with frontend-maven-plugin library https://github.com/kiegroup/appformer/blob/0d8c104f2bdb182ca2768eaa5d3a18e57f6f8d1a/appformer-js/pom.xml#L42 just adding -DnpmRegistryURL=whateverregistryurl
I hope it helps

PMExtra · 2020-05-25T11:53:53Z

Thanks @Daniel15 @Ginxo .
I'm building CI/CD in China. But we have some oversea servers. So I want to determine npm registry mirror by GitLab Runner Enironments while building. But we should lock the packages version with yarn.lock.
So I want to override the host in yarn.lock. And I want as less as possible of the extra dependencies.

Ginxo · 2020-05-25T12:05:08Z

@PMExtra so lock-treatment-tool could be a good solution for you

…(#466) https://github.com/uber/fusionjs/pull/466 Currently, when a package is installing its dependencies and those dependencies are in different subtrees of the monorepo with different yarn registries, the update and sync methods in jazelle are not aware of the difference in registries when it stitches the lockfile back together. During the stitching process, jazelle will index all of the lockfiles and then as it builds the lockfiles back up, it will take the highest versioned package across all deps and use that which is correct. However, if one of the projects was resolved with a different package registry, jazelle will potentially overwrite it indiscriminately since it has no concept of ensuring that the registry is correct for that package. This fix uses liberal use of yarn config get registry before installing missing and transitive deps so that the correct registry for the package will be used. In the case of lockfile syncing, this fix will convert all lockfile resolved paths to a relative path instead to be used as the index. Then when lockfiles are rebuilt, the registry that is local to the package (again using yarn config get registry) will be prepended to the relative path to reconstruct the lockfile. There is an open issue in the yarn repo that discusses changing the paths for good to relative paths but there seems to be no recent movement on this: yarnpkg/yarn#5892 The end result of this PR is that: multiple projects in the monorepo can have their own .yarnrc files that point to different registries jazelle will ensure all dependencies are synced across all projects it controls while respecting the configured registry for each project Co-authored-by: Derek Ju <derek.ju@gmail.com> Co-authored-by: UberOpenSourceBot <UberOpenSourceBot@users.noreply.github.com>

ericis · 2020-12-16T23:54:54Z

👀 some engineers use local registry proxies, like Nexus to test locally, improve downloads, and to work offline (assuming packages are already downloaded). I tried passing --registry [url], as well as working with ".yarnrc" and ".npmrc". It seems we'll have to remove the "yarn.lock" file from source control until this is fixed.

arcanis · 2020-12-17T00:06:33Z

This issue is fixed in Yarn 2+, and the fix will not be backported in 1.x. Please check the Migration guide and continue from there.

arcanis added this to the Yarn 2.0 milestone May 26, 2018

arcanis mentioned this issue May 26, 2018

yarn.lock should not include base registry(http://registry.npmjs.org) #3330

Closed

juanpicado mentioned this issue Jul 3, 2018

verdaccio endpoint in generated lockfiles verdaccio/verdaccio#801

Closed

Gudahtt mentioned this issue Aug 19, 2018

Replace resolved field by hash yarnpkg/rfcs#64

Open

arcanis mentioned this issue Aug 21, 2018

yarn does not honor authentication settings in .npmrc #4451

Closed

arcanis mentioned this issue Oct 4, 2018

Default yarn.lock registry? #6489

Open

arcanis mentioned this issue Oct 31, 2018

Caches the Yarn resolution for faster installs facebook/create-react-app#5270

Merged

arcanis mentioned this issue Jan 24, 2019

Yarn's Future - v2 and beyond #6953

Open

petershin added a commit to petershin/liferay-portal that referenced this issue Apr 3, 2019

LPS-93258 Update hostname in yarn.lock (see github.com/yarnpkg/yarn/i…

8d3a748

…ssues/5892)

petershin added a commit to petershin/liferay-portal that referenced this issue Apr 3, 2019

LPS-93258 Update hostname in yarn.lock (see yarnpkg/yarn#5892)

72d6956

brianchandotcom pushed a commit to liferay/liferay-portal that referenced this issue Apr 3, 2019

LPS-93258 Update hostname in yarn.lock (see yarnpkg/yarn#5892)

e7cd177

arcanis mentioned this issue Apr 15, 2019

Yarn offline mirror filename calculation broken for NPM Enterprise #7199

Closed

arcanis mentioned this issue Jun 20, 2019

[Bug] Registries with non-conventional tarball paths yarnpkg/berry#238

Closed

brianchandotcom pushed a commit to liferay/liferay-portal that referenced this issue Aug 1, 2019

LPS-93258 Update hostname in yarn.lock (see yarnpkg/yarn#5892)

f14c216

jasongrout mentioned this issue Aug 27, 2019

Issues with using a private yarn registry jupyterlab/jupyterlab#3658

Closed

derekju mentioned this issue Aug 29, 2019

Use local package registry when installing packages and dependencies fusionjs/fusionjs#672

Merged

derekju mentioned this issue Aug 30, 2019

Use local package registry when installing packages and dependencies fusionjs/fusionjs#680

Closed

lukasluecke mentioned this issue Sep 23, 2019

Add NPM registry proxy whatwedo/dde#7

Closed

millette mentioned this issue Oct 20, 2019

Use Buffer.from() instead of new Buffer(); require node >= 5.10.0. sergioramos/jsonstream2#4

Merged

BlueCog mentioned this issue Feb 10, 2020

RuntimeError: npm dependencies failed to install - Proxy & Private npm registry issues jupyterlab/jupyterlab#7660

Closed

imsobear mentioned this issue Mar 1, 2020

Fix electron target imsobear/iceworks-next#3

Merged

qzchenwl mentioned this issue May 16, 2020

jupyter lab build raise 404 error. jupyterlab/jupyterlab#8431

Open

arcanis closed this as completed Dec 17, 2020

yarnpkg locked as resolved and limited conversation to collaborators Dec 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove hostnames from the lockfiles #5892

Remove hostnames from the lockfiles #5892

arcanis commented May 26, 2018

hutson commented May 26, 2018

Daniel15 commented May 26, 2018

arcanis commented May 26, 2018

dballance commented Aug 21, 2018

liamjones commented Aug 22, 2018 •

edited

Loading

besh commented Aug 31, 2018 •

edited

Loading

juanpicado commented Sep 1, 2018

arcanis commented Sep 1, 2018

swojit commented Sep 5, 2018

arcanis commented Sep 5, 2018 •

edited

Loading

Swivelgames commented Nov 20, 2018 •

edited

Loading

Ginxo commented Nov 7, 2019 •

edited

Loading

PMExtra commented May 19, 2020

Daniel15 commented May 19, 2020

Ginxo commented May 20, 2020

PMExtra commented May 25, 2020 •

edited

Loading

Ginxo commented May 25, 2020

ericis commented Dec 16, 2020

arcanis commented Dec 17, 2020

Remove hostnames from the lockfiles #5892

Remove hostnames from the lockfiles #5892

Comments

arcanis commented May 26, 2018

hutson commented May 26, 2018

Daniel15 commented May 26, 2018

arcanis commented May 26, 2018

dballance commented Aug 21, 2018

liamjones commented Aug 22, 2018 • edited Loading

besh commented Aug 31, 2018 • edited Loading

juanpicado commented Sep 1, 2018

arcanis commented Sep 1, 2018

swojit commented Sep 5, 2018

arcanis commented Sep 5, 2018 • edited Loading

Swivelgames commented Nov 20, 2018 • edited Loading

Ginxo commented Nov 7, 2019 • edited Loading

PMExtra commented May 19, 2020

Daniel15 commented May 19, 2020

Ginxo commented May 20, 2020

PMExtra commented May 25, 2020 • edited Loading

Ginxo commented May 25, 2020

ericis commented Dec 16, 2020

arcanis commented Dec 17, 2020

liamjones commented Aug 22, 2018 •

edited

Loading

besh commented Aug 31, 2018 •

edited

Loading

arcanis commented Sep 5, 2018 •

edited

Loading

Swivelgames commented Nov 20, 2018 •

edited

Loading

Ginxo commented Nov 7, 2019 •

edited

Loading

PMExtra commented May 25, 2020 •

edited

Loading