[Bug] Cannot add dependency as tar.gz from URL. #2437

EliteMasterEric · 2021-02-04T03:35:00Z

I'd be willing to implement a fix

Describe the bug

GitPKG is a website which allows the installation of NPM packages from a github repo which is a monorepo, i.e. the individual packages to be installed are located in subdirectories of the main repo. This is a situation that typically cannot be resolved by package managers, since the URL of the repo itself cannot be distinguished from a subdirectory of that repo.

In Yarn v1, the following line can be run:

yarn add https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello

This will add the hello repo to the project's dependencies. Accessing the above URL directly will return a tar.gz file containing the repo data.

In Yarn v2, this command will fail with the following stack trace:

> yarn add https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello
Internal Error: Invalid descriptor (https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello)
    at Module.x (D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:2:434115)
    at D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:2:29878
    at Array.map (<anonymous>)
    at M.execute (D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:2:29786)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async M.validateAndExecute (D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:2:659490)
    at async Y.run (D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:17:3854)
    at async Y.runExit (D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:17:4021)
    at async h (D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:2:284223)
    at async r (D:\eric\Documents\Programming\Website\test-icons\.yarn\releases\yarn-berry.cjs:2:282834)

I researched the issue here and found #994 (comment), but running yarn add hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello provides the same error.

To Reproduce

I have reproduced the error, using Sherlock on my local machine.

Reproduction

This is my reproduction case:

// The following command was valid 
// yarn add https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello

const installPromise = yarn(`add`, `https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello`);

// Should install successfully but can't.
await expect(installPromise)
  .resolves.toBeTruthy();

/**
 * Specifying the package name doesn't work either.
 * yarn add hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello
 * @see: https://github.com/yarnpkg/berry/issues/994#issuecomment-628586986
 */

const installPromise = yarn(`add`, `hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello`);

await expect(installPromise)
  .resolves.toBeTruthy();

Environment if relevant (please complete the following information):

OS: Windows
Node version: v14.15.4
Yarn version: 2.4.0

Additional context

I was not able to get Sherlock working properly on the sandbox website due to the following error:

Usage Error: This tool requires a Node version compatible with >=12 <14 || 14.2 - 14.9 || >14.10.0 (got 10.23.0). Upgrade Node, or set `YARN_IGNORE_NODE=1` in your environment.·
Yarn Package Manager - 2.4.0.dev·

I was able to clone the project to execute Sherlock offline and create the above reproduction.

The text was updated successfully, but these errors were encountered:

EliteMasterEric · 2021-02-04T03:36:09Z

The issue appears to be on this line:

berry/packages/yarnpkg-core/sources/structUtils.ts

Line 358 in 14bc23f

const match = strict

yarnbot · 2021-02-04T03:37:48Z

This issue reproduces on master:

Error: expect(received).resolves.toBeTruthy()

Received promise rejected instead of resolved
Rejected to value: [Error: Command failed: /usr/bin/node /github/workspace/scripts/actions/../run-yarn.js add hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello

Usage Error: No project found in /tmp/tmp-20NCgscuoG90y9

$ yarn add [--json] [-E,--exact] [-T,--tilde] [-C,--caret] [-D,--dev] [-P,--peer] [-O,--optional] [--prefer-dev] [-i,--interactive] [--cached] ...
]
    at expect (/github/workspace/.yarn/cache/expect-npm-24.8.0-8c7640c562-0ac41999f0.zip/node_modules/expect/build/index.js:138:15)
    at module.exports (evalmachine.<anonymous>:10:7)
    at /github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-63f998598d.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:56:19
    at executeInTempDirectory (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-63f998598d.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:17:22)
    at Object.executeRepro (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-63f998598d.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:24:18)
    at ExecCommand.execute (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-63f998598d.zip/node_modules/@arcanis/sherlock/lib/commands/exec.js:25:59)
    at async ExecCommand.validateAndExecute (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-a57989414f.zip/node_modules/clipanion/lib/advanced/Command.js:161:26)
    at async Cli.run (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-a57989414f.zip/node_modules/clipanion/lib/advanced/Cli.js:74:24)
    at async Cli.runExit (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-a57989414f.zip/node_modules/clipanion/lib/advanced/Cli.js:83:28)

arcanis · 2021-02-04T08:36:37Z

This is expected; we currently require the package name to be listed, except for local file paths. ie, this isn't supported:

yarn add https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello

This is:

yarn add hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello

I'll clarify the error message in the next release, I agree it's currently quite confusing.

EliteMasterEric · 2021-02-04T18:58:19Z

This is expected; we currently require the package name to be listed, except for local file paths. ie, this isn't supported:
yarn add https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello
This is:
yarn add hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello
I'll clarify the error message in the next release, I agree it's currently quite confusing.

@arcanis Please actually read the issue, I tested the latter (as I researched and found the answer you just gave in #994) and the same error is thrown.

I just got the following error on Ubuntu 20.04/Yarn 2.4.0 (so the issue is cross-platform):

$ yarn add hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello
➤ YN0000: ┌ Resolution step
➤ YN0001: │ Error: hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello isn't supported by any available resolver
    at n.getResolverByDescriptor (/home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:336683)
    at n.bindDescriptor (/home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:336048)
    at d (/home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:357409)
    at async Promise.all (index 0)
    at async ie.resolveEverything (/home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:358614)
    at async /home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:377271
    at async f.startTimerPromise (/home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:389740)
    at async ie.install (/home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:377210)
    at async /home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:33217
    at async Function.start (/home/eric/Documents/Programming/test-yarn/.yarn/releases/yarn-berry.cjs:2:388437)
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 8ms

merceyz · 2021-02-05T14:24:49Z

You need to add the extension, but that seems to highlight another issue

$ yarn add hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello.tgz
➤ YN0001: │ Error: hello@https://gitpkg.now.sh/EqualMa/gitpkg-hello/packages/hello.tgz: ENOENT: no such file or directory, chmod 'C:\Temp\xfs-4d40b2cb\archive.zip'

GitPKG is a website which allows the installation of NPM packages from a github repo which is a monorepo, i.e. the individual packages to be installed are located in subdirectories of the main repo. This is a situation that typically cannot be resolved by package managers, since the URL of the repo itself cannot be distinguished from a subdirectory of that repo.

We can actually https://yarnpkg.com/features/protocols#git

EliteMasterEric · 2021-02-05T21:21:19Z

We can actually https://yarnpkg.com/features/protocols#git

This is interesting. The dependency I am attempting to import is a clone of an existing project (which uses NPM) but it's good to know Yarn supports this.

This is kind of a tangent but I'd really like to see a page in the docs that provides a breakdown for "Why should you migrate your project from NPM?" Points could include stuff like speed benchmarks and lists of features that NPM does not support.

refi64 · 2021-12-11T19:28:20Z

Just as a quick FYI, if anyone here has a site where adding the .tgz is not an option, it seems you can just add it after a #, e.g.:

yarn add https://the-url.com/package#.tgz

in which case it usually isn't counted as part of the URL, so the actual URL is https://the-url.com/package, but Yarn sees the .tgz. (This trick is often used to rename sources in RPM spec files.)

hendriku · 2023-03-09T09:11:43Z

We still have this problem in yarn 3. Using https:// and suffixing with #.tgz also is the workaround for us.

dakom · 2023-08-11T05:59:37Z

that trick does work, but how about when specifying a commit? e.g. this fails:

yarn add recoilize@https://gitpkg.now.sh/open-source-labs/Recoilize/package?524ff2f9650e4a24e89ef00caa638b5148e9e966#.tgz

EliteMasterEric added the bug Something isn't working label Feb 4, 2021

yarnbot added the reproducible This issue can be successfully reproduced label Feb 4, 2021

arcanis closed this as completed Feb 4, 2021

bryanvaz mentioned this issue Apr 10, 2021

[Bug] HTTPS Resolver supported by Yarn v1 does not work in Yarn v2 #2729

Closed

dakom mentioned this issue Aug 11, 2023

yarn v3 install with subdir and commit? ramasilveyra/gitpkg#66

Open

arcanis mentioned this issue Nov 3, 2023

[Bug]: HTTP urls do not support query parameters and do not support extensionless URLs #5929

Closed

1 task

keturiosakys mentioned this issue Dec 11, 2023

Update dependencies to use NPM for @astropub/md HiDeoo/starlight-openapi#15

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] Cannot add dependency as tar.gz from URL. #2437

[Bug] Cannot add dependency as tar.gz from URL. #2437

EliteMasterEric commented Feb 4, 2021

EliteMasterEric commented Feb 4, 2021 •

edited

Loading

yarnbot commented Feb 4, 2021

arcanis commented Feb 4, 2021

EliteMasterEric commented Feb 4, 2021 •

edited

Loading

merceyz commented Feb 5, 2021

EliteMasterEric commented Feb 5, 2021

refi64 commented Dec 11, 2021

hendriku commented Mar 9, 2023

dakom commented Aug 11, 2023

[Bug] Cannot add dependency as tar.gz from URL. #2437

[Bug] Cannot add dependency as tar.gz from URL. #2437

Comments

EliteMasterEric commented Feb 4, 2021

EliteMasterEric commented Feb 4, 2021 • edited Loading

yarnbot commented Feb 4, 2021

arcanis commented Feb 4, 2021

EliteMasterEric commented Feb 4, 2021 • edited Loading

merceyz commented Feb 5, 2021

EliteMasterEric commented Feb 5, 2021

refi64 commented Dec 11, 2021

hendriku commented Mar 9, 2023

dakom commented Aug 11, 2023

EliteMasterEric commented Feb 4, 2021 •

edited

Loading

EliteMasterEric commented Feb 4, 2021 •

edited

Loading