yarn.lock not generated correctly with yarn v4

[a88zach]

Current Behavior

When using Yarn v4 and executing a build using @nx/webpack with "generateLockfile": true, the generated yarn.lock file is not generated correctly.

The first item in the yarn.lock file is generated without a name as seen below

"":
  version: 1.22.8
  resolution: "resolve@patch:resolve@npm%3A1.22.8#optional!builtin<compat/resolve>::version=1.22.8&hash=c3c19d"
  dependencies:
    is-core-module: "npm:^2.13.0"
    path-parse: "npm:^1.0.7"
    supports-preserve-symlinks-flag: "npm:^1.0.0"
  bin:
    resolve: bin/resolve
  checksum: 0446f024439cd2e50c6c8fa8ba77eaa8370b4180f401a96abf3d1ebc770ac51c1955e12764cde449fde3fff480a61f84388e3505ecdbab778f4bef5f8212c729
  languageName: node
  linkType: hard

However, this is the corresponding entry from the root yarn.lock file

"resolve@patch:resolve@npm%3A^1.1.7#optional!builtin<compat/resolve>, resolve@patch:resolve@npm%3A^1.10.0#optional!builtin<compat/resolve>, resolve@patch:resolve@npm%3A^1.12.0#optional!builtin<compat/resolve>, resolve@patch:resolve@npm%3A^1.14.2#optional!builtin<compat/resolve>, resolve@patch:resolve@npm%3A^1.19.0#optional!builtin<compat/resolve>, resolve@patch:resolve@npm%3A^1.20.0#optional!builtin<compat/resolve>, resolve@patch:resolve@npm%3A^1.22.1#optional!builtin<compat/resolve>, resolve@patch:resolve@npm%3A^1.22.4#optional!builtin<compat/resolve>":
  version: 1.22.8
  resolution: "resolve@patch:resolve@npm%3A1.22.8#optional!builtin<compat/resolve>::version=1.22.8&hash=c3c19d"
  dependencies:
    is-core-module: "npm:^2.13.0"
    path-parse: "npm:^1.0.7"
    supports-preserve-symlinks-flag: "npm:^1.0.0"
  bin:
    resolve: bin/resolve
  checksum: 0446f024439cd2e50c6c8fa8ba77eaa8370b4180f401a96abf3d1ebc770ac51c1955e12764cde449fde3fff480a61f84388e3505ecdbab778f4bef5f8212c729
  languageName: node
  linkType: hard

As a result, yarn install will fail with the following error

Internal Error: Invalid descriptor ()

Expected Behavior

The yarn.lock file should include the name of the matching entry from the root yarn.lock so that yarn install runs correctly

GitHub Repo

No response

Steps to Reproduce

1. Set up a new nx repo with Yarn v4
2. Create a new node application that uses webpack and set "generateLockfile": true
3. Build the app
4. Try to install the app dependencies with yarn install

Nx Report

Node   : 20.9.0
   OS     : darwin-arm64
   yarn   : 4.0.0
   
   nx                 : 17.0.2
   @nx/js             : 17.0.2
   @nx/jest           : 17.0.2
   @nx/linter         : 17.0.2
   @nx/eslint         : 17.0.2
   @nrwl/linter       : 15.9.2
   @nx/workspace      : 17.0.2
   @nx/cypress        : 17.0.2
   @nx/devkit         : 17.0.2
   @nx/esbuild        : 17.0.2
   @nx/eslint-plugin  : 17.0.2
   @nx/node           : 17.0.2
   @nx/react          : 17.0.2
   @nx/storybook      : 17.0.2
   @nrwl/tao          : 17.0.2
   @nx/web            : 17.0.2
   @nx/webpack        : 17.0.2
   typescript         : 5.2.2

Failure Logs

No response

Package Manager Version

No response

Operating System

• macOS
• Linux
• Windows
• Other (Please specify)

Additional Information

No response

[veimox]

This is likely to be the same as #17873 that was fixed #18192 but introduced #18726. This last one was fixed by #18759 but brought back the same issue.

Long story short, it is not a new issue, just an edge case for the people using yarn berry (in this case v4) under certain setting where the yarn.lock is in certain way and nx then prunes it incorrectly.

To reproduce it i'm sure we can take any of the example repos in those issues.

[milo526]

We are experiencing the same problem with the same configuration.

I can confirm that re-applying the diff from #18192 does indeed fix the problem.

[yizumi1012xxx]

I have same problem and this is minimum repro code.
https://github.com/yizumi1012xxx/nx-examples-repro-19881

execute yarn nx build server, then generate below yarn.lock file
(when use chokidar in app code, cause this problem)

...

"":
  version: 2.3.2
  resolution: "fsevents@patch:fsevents@npm%3A2.3.2#optional!builtin<compat/fsevents>::version=2.3.2&hash=df0bf1"
  dependencies:
    node-gyp: "npm:latest"
  conditions: os=darwin
  languageName: node
  linkType: hard

...

[cwoebker]

@veimox @meeroslav Since this bug was reintroduced with the recent changes, are there any plans to get this fixed? It would be great to have compatibility with yarn v3 / v4. Thanks for your work!

[pedropereira]

We're also experiencing this issue with latest versions of nx and yarn - any updates?

[robonaut]

Same issue running nx 18.2 and yarn 4.1.1. For one of our services I need to revert to yarn 1 to install the dependencies.

I did some additional tests in our specific bundle and I traced the culprit down to a specific require statement in one file:

const Converter = require("api-spec-converter");

Removing that line fixes the generated yarn.lock.

Creating a nx nest project with only one single main.ts file with that single require statement will reproduce the issue.

[z0w0]

Is this still on the NX team's radar? Yarn berry is widely adopted these days, and not having the ability to easily prune node_modules using yarn for a single service/app/etc. breaks the single package.json philosophy. My team currently has to workaround this by ignoring the generated yarn.lock and reinstalling packages using the generated package.json which is really not ideal.

[0b3r]

I second @z0w0 's statement. Our team just had to do the same thing to make this work.

[sbuys]

Thirding @z0w0. This has become a big pain for our team.

[cwoebker]

@z0w0 As a potential improvement you could also take out the incorrectly generated parts from the yarn.lock file. E.g. in our case I just added a simple command to our Dockerfile:

RUN sed -i '8,13d' yarn.lock

This would take out those incorrect lines and still install everything else according to the lock file, not ideal of course, but better than just installing everything from the package.json file.

This won't work with --immutable though.

[DrewLandgraveCbsi]

@z0w0 As a potential improvement you could also take out the incorrectly generated parts from the yarn.lock file. E.g. in our case I just added a simple command to our Dockerfile:

RUN sed -i '8,13d' yarn.lock

This would take out those incorrect lines and still install everything else according to the lock file, not ideal of course, but better than just installing everything from the package.json file.

This won't work with --immutable though.

This will work you just have to find the correct lines in your yarn.lock file. For me it was `RUN sed -i '8.19d' yarn.lock Where 8 and 19 are the starting and ending lines of thigs you want to delete

[DrewLandgraveCbsi]

Probably a cleaner solution until someone on the nx team gets this fixed is to do this. It's what I ultimately did

COPY .yarnrc.yml ./
COPY .yarn ./.yarn
RUN corepack enable
RUN corepack prepare yarn@stable --activate
RUN rm yarn.lock
RUN yarn install

[bieblebrox]

Probably a cleaner solution until someone on the nx team gets this fixed is to do this. It's what I ultimately did

COPY .yarnrc.yml ./
COPY .yarn ./.yarn
RUN corepack enable
RUN corepack prepare yarn@stable --activate
RUN rm yarn.lock
RUN yarn install

Hi @DrewLandgraveCbsi ,

We are experiencing the same issue, could you briefly explain your solution with corepack please? How is it different than doing clean install based on package.json ?

Thanks a lot

[DrewLandgraveCbsi]

Probably a cleaner solution until someone on the nx team gets this fixed is to do this. It's what I ultimately did

COPY .yarnrc.yml ./
COPY .yarn ./.yarn
RUN corepack enable
RUN corepack prepare yarn@stable --activate
RUN rm yarn.lock
RUN yarn install

Hi @DrewLandgraveCbsi ,

We are experiencing the same issue, could you briefly explain your solution with corepack please? How is it different than doing clean install based on package.json ?

Thanks a lot

@bieblebrox Corepack is required for yarn berry (we're on yarn 4.x) which we are using. If you aren't then you done need it. But ultimately removing the yarn.lock and doing install is doing a fresh install from the package.json

Copying the yarnrc.yml and the .yarn folder were required because GitHub kept complaining without them.

[shahen94]

+1

[cnorthfield]

Is this still on the NX team's radar? Yarn berry is widely adopted these days, and not having the ability to easily prune node_modules using yarn for a single service/app/etc. breaks the single package.json philosophy. My team currently has to workaround this by ignoring the generated yarn.lock and reinstalling packages using the generated package.json which is really not ideal.

We are in the same situation. We would appreciate an immediate fix to this.

[sebastiaan-dev]

+1

[AdamJHall]

Having the same issue on a project of ours, only workarounds have been to revert to using yarn 1.x or install from package.json both if which aren't ideal.

[a88zach]

@FrozenPandaz @MaxKless - submitted a proposed fix in the above PR if you could take a look. This one is a bit over a year old now.

[MaxKless]

sorry I missed this @a88zach but thank you so much for the PR!

[a88zach]

For anyone following, there is one more issue.

This code needs to be updated for v4: https://github.com/a88zach/nx/blob/master/packages/nx/src/plugins/js/lock-file/yarn-parser.ts#L601

In v3 the dependencies are listed with jus the semvar: 1.1.1
In v4 the dependencies are listed with the npm: prefix: npm:1.1.1

Looking for a fix and will hopefully have another PR soon

[a88zach]

#29451

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.