[Bug] Workspaces foreach always exits with zero even on error #2486

MarsianMan · 2021-02-17T02:54:56Z

I'd be willing to implement a fix

Describe the bug

When a build or test fails using workspaces foreach run test yarn should exit with a non-zero status.

To Reproduce

Checkout https://github.com/marcneander/yarn-foreach-error.git

From root directory run yarn run test && echo "done" or yarn workspaces foreach run -pvai test && echo "done"

Expectation: in the output below, done should not output when using foreach

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn set version berry
➤ YN0000: Downloading https://github.com/yarnpkg/berry/raw/master/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-2.4.0.cjs
➤ YN0000: Done in 2s 116ms
/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn run test && echo "done"
done
/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn workspace failure run test && echo "done"
/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$

Note that if I run the test on the workspace directly by name, then the exit code is correct.

Screenshots

Environment if relevant (please complete the following information):

OS: [OSX, Linux]
Node version [15]
Yarn version [2.4.0]

Additional context

In a non-sample project, I am trying to use yarn to run cucumber.js tests in workspaces. The tests are failing, but the build is passing.

yarn workspaces foreach -pviA  run test && echo "done"
➤ YN0000: [workspace A]: Seeing profile=XYZ
➤ YN0000: [common-test]: Seeing profile=XYZ
➤ YN0000: [workspace A]: ..F-
➤ YN0000: [workspace A]: 
➤ YN0000: [workspace A]: Failures:
➤ YN0000: [workspace A]: 
➤ YN0000: [workspace A]: 1) Scenario: Can reach API Proxy # test/features/apigee.feature:9
➤ YN0000: [workspace A]:    ✔ Given I have a REST client # ../../common-test/test/features/step-definitions/apickli.ts:28
➤ YN0000: [workspace A]:    ✔ Given I set "User-Agent" header to 'XXXXXXXX' # ../../common-test/test/features/step-definitions/apickli.ts:34
➤ YN0000: [workspace A]:    ✖ When I GET "/ping" # ../../common-test/test/features/step-definitions/apickli.ts:88
➤ YN0000: [workspace A]:        Error: Error: socket hang up
➤ YN0000: [workspace A]:            at /Volumes/HDD2/apigee-config/src/common-test/test/features/step-definitions/apickli.ts:91:22
➤ YN0000: [workspace A]:            at Request._callback (/Volumes/HDD2/apigee-config/.yarn/cache/apickli-npm-2.3.3-e973675e47-469a7ec0f4.zip/node_modules/apickli/apickli.js:481:14)
➤ YN0000: [workspace A]:            at self.callback (/Volumes/HDD2/apigee-config/.yarn/cache/request-npm-2.88.2-f4a57c72c4-7a74841f30.zip/node_modules/request/request.js:185:22)
➤ YN0000: [workspace A]:            at Request.emit (node:events:329:20)
➤ YN0000: [workspace A]:            at Request.EventEmitter.emit (node:domain:467:12)
➤ YN0000: [workspace A]:            at Request.onRequestError (/Volumes/HDD2/apigee-config/.yarn/cache/request-npm-2.88.2-f4a57c72c4-7a74841f30.zip/node_modules/request/request.js:877:8)
➤ YN0000: [workspace A]:            at ClientRequest.emit (node:events:329:20)
➤ YN0000: [workspace A]:            at ClientRequest.EventEmitter.emit (node:domain:467:12)
➤ YN0000: [workspace A]:            at TLSSocket.socketOnEnd (node:_http_client:502:9)
➤ YN0000: [workspace A]:            at TLSSocket.emit (node:events:341:22)
➤ YN0000: [workspace A]:            at TLSSocket.EventEmitter.emit (node:domain:467:12)
➤ YN0000: [workspace A]:            at endReadableNT (node:internal/streams/readable:1294:12)
➤ YN0000: [workspace A]:            at processTicksAndRejections (node:internal/process/task_queues:80:21)
➤ YN0000: [workspace A]:    - Then response code should be 200 # ../../common-test/test/features/step-definitions/apickli.ts:164
➤ YN0000: [workspace A]: 
➤ YN0000: [workspace A]: 1 scenario (1 failed)
➤ YN0000: [workspace A]: 4 steps (1 failed, 1 skipped, 2 passed)
➤ YN0000: [workspace A]: 0m00.590s (executing steps: 0m00.578s)
➤ YN0000: [common-test]: ..F-
➤ YN0000: [common-test]: 
➤ YN0000: [common-test]: Failures:
➤ YN0000: [common-test]: 
➤ YN0000: [common-test]: 1) Scenario: Can reach API Proxy # test/features/apigee.feature:9
➤ YN0000: [common-test]:    ✔ Given I have a REST client # test/features/step-definitions/apickli.ts:28
➤ YN0000: [common-test]:    ✔ Given I set "User-Agent" header to 'XXXXX' # test/features/step-definitions/apickli.ts:34
➤ YN0000: [common-test]:    ✖ When I GET "/ping" # test/features/step-definitions/apickli.ts:88
➤ YN0000: [common-test]:        Error: Error: socket hang up
➤ YN0000: [common-test]:            at /Volumes/HDD2/apigee-config/src/common-test/test/features/step-definitions/apickli.ts:91:22
➤ YN0000: [common-test]:            at Request._callback (/Volumes/HDD2/apigee-config/.yarn/cache/apickli-npm-2.3.3-e973675e47-469a7ec0f4.zip/node_modules/apickli/apickli.js:481:14)
➤ YN0000: [common-test]:            at self.callback (/Volumes/HDD2/apigee-config/.yarn/cache/request-npm-2.88.2-f4a57c72c4-7a74841f30.zip/node_modules/request/request.js:185:22)
➤ YN0000: [common-test]:            at Request.emit (node:events:329:20)
➤ YN0000: [common-test]:            at Request.EventEmitter.emit (node:domain:467:12)
➤ YN0000: [common-test]:            at Request.onRequestError (/Volumes/HDD2/apigee-config/.yarn/cache/request-npm-2.88.2-f4a57c72c4-7a74841f30.zip/node_modules/request/request.js:877:8)
➤ YN0000: [common-test]:            at ClientRequest.emit (node:events:329:20)
➤ YN0000: [common-test]:            at ClientRequest.EventEmitter.emit (node:domain:467:12)
➤ YN0000: [common-test]:            at TLSSocket.socketOnEnd (node:_http_client:502:9)
➤ YN0000: [common-test]:            at TLSSocket.emit (node:events:341:22)
➤ YN0000: [common-test]:            at TLSSocket.EventEmitter.emit (node:domain:467:12)
➤ YN0000: [common-test]:            at endReadableNT (node:internal/streams/readable:1294:12)
➤ YN0000: [common-test]:            at processTicksAndRejections (node:internal/process/task_queues:80:21)
➤ YN0000: [common-test]:    - Then response code should be 200 # test/features/step-definitions/apickli.ts:164
➤ YN0000: [common-test]: 
➤ YN0000: [common-test]: 1 scenario (1 failed)
➤ YN0000: [common-test]: 4 steps (1 failed, 1 skipped, 2 passed)
➤ YN0000: [common-test]: 0m00.476s (executing steps: 0m00.464s)
done

I expect a non-zero exit code and "done" to be missing from the output.

The text was updated successfully, but these errors were encountered:

arcanis · 2021-02-17T09:20:25Z

I don't understand - done is not printed when using foreach, as shown in the output you shared (your prompt makes it fairly difficult to parse btw):

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn set version berry
➤ YN0000: Downloading https://github.com/yarnpkg/berry/raw/master/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-2.4.0.cjs
➤ YN0000: Done in 2s 116ms

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn run test && echo "done"
done

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn workspace failure run test && echo "done"

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$

arcanis · 2021-02-17T09:22:15Z

Also note that we purposefully have code setting the exit code to 1 if anything fails, so I suspect your test framework doesn't actually exits with a non-zero exit code:

berry/packages/plugin-workspace-tools/sources/commands/foreach.ts

Lines 323 to 328 in 909a125

    
           const errorCode = exitCodes.find(code => code !== 0); 
        
           // The order in which the exit codes will be processed is fairly 
        
           // opaque, so better just return a generic "1" for determinism. 
        
           if (finalExitCode === null) 
        
             finalExitCode = typeof errorCode !== `undefined` ? 1 : finalExitCode;

MarsianMan · 2021-02-17T18:51:12Z

I don't understand - done is not printed when using foreach, as shown in the output you shared (your prompt makes it fairly difficult to parse btw):

You have misread the commandline output.
Steps in output

check out repo (not shown)
Set version berry
Run workspace test directly (fails as expected)
Run workspace test via foreach (unexpectedly passes)

Cleaned up some output

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn set version berry
➤ YN0000: Downloading https://github.com/yarnpkg/berry/raw/master/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-2.4.0.cjs
➤ YN0000: Done in 2s 116ms

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn workspace failure run test && echo "done"

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn run test && echo "done"
done

Running test using the workspace the test fails correctly (no done)

$ yarn workspace failure run test && echo "done"
$

Even running the test explicitly with foreach does NOT fail correctly

$ yarn workspaces foreach run test -A && echo "done"
done
$

Also note that we purposefully have code setting the exit code to 1 if anything fails, so I suspect your test framework doesn't actually exits with a non-zero exit code:

🤷🏾‍♂️ That is nice and well, but this issue is easily reproducible for me. The test fails when running without foreach so unlikely to be a test framework issue. My test framework is Cucumber.js which is pretty battle tested and also NOT part of the reproducible example 😉

MarsianMan · 2021-02-17T20:20:54Z

In case the problem was the way the test was boostrapped...I tried using false and exit 1 as the failing "test" command (for workspace failure) and both have the same behavior.

NOTE: I did not create the linked repo, but reproducing the same/similar issue with it.

MarsianMan · 2021-03-01T23:41:24Z

Just following up to check if any further clarification is needed?

DanielOrtel · 2021-03-02T04:19:21Z

yeah, having the same problem with yarn workspaces foreach, for both jest and eslint. With the following workspace:

{
  "name": "@my-scope/package",
  "version": "1.0.0",
  "private": true,
  "main": "src/index.js",
  "devDependencies": {
    "eslint": "^7.20.0",
    "jest": "^26.6.3"
  },
  "scripts": {
    "lint": "eslint",
    "unit-test": "jest --coverage",
    "test": "yarn lint && yarn unit-test",
  }
}

Running either yarn workspaces foreach lint, yarn workspaces foreach unit-test or yarn workspaces foreach test will produce a successful outcome, even if any/all of these are failing. It does log that, Process exited without output (exit code 0), but still exits with the same 0 exit code.

bfricka · 2021-03-03T17:56:26Z

We use yarn workspaces foreach -piv run test and it exits w/ the expected non-zero code. I tested this w/ roughly the same echo test. Very odd. That said, I did test it w/ from sources not berry, since we're using TS 4.2.

jasonwilliams · 2021-03-17T11:46:52Z

Also having the same issue but only on Node v15 or above.
On an environment where Node v14 is installed i get 1 as the exit code (which is correct).

Switching to Node v15 i get 0 (incorrect as the child command fails), also the output is cut off like it didn't finish properly.

@MarsianMan you mentioned Node 15 in your issue, could you try downgrading to v14 and running it again?

merceyz · 2021-03-17T11:53:56Z

Switching to Node v15 i get 0 (incorrect as the child command fails), also the output is cut off like it didn't finish properly.

Try using yarn from master - #2255

yarn set version from sources

jasonwilliams · 2021-03-17T12:15:06Z

Thanks @merceyz

I just get

Usage Error: This plugin cannot access the package referenced via yup which is neither a builtin, nor an exposed entry (when initializing @yarnpkg/plugin-workspace-tools, defined in /path/to/project/.yarnrc.yml)

━━━ Yarn Package Manager - 2.4.0-git.20210311.hash-9705c122 ━━━━━━━━━━━━━━━━━━━━━

  $ yarn <command>

You can also print more details about any of these commands by calling them with 
the `-h,--help` flag right after the command name.

When switching to the sources version

merceyz · 2021-03-17T15:02:27Z

You need to import your plugins from sources as well https://yarnpkg.com/cli/plugin/import/from/sources

yarnbot · 2021-04-16T15:05:01Z

Hi! 👋

This issue looks stale, and doesn't feature the reproducible label - which implies that you didn't provide a working reproduction using Sherlock. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it or you edit your first post to include a formal reproduction (you can use the playground for that).

Note that we require Sherlock reproductions for long-lived issues (rather than standalone git repositories or similar) because we're a small team. Sherlock gives us the ability to check which bugs are still affecting the master branch at any given point, and decreases the amount of code we need to run on our own machines (thus leading to faster bug resolutions). It helps us help you! 😃

If you absolutely cannot reproduce a bug on Sherlock (for example because it's a Windows-only issue), a maintainer will have to manually add the upholded label. Thanks for helping us triaging our repository! 🌟

merceyz · 2021-04-17T15:45:03Z

Can reproduce with the version in the repro but not with 3.0.0-rc.2 so closing as fixed

diesal11 · 2021-06-29T07:47:17Z

~~Any reason a fix can't be backported to Yarn V2?~~

~~This is causing our CI to return a false positive when one of our workspaces build or test fails (I.E. Not reporting the error).~~
~~Moving our project to a Release Candidate version of V3, which also includes breaking changes, doesn't seem like a reasonable solution here.~~

Edit: Downgrading to Node 14 seems to be a temporary fix until Yarn V3 is stable/launched.

akphi · 2021-08-19T02:26:32Z

@merceyz Somehow, I still can reproduce this bug.

This is my repro: akphi/issue-repo#1. I'm using Yarn@3.0.1 and Node@v14.17.1 on Mac

I setup a test monorepo with 2 workspaces pkg1 and pkg2. Each workspace has a script called doSomething. pkg1 will exit with code 1 since the command error is not known.

At the root directory, if we run the following command, we got some unexpected result:

yarn workspaces foreach run doSomething                                          # no error
yarn workspaces foreach --all run doSomething                                    # no error
yarn workspaces foreach --parallel run doSomething                               # no error
yarn workspaces foreach --all --parallel --topological-dev run doSomething       # error

Let me know if you want me to file this as a different issue.

MarsianMan added the bug Something isn't working label Feb 17, 2021

MarsianMan changed the title ~~[Bug] For each exit codes always zero~~ [Bug] Workspaces foreach always exits with zero even when erroring Feb 17, 2021

MarsianMan changed the title ~~[Bug] Workspaces foreach always exits with zero even when erroring~~ [Bug] Workspaces foreach always exits with zero even on error Feb 17, 2021

arcanis added the waiting for feedback Will autoclose in a while unless more data are provided label Feb 17, 2021

akphi mentioned this issue Mar 9, 2021

Dependencies Watchman finos/legend-studio#83

Closed

17 tasks

yarnbot added the stale Issues that didn't get attention label Apr 16, 2021

merceyz added upholded Real issues without formal reproduction and removed waiting for feedback Will autoclose in a while unless more data are provided labels Apr 17, 2021

merceyz closed this as completed Apr 17, 2021

akphi mentioned this issue Aug 19, 2021

Bug: Yarn workspace foreach do not exit when sub-process fails akphi/issue-repo#1

Closed

akphi mentioned this issue Aug 23, 2021

[Bug]: yarn workspaces foreach always exits with status 0 even on error #3334

Closed

1 task

akphi mentioned this issue Apr 18, 2022

Bug: Yarn workspace foreach do not exit when sub-process fails akphi/issue-repo#4

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] Workspaces foreach always exits with zero even on error #2486

[Bug] Workspaces foreach always exits with zero even on error #2486

MarsianMan commented Feb 17, 2021 •

edited

Loading

arcanis commented Feb 17, 2021

arcanis commented Feb 17, 2021 •

edited by merceyz

Loading

MarsianMan commented Feb 17, 2021 •

edited

Loading

MarsianMan commented Feb 17, 2021

MarsianMan commented Mar 1, 2021

DanielOrtel commented Mar 2, 2021

bfricka commented Mar 3, 2021

jasonwilliams commented Mar 17, 2021 •

edited

Loading

merceyz commented Mar 17, 2021

jasonwilliams commented Mar 17, 2021 •

edited

Loading

merceyz commented Mar 17, 2021

yarnbot commented Apr 16, 2021

merceyz commented Apr 17, 2021

diesal11 commented Jun 29, 2021 •

edited

Loading

akphi commented Aug 19, 2021 •

edited

Loading

[Bug] Workspaces foreach always exits with zero even on error #2486

[Bug] Workspaces foreach always exits with zero even on error #2486

Comments

MarsianMan commented Feb 17, 2021 • edited Loading

arcanis commented Feb 17, 2021

arcanis commented Feb 17, 2021 • edited by merceyz Loading

MarsianMan commented Feb 17, 2021 • edited Loading

MarsianMan commented Feb 17, 2021

MarsianMan commented Mar 1, 2021

DanielOrtel commented Mar 2, 2021

bfricka commented Mar 3, 2021

jasonwilliams commented Mar 17, 2021 • edited Loading

merceyz commented Mar 17, 2021

jasonwilliams commented Mar 17, 2021 • edited Loading

merceyz commented Mar 17, 2021

yarnbot commented Apr 16, 2021

merceyz commented Apr 17, 2021

diesal11 commented Jun 29, 2021 • edited Loading

akphi commented Aug 19, 2021 • edited Loading

MarsianMan commented Feb 17, 2021 •

edited

Loading

arcanis commented Feb 17, 2021 •

edited by merceyz

Loading

MarsianMan commented Feb 17, 2021 •

edited

Loading

jasonwilliams commented Mar 17, 2021 •

edited

Loading

jasonwilliams commented Mar 17, 2021 •

edited

Loading

diesal11 commented Jun 29, 2021 •

edited

Loading

akphi commented Aug 19, 2021 •

edited

Loading