Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supporting vitest tracking issue #23882

Open
marvinhagemeister opened this issue May 18, 2024 · 26 comments
Open

Supporting vitest tracking issue #23882

marvinhagemeister opened this issue May 18, 2024 · 26 comments

Comments

@marvinhagemeister
Copy link
Contributor

marvinhagemeister commented May 18, 2024

This is an issue to track the progress on getting the vitest test runner to work. I'm currently using the test suite from urql to check our progress.

@birkskyum
Copy link
Contributor

birkskyum commented Jun 30, 2024

With the standard example it appear to still (deno 1.44.4+bc8a0e6) hang at:

➜ deno task test              
Task test vitest

 DEV  v1.6.0 

Both w/o DENO_FUTURE=1

nathanwhit added a commit that referenced this issue Jul 30, 2024
Fixes #24756. Fixes
#24796.

This also gets vitest working when using
[`--pool=forks`](https://vitest.dev/guide/improving-performance#pool)
(which is the default as of vitest 2.0). Ref
#23882.

---

This PR resolves a handful of issues with child_process IPC. In
particular:

- We didn't support sending typed array views over IPC
- Opening an IPC channel resulted in the event loop never exiting
- Sending a `null` over IPC would terminate the channel
- There was some UB in the read implementation (transmuting an `&[u8]`
to `&mut [u8]`)
- The `send` method wasn't returning anything, so there was no way to
signal backpressure (this also resulted in the benchmark
`child_process_ipc.mjs` being misleading, as it tried to respect
backpressure. That gave node much worse results at larger message sizes,
and gave us much worse results at smaller message sizes).
- We weren't setting up the `channel` property on the `process` global
(or on the `ChildProcess` object), and also didn't have a way to
ref/unref the channel
- Calling `kill` multiple times (or disconnecting the channel, then
calling kill) would throw an error
- Node couldn't spawn a deno subprocess and communicate with it over IPC
crowlKats pushed a commit that referenced this issue Jul 31, 2024
Fixes #24756. Fixes
#24796.

This also gets vitest working when using
[`--pool=forks`](https://vitest.dev/guide/improving-performance#pool)
(which is the default as of vitest 2.0). Ref
#23882.

---

This PR resolves a handful of issues with child_process IPC. In
particular:

- We didn't support sending typed array views over IPC
- Opening an IPC channel resulted in the event loop never exiting
- Sending a `null` over IPC would terminate the channel
- There was some UB in the read implementation (transmuting an `&[u8]`
to `&mut [u8]`)
- The `send` method wasn't returning anything, so there was no way to
signal backpressure (this also resulted in the benchmark
`child_process_ipc.mjs` being misleading, as it tried to respect
backpressure. That gave node much worse results at larger message sizes,
and gave us much worse results at smaller message sizes).
- We weren't setting up the `channel` property on the `process` global
(or on the `ChildProcess` object), and also didn't have a way to
ref/unref the channel
- Calling `kill` multiple times (or disconnecting the channel, then
calling kill) would throw an error
- Node couldn't spawn a deno subprocess and communicate with it over IPC

(cherry picked from commit cd59fc5)
@birkskyum
Copy link
Contributor

birkskyum commented Jul 31, 2024

After this PR ( #24763 ) I tried this again with deno 1.45.5, and these scenarios does indeed appear to work now:

  • using "--pool=forks" with Vitest 1.x w/ DENO_FUTURE=1
  • using Vitest 2.x w/ DENO_FUTURE=1

Scenarios that doesn't work out-of-the-box, but which will resolve over time / has workarounds:

  • using Deno 1.x without the DENO_FUTURE=1 flag
  • using vitest 1.x without the --pool=forks flag

@birkskyum
Copy link
Contributor

birkskyum commented Sep 10, 2024

Alright, so I'm on Vitest 2 (it defaults to pool: forks) and Deno 2 (deno 2.0.0-rc.1+7bfcb4d) now.

Repro

deno.json

{
  "nodeModulesDir": "auto",
  "tasks": {
    "test": "vitest"
  },
  "imports": { "jsdom": "npm:jsdom@^25.0.0", "vitest": "npm:vitest@^2.0.5" }
}

sum.test.ts

import { expect, test } from "vitest";

export function sum(a, b) {
  return a + b
}

test('adds 1 + 2 to equal 3', () => {
  expect(sum(1, 2)).toBe(3)
})

vitest.config.ts

import {defineConfig} from "vitest/config";

export default defineConfig({
  test: {
    environment: 'jsdom'
  }
})

Run

  • deno install
  • deno task test

Error:

 DEV  v2.0.5 /Users/admin/repos/deno-kitchensink/vitest-demo


⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

Vitest caught 1 unhandled error during the test run.
This might cause false positive tests. Resolve unhandled errors to make sure your tests are not affected.

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Error ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
TypeError: Cannot redefine property: location
 ❯ populateGlobal node_modules/.deno/vitest@2.0.5/node_modules/vitest/dist/chunks/index.lVXYBqEP.js:374:12
 ❯ Object.setup node_modules/.deno/vitest@2.0.5/node_modules/vitest/dist/chunks/index.lVXYBqEP.js:526:33
 ❯ withEnv node_modules/.deno/vitest@2.0.5/node_modules/vitest/dist/chunks/runBaseTests.CyvqmuC9.js:92:15
 ❯ run node_modules/.deno/vitest@2.0.5/node_modules/vitest/dist/chunks/runBaseTests.CyvqmuC9.js:116:3
 ❯ runBaseTests node_modules/.deno/vitest@2.0.5/node_modules/vitest/dist/chunks/base.CC5R_kgU.js:31:3
 ❯ ForksBaseWorker.executeTests node_modules/.deno/vitest@2.0.5/node_modules/vitest/dist/workers/forks.js:25:7
 ❯ execute node_modules/.deno/vitest@2.0.5/node_modules/vitest/dist/worker.js:115:5
 ❯ onMessage node_modules/.deno/tinypool@1.0.1/node_modules/tinypool/dist/entry/process.js:55:20

⎯⎯⎯⎯⎯⎯⎯⎯

If I remove environment: "jsdom" it works:

 DEV  v2.0.5 /Users/admin/repos/deno-kitchensink/vitest-demo

 ✓ sum.test.ts (1)
   ✓ adds 1 + 2 to equal 3

 Test Files  1 passed (1)
      Tests  1 passed (1)
   Start at  23:47:08
   Duration  170ms (transform 15ms, setup 0ms, collect 15ms, tests 1ms, environment 0ms, prepare 46ms)

@birkskyum
Copy link
Contributor

birkskyum commented Sep 22, 2024

Commenting out the "location" key from jsdom-keys here will actually make the test pass, so it's pretty close - seems to just be the location that's left. The location error is that vitest can't re-declare or delete it on the global object. Does Deno set a location globally that i.e. Node doesn't do?

deno task test
Task test vitest

 DEV  v2.1.1 /Users/admin/repos/deno-kitchensink/vitest-testsum.test.ts (1)
   ✓ adds 1 + 2 to equal 3

 Test Files  1 passed (1)
      Tests  1 passed (1)
   Start at  00:37:05
   Duration  473ms (transform 15ms, setup 0ms, collect 15ms, tests 1ms, environment 317ms, prepare 41ms)


 PASS  Waiting for file changes...
       press h to show help, press q to quit

@bartlomieju
Copy link
Member

Yes, globalThis.location is undefined and writeable in Deno (unless you run with --location flag). So vitest should be able to redeclare it.

@birkskyum
Copy link
Contributor

birkskyum commented Sep 22, 2024

It uses a Object.defineProperty(global, 'location', value) syntax and a delete global2['location']; and they both break. When it says it's a TypeError, is it then the compilation that breaks, and not the code running?

@bartlomieju
Copy link
Member

bartlomieju commented Sep 22, 2024

Executing script like this:

console.log(globalThis.location);
Object.defineProperty(globalThis, "location", { value: "https://deno.com" });
console.log(globalThis.location);
delete globalThis["location"];
console.log(globalThis.location);

I get this on Deno v2.0.0-rc.4

undefined
https://deno.com
error: Uncaught (in promise) TypeError: Cannot delete property 'location' of #<Window>
delete globalThis["location"];
^
    at file:///Users/ib/dev/deno/try.js:4:1

So indeed the deletion code might be the problem here.

@birkskyum could you provide a reproduction how you got this error?

@birkskyum
Copy link
Contributor

@bartlomieju , it's by following these steps.

@bartlomieju
Copy link
Member

Sorry, missed that. I can indeed reproduce the problem here. PR incoming, should be a simple fix provided that WPT doesn't complain.

@bartlomieju
Copy link
Member

See #25812

bartlomieju added a commit that referenced this issue Sep 23, 2024
This commit changes `globalThis.location` property to be configurable
so that packages wanting to override it (or delete it) work properly.

Towards #23882

This change makes reproduction from
#23882 (comment)
pass properly.
@marvinhagemeister
Copy link
Contributor Author

Using the reproduction posted earlier here #23882 (comment) the test run passes in watch mode without issues. But when running the tests with vitest's run command like deno task test run, the program exits with:

error: Uncaught (in promise) BrokenPipe: Broken pipe (os error 32)

@birkskyum
Copy link
Contributor

birkskyum commented Oct 18, 2024

any idea why the test run is breaking?

@bartlomieju
Copy link
Member

@birkskyum do you get any error there?

@birkskyum
Copy link
Contributor

birkskyum commented Oct 18, 2024

@bartlomieju i get the one here #23882 (comment)

@birkskyum
Copy link
Contributor

it runs the test fine, but gives this error after:

Screenshot 2024-10-19 at 00 18 10

@bartlomieju
Copy link
Member

I believe this might be related to IPC pipes handling that @nathanwhit worked on a few weeks back.

nathanwhit added a commit that referenced this issue Oct 24, 2024
Fixes the issue described in
#23882 (comment).

The parent was starting to send a message right before the process would
exit, and the channel closed in the middle of the write. Unlike with
reads, we weren't cancelling the pending writes, which resulted in a
`Broken pipe` error surfacing to the user.
@nathanwhit
Copy link
Member

any idea why the test run is breaking?

@birkskyum this should be fixed on canary now

@birkskyum
Copy link
Contributor

@nathanwhit , awesome! tried it successfully just now, thanks!

@birkskyum
Copy link
Contributor

birkskyum commented Oct 24, 2024

With basic deno task test run working, I tried was using it for Coverage Reporting deno task test run --coverage and it appear to break with:

⇣9% ➜ deno run test run --coverage
Task test vitest "run" "--coverage"
Loaded  vitest@2.1.1  and  @vitest/coverage-v8@2.1.3 .
Running mixed versions is not supported and may lead into bugs
Update your dependencies and make sure the versions match.

 RUN  v2.1.1 
      Coverage enabled with v8

Vitest caught 1 unhandled error during the test run.
This might cause false positive tests. Resolve unhandled errors to make sure your tests are not affected.

Error: Not implemented: inspector.Session.prototype.connect
 ❯ notImplemented ext:deno_node/_utils.ts:9:9
 ❯ Session.connect node:inspector:12:5
 ❯ Object.startCoverage node_modules/.deno

There's a ticket for that specifically in here too:

@marvinhagemeister
Copy link
Contributor Author

marvinhagemeister commented Oct 24, 2024

This error occurs because our current node:inspector code is mostly stubs. The proper implementation is being worked on here #26471

bartlomieju pushed a commit that referenced this issue Oct 25, 2024
Fixes the issue described in
#23882 (comment).

The parent was starting to send a message right before the process would
exit, and the channel closed in the middle of the write. Unlike with
reads, we weren't cancelling the pending writes, which resulted in a
`Broken pipe` error surfacing to the user.
@birkskyum
Copy link
Contributor

birkskyum commented Oct 26, 2024

Hit a new issue today with Deno + Vitest + SolidJS

@birkskyum
Copy link
Contributor

birkskyum commented Nov 10, 2024

for deno 2.0.6

@lishaduck
Copy link

Coverage no longer crashes! It currently reports it all as zero, but that's better than it was!

See #27003.

@lishaduck
Copy link

lishaduck commented Nov 24, 2024

Bumped into another issue:
If you use a package.json + .npmrc to resolve JSR deps, everything works fine (they're put in node_modules), but when using a deno.json and native jsr: specifiers, they don't get put in the node_modules folder, so Vitest fails, even when using npm:@deno/vite-plugin. I thought it might be because I'm using workspaces (because of denoland/deno-vite-plugin#19), but hoisting the dependency in question (jsr:@std/path) to the root didn't help, so I assume it's an issue with the Vitest runner. If not, what repo should I file this in?

@birkskyum
Copy link
Contributor

birkskyum commented Nov 24, 2024

Bumped into another issue: If you use a package.json + .npmrc to resolve JSR deps, everything works fine (they're put in node_modules), but when using a deno.json and native jsr: specifiers, they don't get put in the node_modules folder, so Vitest fails, even when using npm:@deno/vite-plugin. I thought it might be because I'm using workspaces (because of denoland/deno-vite-plugin#19), but hoisting the dependency in question (jsr:@std/path) to the root didn't help, so I assume it's an issue with the Vitest runner. If not, what repo should I file this in?

@lishaduck Does it help to add in the deno.json { "nodeModulesDir": "auto" }, to generate the node_modules (docs)?

@lishaduck
Copy link

@lishaduck Does it help to add in the deno.json { "nodeModulesDir": "auto" }, to generate the node_modules (docs)?

I'm using "nodeModulesDir": "manual", but I installed via deno install first.
Here's some ci logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants