Memory leak when running with Jest

[kirillgroshkov]

We use @google-cloud/datastore in our unit tests with Jest.

We discovered that just importing @google-cloud/datastore (as const { Datastore } = require('@google-cloud/datastore')) leaks memory when running in Jest tests.

I've created a minimal repro here to showcase the issue: https://github.com/kirillgroshkov/datastore-jest-leak-repro

Good to note that Jest has memory leak issues with many libraries, not just this one. Some popular libraries, e.g graceful-fs were "fixed" to not leak memory with Jest (given that Jest itself is also a "popular library", number 1 in "StateOfJS 2018" to be precise).

I will open and link a symmetric issue in Jest repo to track, cause I'm expecting some back-and-forth between this repo and Jest. (UPD: jestjs/jest#8247)

Why this issue is important - it blocks us to run our test suite in our CI environment (CircleCI), which has 4Gb memory constraint. It runs out-of-memory already with ~150 test files that we have in our project. See steps to reproduce further.

Some related issues in Jest:
jestjs/jest#6814
jestjs/jest#6738
jestjs/jest#7311
jestjs/jest#6399

Environment details

• OS: MacOS
• Node.js version: v10.15.0
• npm version: 6.9.0
• @google-cloud/datastore version: 3.1.2

Steps to reproduce

Use minimal repro repository: https://github.com/kirillgroshkov/datastore-jest-leak-repro
Run npm run test-leaking or npm run test-leaking-detectLeaks to see the issue.

[bcoe]

@kirillgroshkov thanks for the bug report and reproduction 👍

With the other memory leaks you've read about, e.g., graceful-fs, is it that Jest surfaced legitimate memory leaks in the constructor, or that Jest runs libraries in a unorthodox manner?

[kirillgroshkov]

Thanks to @scotthovestadt, he found the root cause of memory leak: jestjs/jest#8247 (comment)

So, what can we do here in @google-cloud/datastore to fix it? Or should someone reach to https://www.npmjs.com/package/long ?

[kirillgroshkov]

With the other memory leaks you've read about, e.g., graceful-fs, is it that Jest surfaced legitimate memory leaks in the constructor, or that Jest runs libraries in a unorthodox manner?

Sorry, I don't really understand what you mean here.

At this point I can't tell if it's something special with Jest or not, or if same memory leak is reproducible with other test runners (Mocha, etc). Haven't tried that. We're using Jest exclusively.

But please look at my comment above.

[bcoe]

@kirillgroshkov thanks for all the great info 👍 and @scotthovestadt thanks for digging into the issue, and finding a reproduction.

I've created a tracking issue on long and am closing this issue, as the underlying issue seems to be more related to the interaction between Jest and WASM, vs., anything specific to this library.