Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make webworker-threads a peer-dependency #368

Closed
OoDeLally opened this issue Aug 23, 2017 · 16 comments
Closed

Make webworker-threads a peer-dependency #368

OoDeLally opened this issue Aug 23, 2017 · 16 comments

Comments

@OoDeLally
Copy link

I'm running natural in Electron.
Natural crashes when imported on electron, because it tries to use webworker-threads, which only works on node.
It works fine when I delete the webworker-threads folder from node_modules folder. But this is tedious.
As far as I know there is no way to specify in my package.json not to install the webworker-threads optional dependency of natural (this seems to exist only on command-line with --no-optional).

A solution to that could be to set webworker-threads as peer-dependencies instead of optional-dependencies in the package.json. The user would have the choice to include webworker-threads in the case he wants it.

Also, electron also has builtin webworkers, so it would be nice if natural can use them. But for now, it would be quite a low-hanging fruit not to use them at all.

@caseywreed
Copy link

This comment helped me a ton. Was trying to get natural running with an AWS Lambda function and AWS crashed because of webworker-threads. Deleted the folder and I was good to go.

For a further explanation of the problem, I found this great blog post. It might help out, too. Thanks again!

@theSiberman
Copy link

Same issue running on IBM OpenWhisk

@kkoch986 kkoch986 self-assigned this Sep 29, 2017
@kkoch986
Copy link
Member

makes sense to me ill set that up asap

@rawpixel-vincent
Copy link
Contributor

On server side, I had to delete the webworker-threads folder in node_modules before starting the app. Otherwise I got a runtime error.

@copitz
Copy link

copitz commented Mar 1, 2018

👍

However, @OoDeLally, electron-rebuild seems to help with this

@leonardorifeli
Copy link

leonardorifeli commented Apr 25, 2018

I'm have the same problem in my function (aws lambda).

2018-04-25T03:43:07.586Z	c1686f76-483a-11e8-a644-4958112b2bf6	Error: /var/task/node_modules/webworker-threads/build/Release/WebWorkerThreads.node: undefined symbol: _ZN2v811ArrayBuffer9Allocator7ReserveEm
at Error (native)
at Object.Module._extensions..node (module.js:597:18)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
at Module.require (module.js:497:17)
at require (internal/module.js:20:19)
at bindings (/var/task/node_modules/bindings/bindings.js:81:44)
at Object.<anonymous> (/var/task/node_modules/webworker-threads/index.js:1:105)
at Module._compile (module.js:570:32)

But, I'm using circleci to deploy my function.

image

I'm can not send node_modules because I'm versioning the function with github. It's not a good pattern.

@caseywreed / @kkoch986 do you have a suggestion for that?

@leonardorifeli
Copy link

@caseywreed / @kkoch986 I solved my problem.

In the circleci configuration file, I set it to use the lambci / lambda: build-nodejs6.10 docker image

docker:
     - image: lambci / lambda: build-nodejs6.10

Thanks.

@3rd-Eden
Copy link

webworker-threads should be completely opt-in, I don't want to install a native dependency just because I try to a simple NLP task.

@Hugo-ter-Doest
Copy link
Collaborator

What does that mean for the dependencies in package.json. Should it be in optional-dependencies or in peer-dependencies?

Hugo

@3rd-Eden
Copy link

Optional dependencies are still being installed by default, but npm will not throw a tantrum when it fails to compile or install. With peerDependencies they not installed, but the user is presented with a warning if they do no install the said dependency them selfs upon installation. Last option is non of the above and remove the dependency from the package.json completely. If people want to leverage the webworker-threads package they can install it them selfs. npm flattens the dependency tree by default so if a user installs it, then the natural package will be able to access it as well.

The last option is what we've decided implement for the ws library (which I maintain) as well as it has optional binary modules that can increase speed and validation of websocket frames. It still functions without it. We tried optionalDependencies as well, but found that in rare cases it would still stop the npm installation process when the compilation failed.

@Hugo-ter-Doest
Copy link
Collaborator

Ok. I simplified things as follows. I removed the webworker-threads dependency from package.json. The parallel methods of classifier are now only available if the webworker-threads could be loaded. So as a developer, if you want to call a parallel method like trainParallel, you first have to check if it is available. Is this going to work in other frameworks and in browsers? Please let me know.

Hugo

@Hugo-ter-Doest
Copy link
Collaborator

PS Downside of this is that the parallel methods are not tested. If you want to test this, you first have to install webworker-threads manually and then run npm test again.

Hugo

@Hugo-ter-Doest
Copy link
Collaborator

I think this is issue is now solved. If not, please comment.

Hugo

@3rd-Eden
Copy link

3rd-Eden commented May 1, 2018

You could still run the tests if you want, e.g:

npm uninstall webworker-threads && npm test && npm install webworker-threads && npm test

So at least you test if works with, and without the module.

@dcsan
Copy link

dcsan commented Aug 5, 2019

is there a specific release of the npm version that fixes this? I still get the problem on ubuntu, but seem to be able to get it to compile on OSX.

it's failing for me with


>   "version": "0.6.3",

the compile ( `npm i` ) does seem to work now but get errirs at startup

npm WARN rap@1.0.0 No description
npm WARN rap@1.0.0 No repository field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.9 (node_modules/jest-haste-map/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.9 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: webworker-threads@0.7.17 (node_modules/sentence-similarity/node_modules/natural/node_modules/webworker-threads):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: webworker-threads@0.7.17 install: `node-gyp rebuild`
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: Exit status 1

added 1221 packages from 1102 contributors and audited 877612 packages in 43.617s
found 6 vulnerabilities (1 low, 4 moderate, 1 high)
  run `npm audit fix` to fix them, or `npm audit` for details
root@bot-wiz2:/mnt/ext250/web-apps/rap/server# npm run plain

> rap@1.0.0 plain /mnt/ext250/web-apps/rap/server
> APPNAME=rap PORT=6835 DEBUG=*,-express*,-nodemon*,-superagent,-AmpMetrics,-body-parser* node server.js

Importing data...
Combining data...
Building search index...
Ready!
/mnt/ext250/web-apps/rap/server/node_modules/sentence-similarity/node_modules/natural/lib/natural/classifiers/classifier.js:32
    if (e.code !== 'MODULE_NOT_FOUND') throw e;
                                       ^

Error: Could not locate the bindings file. Tried:
 → /mnt/ext250/web-apps/rap/node_modules/webworker-threads/build/WebWorkerThreads.node
 → /mnt/ext250/web-apps/rap/node_modules/webworker-threads/build/Debug/WebWorkerThreads.node
 → /mnt/ext250/web-apps/rap/node_modules/webworker-threads/build/Release/WebWorkerThreads.node
 → /mnt/ext250/web-apps/rap/node_modules/webworker-threads/out/Debug/WebWorkerThreads.node
 → /mnt/ext250/web-apps/rap/node_modules/webworker-threads/Debug/WebWorkerThreads.node

@dcsan
Copy link

dcsan commented Aug 5, 2019

i tried

npm uninstall webworker-threads
npm rebuild   # which worked surprisingly

but then running my app

Combining data...
Building search index...
Ready!
/mnt/ext250/web-apps/rap/server/node_modules/sentence-similarity/node_modules/natural/lib/natural/classifiers/classifier.js:32
    if (e.code !== 'MODULE_NOT_FOUND') throw e;
                                       ^

Error: Could not locate the bindings file. Tried:
 → /mnt/ext250/web-apps/rap/node_modules/webworker-threads/build/WebWorkerThreads.node
 → /mnt/ext250/web-apps/rap/node_modules/webworker-threads/build/Debug/WebWorkerThreads.node

any other ideas appreciated, otherwise this library is basically unusable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants