Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it worker_thread compatible #168

Merged
merged 1 commit into from
Nov 12, 2020
Merged

Make it worker_thread compatible #168

merged 1 commit into from
Nov 12, 2020

Conversation

linkeo
Copy link

@linkeo linkeo commented Sep 15, 2020

In my project, I'm using worker_thread (by workerpool) to asynchronize and speed up heavy works to avoid main thread blocking.

But nodejieba is not compatible to be used with worker_thread. According to josdejong/workerpool#165, Automattic/node-canvas#1394 and nodejs/nan#844 (comment), it seems that we can simply change NODE_MODULE into NAN_MODULE_WORKER_ENALBED macro to make it compatible.

Tests:

  • npm test all test is passing
  • worker_thread compatible tested with workerpool (sorry, I don't know how to write multi-thread tests using should, I've post test code below)

// thread_test/lib.worker.js
const jieba = require("../index");
const workerpool = require("workerpool");

console.log("worker initialized");

function jiebaCutHMM(text) {
  return jieba.cutHMM(text);
}

workerpool.worker({
  jiebaCutHMM,
});
// thread_test/lib.js
const workerpool = require("workerpool");
const pool = workerpool.pool(require.resolve("./lib.worker.js"), {
  workerType: "thread",
});

function exit() {
  pool.terminate();
}

async function jiebaCutHMM(text) {
  return await pool.exec("jiebaCutHMM", [text]);
}

exports.exit = exit;
exports.jiebaCutHMM = jiebaCutHMM;
// thread_test/test.js
const lib = require('./lib');

Promise.all(
  Array.from({ length: 20 }, (v, k) => {
    const t = `上海大厦#${k + 1}`;
    const p = lib.jiebaCutHMM(t);
    console.log('jiebaCutHMM is called asynchronously, main thread is not blocking!');
    return p;
  })
)
  .then(console.log, console.log)
  .finally(() => {
    lib.exit();
  });
// output
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
jiebaCutHMM is called asynchronously, main thread is not blocking!
script end.
worker initialized
worker initialized
worker initialized
worker initialized
worker initialized
worker initialized
worker initialized
[
  [ '上海大厦', '#', '1' ],  [ '上海大厦', '#', '2' ],
  [ '上海大厦', '#', '3' ],  [ '上海大厦', '#', '4' ],
  [ '上海大厦', '#', '5' ],  [ '上海大厦', '#', '6' ],
  [ '上海大厦', '#', '7' ],  [ '上海大厦', '#', '8' ],
  [ '上海大厦', '#', '9' ],  [ '上海大厦', '#', '10' ],
  [ '上海大厦', '#', '11' ], [ '上海大厦', '#', '12' ],
  [ '上海大厦', '#', '13' ], [ '上海大厦', '#', '14' ],
  [ '上海大厦', '#', '15' ], [ '上海大厦', '#', '16' ],
  [ '上海大厦', '#', '17' ], [ '上海大厦', '#', '18' ],
  [ '上海大厦', '#', '19' ], [ '上海大厦', '#', '20' ]
]

@linkeo linkeo changed the title Support worker_thread Make it worker_thread compatible Sep 15, 2020
@McShelby
Copy link
Contributor

That's not enough to make it compatible. The code needs to get rid of the global variable. See my PR #173

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants