-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proxy known handlers back to the main thread #17925
Conversation
Reading this code, I'm not sure how it works 😄 it seems like it renames |
5396372
to
f4bf880
Compare
The reason for the emscripten/src/runtime_debug.js Lines 66 to 79 in fc3a217
Causes conflicts with the emscripten/src/library_pthread.js Line 344 in f4bf880
Since it overwrites I just split this into a separate commit with f4bf880, but this should probably be included in a seperate PR. |
} | ||
#endif | ||
} else if (cmd === 'callHandler') { | ||
Module[d['handler']](...d['args']); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if the handler is not present? (expectToReceiveOnModule
just means that it could be present)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Module[d['handler']]
check is not necessary here since it's checked below with Module.hasOwnProperty(handler)
, and likewise for those expectToReceiveOnModule
checks. Thus, this handler is called only if it's present in INCOMING_MODULE_JS_API
and if it's explicitly set on the module.
test/test_other.py
Outdated
print: (text) => console.log(text.replace('world', 'earth')) | ||
}); | ||
} | ||
main(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this needs need to be using MODULARIZE/EXPORT_NAME? Could it not just use onRuntimeInitialized
to run this code?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this bug now also effect non-wasmfs? Any not just write this test without WASMFS at all?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Simplified the test with commit 2be3a53. Note that this bug only occurs when the handler is set outside the module.
I see, thanks. Makes sense. Yes, please let's do that part in a separate PR - I am a little concerned about backwards compatibility there, as it seems like it might be a noticeable change for some? |
I just split this change to PR #17955 in a backwards compatible manner. |
043cced
to
bad877f
Compare
When running on a pthread, none of the incoming parameters on the module object are present. Proxy known handlers back to the main thread if specified. This simplifies the previous `onAbort` pthread logic for a more generic solution that also supports the `onExit`, `print` and `printErr` handlers that can be overridden on the incoming module. See: emscripten-core#17688.
bad877f
to
096a5de
Compare
Rebased now that PR #17955 has been merged. |
'print', | ||
#endif | ||
#if expectToReceiveOnModule('printErr') | ||
'printErr', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So print and printErr were previously not proxied? What happened when they were called from the thread previously?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indeed, these module handlers were previously not proxied causing the Module['print']
/Module['printErr']
handlers to be skipped (i.e. it's always printed on the terminal) on pthreads. See for example the current implementation of out
/err
:
var out = Module['print'] || defaultPrint;
var err = Module['printErr'] || defaultPrintErr;
Lines 428 to 429 in 30f50a9
{{{ makeModuleReceiveWithVar('out', 'print', 'defaultPrint', true) }}} | |
{{{ makeModuleReceiveWithVar('err', 'printErr', 'defaultPrintErr', true) }}} |
For simple printf()
usage (without using WasmFS), however, this issue did not occur, since almost all syscalls, including fd_write()
, are proxied back to the main thread.
emscripten/src/library_syscall.js
Lines 1087 to 1100 in 30f50a9
#if USE_PTHREADS | |
// Most syscalls need to happen on the main JS thread (e.g. because the | |
// filesystem is in JS and on that thread). Proxy synchronously to there. | |
// There are some exceptions, syscalls that we know are ok to just run in | |
// any thread; those are marked as not being proxied with | |
// __proxy: false | |
// A syscall without a return value could perhaps be proxied asynchronously | |
// instead of synchronously, and marked with | |
// __proxy: 'async' | |
// (but essentially all syscalls do have return values). | |
if (library[x + '__proxy'] === undefined) { | |
library[x + '__proxy'] = 'sync'; | |
} | |
#endif |
This issue occurred with WasmFS, as that implements stdout
/stderr
using _emscripten_out
/_emscripten_err
, which doesn't have this proxy logic.
emscripten/system/lib/wasmfs/special_files.cpp
Lines 110 to 116 in 30f50a9
// Node and worker threads issue in Emscripten: | |
// https://github.com/emscripten-core/emscripten/issues/14804. | |
// Issue filed in Node: https://github.com/nodejs/node/issues/40961 | |
// This is confirmed to occur when running with EXIT_RUNTIME and | |
// PROXY_TO_PTHREAD. This results in only a single console.log statement | |
// being outputted. The solution for now is to use out() and err() instead. | |
return writeToJS(buf, len, &_emscripten_out, writeBuffer); |
Lines 3364 to 3369 in 30f50a9
_emscripten_out: function(str) { | |
#if ASSERTIONS | |
assert(typeof str == 'number'); | |
#endif | |
out(UTF8ToString(str)); | |
}, |
(+ it would be a performance penalty if these were always proxied to the main thread, as noticed in #17688 (comment))
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW, this bug only occurs when the handler is set outside the module. For example, it did not occur when emitting the handlers via --pre-js
, since the pthread worker(s) will import()
that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we avoid the needless proxying in the case when the handler is defined in the --pre-js
maybe?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we avoid the needless proxying in the case when the handler is defined in the
--pre-js
maybe?
I'm not sure this can be done without too much hassle. For example, if someone does this in --pre-js
:
if (!Module.hasOwnProperty('print')) {
Module['print'] = function(x) {
console.log(x);
};
}
if (!Module.hasOwnProperty('printErr')) {
Module['printErr'] = function(x) {
console.error(x);
};
}
(i.e. set the default print
/printErr
module handlers, if it is not already overridden outside the module - commit 5c2d544 might be relevant here)
It implies that Module.hasOwnProperty('print')
and Module.hasOwnProperty('printErr')
both return true
after that.
// Use `const` here to ensure that the variable is scoped only to | ||
// that iteration, allowing safe reference from a closure. | ||
for (const handler of e.data.handlers) { | ||
Module[handler] = function() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about only setting this if handler
doesn't already exist as property of the module? If the handler is already defined on the module there should be no need to proxy it right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Those handlers (onExit
, onAbort
, print
and printErr
) are only proxied if it's present in INCOMING_MODULE_JS_API
and if it's explicitly set on the module. The latter is controlled by this:
emscripten/src/library_pthread.js
Lines 346 to 348 in 096a5de
if (Module.hasOwnProperty(handler)) { | |
handlers.push(handler); | |
} |
So, in most cases, e.data.handlers
is an empty array. Perhaps I should clarify this with a unit test?
When running on a pthread, none of the incoming parameters on the module object are present. Proxy known handlers back to the main thread if specified.
This simplifies the previous
onAbort
pthread logic for a more generic solution that also supports theonExit
,print
andprintErr
handlers that can be overridden on the incoming module.See: #17688.