Wasm workers (#12833)

* Add Wasm Workers

Add Wasm Workers

* Add TLS test, ES6ify.

* Update test

* Add TLS support to Wasm Workers

* Add C++11 thread_local keyword test.

* Add test for C11 _Thread_local.

* Add emscripten_malloc_wasm_worker and rename creation API a little.

* Add documentation for Wasm Workers.

* Flake and lint and fix build error

* Remove deps_info dependency that does not work in current setup

* __builtin_wasm_tls_align() can be zero

* Add more notes about __builtin_wasm_tls_align() being zero

* Add test for GCC __thread keyword.

* Fix test_wasm_worker_malloc

* Fix emscripten_lock_async_acquire()

* Fix thread stack creation.

* Fix wasm64 build

* Add slack to lock_busyspin_wait_acquire

* Fix typo in setting

* Remove removal of TextDecoder in threads.

* Fix non-Wasm Workers build

* Fix file system case sensitivity

* Fix Wasm Workers proxying mode generation.

* Skip TLS tests on Linux, they produce an internal compiler error.

* Fix typo

* Fix wasm_worker.h include from C code.

* Add library_wasm_worker_stub.c.

* Wasm Workers working on default runtime.

* flake

* Disable most wasm workers tests to debug CI

* Fix non-minimal runtime wasm workers startup. Add test for WASM_WORKERS=2 build mode.

* Simplify in MINIMAL_RUNTIME preamble assignment for wasm maximum memory.

* Fix USE_PTHREADS+WASM_WORKERS line.

* Add support for simultaneous pthreads + Wasm workers.

* Do not pass redundant TLS size to Wasm Worker creation side.

* Update emcc.py wasm worker deps

* Remove special handling of .S files in system_libs build

* Update documentation

* Add code size test.

* flake

* Update tests and wasm worker MT build

* Fix mt build

* Adjust mt build

* Update code size test

* Update hello worker wasm

* flake

* Address review: Allow building with -sSHARED_MEMORY and add a test. Move code from emcc.py to library_wasm_worker.js.

* Remove unnecessary dynCall statements

* Update mention of C11 and C++11 Atomics APIs

* Remove old code.

* Utilize runOnMainThread() in MINIMAL_RUNTIME ready handler.

* Simplify code

* #error quotes

* Clean typo

* Cleanup tests

* Update ChangeLog

* Fixes

* Add test files.

* Fix pthreads

* Remove moved test

* Address review

* Small code size optimization

* Small code size opt

* Flake

* Update Wasm Workers code size test

.eslintrc.yml

@@ -21,6 +21,7 @@ ignorePatterns:
   - "src/embind/"
   - "src/emrun_postjs.js"
   - "src/worker.js"
+  - "src/wasm_worker.js"
   - "src/wasm2js.js"
   - "src/webGLClient.js"
   - "src/webGLWorker.js"

ChangeLog.md

@@ -26,6 +26,10 @@ See docs/process.md for more on how version tagging works.
 - Fix deadlock in `munmap` that was introduced in 3.1.5.  The deadlock would
   occur in multi-threaded programs when a partial unmap was requested (which
   emscripten does not support). (#16413)
+- Added new compiler+linker option -sSHARED_MEMORY=1, which enables targeting
+  a shared WebAssembly.Memory. (#16419)
+- Added new API "Wasm Workers", which is an alternative to pthreads for building
+  multithreaded applications, enabled via -sWASM_WORKERS=1 (#12833)
 
 3.1.6 - 02/24/2022
 ------------------

emcc.py

@@ -833,6 +833,9 @@ def get_cflags(user_args):
   if settings.SHARED_MEMORY:
     cflags.append('-D__EMSCRIPTEN_SHARED_MEMORY__=1')
 
+  if settings.WASM_WORKERS:
+    cflags.append('-D__EMSCRIPTEN_WASM_WORKERS__=1')
+
   if not settings.STRICT:
     # The preprocessor define EMSCRIPTEN is deprecated. Don't pass it to code
     # in strict mode. Code should use the define __EMSCRIPTEN__ instead.
@@ -1424,11 +1427,17 @@ def phase_setup(options, state, newargs, user_settings):
   if settings.MAIN_MODULE or settings.SIDE_MODULE:
     settings.RELOCATABLE = 1
 
-  if settings.USE_PTHREADS:
+  # Pthreads and Wasm Workers require targeting shared Wasm memory (SAB).
+  if settings.USE_PTHREADS or settings.WASM_WORKERS:
     settings.SHARED_MEMORY = 1
 
-  if settings.SHARED_MEMORY and '-pthread' not in newargs:
+  if settings.USE_PTHREADS and '-pthread' not in newargs:
     newargs += ['-pthread']
+  elif settings.SHARED_MEMORY:
+    if '-matomics' not in newargs:
+      newargs += ['-matomics']
+    if '-mbulk-memory' not in newargs:
+      newargs += ['-mbulk-memory']
 
   if 'DISABLE_EXCEPTION_CATCHING' in user_settings and 'EXCEPTION_CATCHING_ALLOWED' in user_settings:
     # If we get here then the user specified both DISABLE_EXCEPTION_CATCHING and EXCEPTION_CATCHING_ALLOWED
@@ -2064,6 +2073,18 @@ def phase_linker_setup(options, state, newargs, user_settings):
   if settings.INCLUDE_FULL_LIBRARY and not settings.DISABLE_EXCEPTION_CATCHING:
     settings.EXPORTED_FUNCTIONS += ['_emscripten_format_exception', '_free']
 
+  if settings.WASM_WORKERS:
+    # TODO: After #15982 is resolved, these dependencies can be declared in library_wasm_worker.js
+    #       instead of having to record them here.
+    wasm_worker_imports = ['_emscripten_wasm_worker_initialize']
+    settings.EXPORTED_FUNCTIONS += wasm_worker_imports
+    building.user_requested_exports.update(wasm_worker_imports)
+    settings.DEFAULT_LIBRARY_FUNCS_TO_INCLUDE += ['_wasm_worker_initializeRuntime']
+    # set location of Wasm Worker bootstrap JS file
+    if settings.WASM_WORKERS == 1:
+      settings.WASM_WORKER_FILE = unsuffixed(os.path.basename(target)) + '.ww.js'
+    settings.JS_LIBRARIES.append((0, shared.path_from_root('src', 'library_wasm_worker.js')))
+
   if settings.FORCE_FILESYSTEM and not settings.MINIMAL_RUNTIME:
     # when the filesystem is forced, we export by default methods that filesystem usage
     # may need, including filesystem usage from standalone file packager output (i.e.
@@ -2803,8 +2824,8 @@ def phase_final_emitting(options, state, target, wasm_target, memfile):
   # src = re.sub(r'\n+[ \n]*\n+', '\n', src)
   # write_file(final_js, src)
 
+  target_dir = os.path.dirname(os.path.abspath(target))
   if settings.USE_PTHREADS:
-    target_dir = os.path.dirname(os.path.abspath(target))
     worker_output = os.path.join(target_dir, settings.PTHREAD_WORKER_FILE)
     contents = shared.read_and_preprocess(utils.path_from_root('src/worker.js'), expand_macros=True)
     write_file(worker_output, contents)
@@ -2814,6 +2835,17 @@ def phase_final_emitting(options, state, target, wasm_target, memfile):
       minified_worker = building.acorn_optimizer(worker_output, ['minifyWhitespace'], return_output=True)
       write_file(worker_output, minified_worker)
 
+  # Deploy the Wasm Worker bootstrap file as an output file (*.ww.js)
+  if settings.WASM_WORKERS == 1:
+    worker_output = os.path.join(target_dir, settings.WASM_WORKER_FILE)
+    with open(worker_output, 'w') as f:
+      f.write(shared.read_and_preprocess(shared.path_from_root('src', 'wasm_worker.js'), expand_macros=True))
+
+    # Minify the wasm_worker.js file in optimized builds
+    if (settings.OPT_LEVEL >= 1 or settings.SHRINK_LEVEL >= 1) and not settings.DEBUG_LEVEL:
+      minified_worker = building.acorn_optimizer(worker_output, ['minifyWhitespace'], return_output=True)
+      open(worker_output, 'w').write(minified_worker)
+
   # track files that will need native eols
   generated_text_files_with_native_eols = []
 

site/source/docs/api_reference/index.rst

@@ -21,6 +21,9 @@ This section lists Emscripten's public API, organised by header file. At a very
 - :ref:`Fetch-API`:
   API for managing accesses to network XHR and IndexedDB.
 
+- :ref:`wasm_workers`:
+  Enables writing multithreaded programs using a web-like API.
+
 - :ref:`Module`:
   Global JavaScript object that can be used to control code execution and access exported methods.
 
@@ -53,6 +56,7 @@ This section lists Emscripten's public API, organised by header file. At a very
    bind.h
    trace.h
    fiber.h
+   wasm_workers
    advanced-apis