Multithreading 1/N: JS scripts via Blob URLs

[juj]

Users sometimes add the main .js files via Blob URLs into the DOM. For example, Unreal Engine 4 has the following flow to implement IndexedDB based caching of downloads: (condensed greatly to remove error handling to show the guts)

function addScriptToDom(blob) {
  return new Promise(function(resolve, reject) {
    var script = document.createElement('script');
    var objectUrl = URL.createObjectURL(blob);
    script.src = objectUrl;
    script.onload = function() {
      URL.revokeObjectURL(objectUrl);
      resolve();
    }
    document.body.appendChild(script);
  });
}

function fetchOrDownloadAndStore(db, url, responseType) {
  return new Promise(function(resolve, reject) {
    fetchFromIndexedDB(db, url)
    .then(function(data) { return resolve(data); })
    .catch(function(error) {
      return download(url, responseType).then(function(data) {
        storeToIndexedDB(db, url, data).then(function() { return resolve(data); })
      })
    });
  });
}

var mainJsDownload = fetchOrDownloadAndStore(db, Module.locateFile('UE4Game.js'), 'blob').then(function(data) {
    return addScriptToDom(data).then(function() {
      addRunDependency('wait-for-compiled-code');
    });
  });

This is practically the same machinery as our -s PRECISE_F32=2 and -s USE_PTHREADS=2 support is, but for the main .js file, as opposed to the asm.js/wasm files.

The above flow however creates trouble when used with asm.js multithreading, because in that case, document.currentScript.src refers to the URL of the blob, such as blob://UUID, and that blob is constrained to the scope of the main browser thread. That is, if I URL.createObjectURL() on the main thread, and pass the received URL to a worker, then that worker cannot use that URL to load the script, but will receive a not found error. Because of that, it is necessary to pass the actual Blob over to the Worker so that it can be accessed via importScripts. I have been unable to come up with a better mechanism than allow such html files to explicitly assist the pthread creation routine with a Module['mainScriptUrlOrBlob'] field.

[curiousdannii]

Does this need to be a full URL, or a partial URL relative to Module['scriptDirectory']? (#5484)

[juj]

Module['mainScriptUrlOrBlob'] can be a relative URL, in which case it will be treated with respect to Module['scriptDirectory'], which, if relative, will be treated relative to https://developer.mozilla.org/en-US/docs/Web/HTML/Element/base.

[kripken]

could we always create a blob here and send that over?

[juj]

Technically we could, but doing that has disadvantages, the URL changes from human readable one to a blob UUID one, which affects CSP and CORS behavior, profiling, printed callstacks and logged messages, and we'd have to add a new extra code to do the XHRs in. It would be preferred to allow users to keep loading the .js files when they want to do that.

[kripken]

Got it, lgtm.

[alinpopescu]

Hello,

I have a similar problem when trying to load opencv.js compiled with flags:
-s WASM=1 -s USE_PTHREADS=1 -s ALLOW_MEMORY_GROWTH=1 -s WASM_MEM_MAX=1610612736 -s PTHREAD_POOL_SIZE=8.

I have added some multithreading module to it which I cannot make public.
First when I simply try to load opencv.js, I get these errors:

Uncaught TypeError: Failed to execute 'createObjectURL' on 'URL': No function was found that matched the signature provided.

I did some investigation and found the problem is in pthread-main.js:101 because e.data.urlOrBlob is undefined. For some reason this line evaluates to undefined:
urlOrBlob: Module["mainScriptUrlOrBlob"] || currentScriptUrl

Can you please help me fix this problem as I'm facing it for some time and I cannot sort it out?

If I compile the code without -s PTHREAD_POOL_SIZE=8 then the code loads fine, but it blocks when I run it (when the browser tries to load first thread).

Thank you very much for your help,
Alin