Use Fetch in FileLoader #22510
Use Fetch in FileLoader #22510
Conversation
I forgot about the You should be able to use the "ReadableStreams" API to progressively read chunks of the response body to get load progress and look at the response headers to check the One last thing to note is that |
|
Could ReadableStreams be popped into: switch ( this.responseType ) {
case 'arraybuffer':
// ...
case 'stream':
return res.body.getReader();Backwards compatibility is a good idea and we should probably stick to that. What was the thinking behind the onProgress event and how granular are we talking? In this PR the onProgress event is equivalent to saying "the fetch has started but you'll have to wait for the responseType you requested", it could've just as easily been "the fetch has succeeded but the response is so big that you should stream it". There's other ways of thinking what progress means too so... yeah. What would we like? |
The goal would be to provide progress callbacks consistent with XMLHTTPRequest so the granularity is up to the browser and how often the reader callback fires. It shouldn't be necessary to add new responseTypes to support it. I think it's best to get familiar with the current behavior of the onProgress callback. You can check out the MDN docs for the XMLHTTPRequest "progress" callback here. The event can get fired multiple times over the course of a download with updates to download percent and gets fired regardless of the set |
|
@gkjohnson Could you help me with sorting out the progress properly. I think this is approximately what we're looking for but I can't just throw it above the const callbacks = loading[ url ];
const reader = response.body.getReader()
let loaded = 0
let total = parseInt(response.headers.get("Content-Length"))
const progress = reader => {
return reader.read()
.then(({done, value}) => {
if (done) return;
loaded += value.length
const event = new ProgressEvent('progress', {lengthComputable: true, loaded, total})
for ( let i = 0, il = callbacks.length; i < il; i ++ ) {
const callback = callbacks[ i ];
if ( callback.onProgress ) callback.onProgress(event);
}
return progress(reader);
});
}
progress(reader) |
|
From the MDN writeup I linked you can structure a fetch to create a new response object like so: fetch( 'url.ext' )
.then( response => {
const reader = response.body.getReader();
let loadedData = 0;
// periodically read data into the new stream tracking while download progress
return new ReadableStream( {
start( controller ) {
readData();
function readData() {
reader.read().then( info => {
if ( info.done ) {
controller.close()
} else {
loadedData += info.value.byteLength;
/*
* fire on progress event here
*/
controller.enqueue( info.value );
readData();
}
} );
}
}
} );
} )
.then( stream => {
// create a new response and read the data as the requested format
const response = new Response( stream );
switch ( responseType ) {
case 'blob':
return response.blob();
/* ... */
}
} );I wonder if there are any performance implications to using this setup, though. @mrdoob are there any browser-knowledgeable people to get in touch with who would know if there are any pitfalls here compared to XMLHTTPRequest? |
|
thank you @gkjohnson, I was looking into that article but my brain was struggling to wrap itself around the |
|
Alright, lets see how this goes! 🤞 |
|
Thanks! |
not quite stellar, now my progress bars are broken. because, unlike with XHR, the server now has to explicitly give permission to read 'Content-Length' header - see https://stackoverflow.com/a/48266945 |
|
bummer. Not sure what we could use in the way of alternatives for getting the length. Any suggestions? |
|
Good to know about the HTTP header requirement. One benefit of the change to |
try xhr head request, maybe it will work (also the problem is limited to cross-origin requests, so you might want to match url to window.location) |
|
This doesn't seem to be how MDN describes the behavior of Access-Control-Expose-Headers, though. According to those docs it's only relevant for CORS requests:
And even then there is a safe list of headers that should be exposed by default even when @makc can you put together a jsfiddle repro? |
|
actually now when I test it again, I only get 0 in firefox - chrome does return the size |
|
safari also reports 0 |
Which version of Firefox are you using? I'm seeing the fiddle work in both Firefox (v94.0) and Chrome (94.0.4606.81) on Windows 10. |
|
😔 good point, I need to re-test on more recent os |
|
An issue I'm facing with this is that polyfills like GitHub's fetch don't have streaming support (related issue), and those that do implement it in a very limited capacity. That can pair with this PR to break in environments like react-native/node which depend on that polyfill or relied on Is there a way to fake the old FileLoader or add backwards-compat here? Alternatively, and somewhat offtopic, is there a good polyfill for this that follows the spec? |
|
From that issue it looks like a fork has been produced to support the react-native use case with streams, though it looks like it may only be limited to supporting text: https://www.npmjs.com/package/react-native-fetch-api And if node is something we want to support we could only use a ReadableStream if an "onProgress" callback is provided and is supported by the browser so at least it doesn't fail in contexts without ReadableStream support. |
|
Fetch API cannot load 'file' url scheme. Such as use in webview |
I use three.js in puppeteer to load file in local too. although I set '--disable-web-security' in browser. fetch API still can't fetch URL scheme |
|
Use |
Related issues: #22468, #21286
Use fetch in FileLoader
Related comment