ReadableStream.prototype.pipeTo's first argument must be a WritableStream

[ibudisteanu]

Trying to follow example 1 https://github.com/transcend-io/conflux/blob/master/example/pipes/index.js in browser and I get ReadableStream.prototype.pipeTo's first argument must be a WritableStream. Any idea ?

[jimmywarting]

It might be possible that you are mixing both native and polyfilled whatwg streams?

eg: you are getting a native readable stream from Response.prototype.body and tries to pipe the response to a polyfilled WritableStream from web-streams-polyfill?

(they are not compatible with eachother)

can't say for sure, you need to share some of your code also.

---

If you want to be supportive give this issue a 👍 and maybe even ask if you can help out in some way. I would really like to see proper polyfill using native streams.

[ibudisteanu]

Hi Jimmy! I love the StreamSaver and I have already integrated it in my research code, but I can't integrate the zip.

I need to use your library because we believe that most of streams will be big ( > 4gb maybe 10 gb). These binary data (files) in the browser will be stored locally (index db) into chunks of binary data (buffers with 32 mb chunk). I was able to use the StreamSaver to saveAs one of these files in the browser, but I simply can't integrate the conflux (we need it because we have also directories as well).

Here is a snippet of my entire code (simplified for your review).

        const streamsaver = require('streamsaver')
        const stream = require('stream')
        const { Writer } = require("@transcend-io/conflux");
        //....

        const streamer = streamsaver.createWriteStream( name + '.zip', )

        const {readable, writable } = new Writer();
        const writer = writable.getWriter();

        async.eachLimit( all_streams_to_save, 1, ( current_stream_to_save, next) => {
       
                const zip_stream = new stream.PassThrough();
                // Add a file... can zip_stream be included here before any of the data was loaded?
               writer.write({
                      name: current_stream_to_save.path,
                      lastModified: new Date(0),
                      stream: zip_stream,
               });

                async.each( current_stream_to_save.chunks, ( chunkIndex, next2 )=>{

                    getChunk(current_stream_to_save, chunkIndex, (err, buffer) => {

                       //buffer loaded from index db and could be 32 mb... for eq
                        zip_stream.write(buffer);
                        next2();

                    });

                }, (err, out ) => {

    
                    mystream.end();

                    next();

                });
          

        }, (err) => {

            readable.pipeTo(streamer);
            writer.close();
            cb(null, { done: true } )

        } );

Here is the entire function if it helps https://github.com/Pandora-Protocol/pandora-protocol-reference/blob/d97aa8f18615f758c1a00baa80f145f1e4948368/src/pandora-locations/browser/browser-pandora-locations.js#L107

LE: we do use webpack as well. Any idea of how to fix this issue ?
I am not an expert into node.js's streams and I find them very confusing.

[ibudisteanu]

readable.pipeTo(streamer);

throws the following error

Namely,

streamer is missing _controlledWritableStream

and nothing happens.

[ibudisteanu]

@jimmywarting helped me out with great communications and support! He was able to identity the issue and refactor my code to make it working.

First issue was that we used native node buffers instead of using whatwg streams.

The 2nd issue was that we had to use web-streams-polyfill to make it work.
Pandora-Protocol/pandora-protocol-reference#6 (comment)

Here is the working code https://github.com/Pandora-Protocol/pandora-protocol-reference/blob/7f13e460d64060e7b73a06cdb40ce65a2c7b69ac/src/pandora-locations/browser/browser-pandora-locations.js#L111

Thank you so much again for taking the time to support the research code we do and for integrating conflux into our code.

[bencmbrook]

This should be resolved by #66 (orig: #53)