support bigger files

[dannycoates]

Our current implementation doesn't work for me on files > 2GB. It tries to load the whole file into the browser before encrypting. We'll probably need to do some kind of chunking. There's some prior art for webrtc that we might be able to get inspiration from: https://github.com/otalk/filetransfer/blob/master/filetransfer.js

[marco-c]

readAsArrayBuffer reads everything in memory, you can use Blob.slice() to read the file in chunks and encrypt it progressively.

Also, you probably want to encrypt it in a worker.

[ehuggett]

encrypt it progressively

I'm now doing this in python(send-cli, not pushed the changes yet), memory usage is now irrelevant even with 2Gb files, but I don't think its possible with the webCryptoAPI? (I'm also using temp files, also unavailable in javascript?)

There are some javascript crypto libraries which seem to support it, but it would seem a shame to have to use them for this (presumably the webCryptoAPI was intended to make them unnecessary?)

I believe at the moment we would have to encrypt each chunk separately, something like #174. There is an open issue for progressive encryption on the standard, perhaps we should chime in? (I'm happy to do so, but I don't feel its my place to be presuming this is a issue for Send)

Bug 27755 - Using the Subtle Crypto Interface with Streams

I think this could be combined a websocket (or perhaps chunked encoding with post?) to allow encrypted data to be streamed to the server as we process each chunk, which would be a lot simpler than RFC 8188 (but wouldn't give us the same advantages)

Another problem is I don't think we should be outputting decrypted data to the user before we have verified it. Currently this is done with the gcm-tag, and its not possible to determine the result before the last encrypted chunk has been received (the api may place further restrictions on us)

• I currently cheat in send-cli, decrypted data is written out to a temp file as the encrypted data is loaded in, but its not moved to the expected location if verification fails after this
• Assuming no temp files It would be awkward, but on very limited devices perhaps we should offer them the option of downloading the encrypted data directly (with the decryption key & metadata added into the file name/header or something by the client), which would require them to open the file again with send (now or in the future) to actually decrypt it (and potentially try again from a more capable device if it still fails, by copying the encrypted downloaded file onto it).
• chunking the data as per RFC 8188 would allow us to check the tag of every chunk as we process it (before writing it out to the file), which avoids outputing unverified data and the need for a temp file. I'm assuming we can append to the file we are "outputting" to (after the user has been asked to "save" it) but i don't actually know.

[dannycoates]

closing as done in vnext