100GB file from google drive

[mifi]

Now that tusd.tusdemo.net and companion.uppy.io are running the latest and greatest versions that support 100gb and #3159 merged, I did a test with a 100GB file on Google Drive.

It went quite slow, but it did progress steadily. After 27.7% (27GB) completion it failed with an error ECONNRESET which had a message of simply "aborted". (after 2084 sec / 34.7 min).

Logs from companion:

2021-11-03T10:39:34.022844+00:00 heroku[router]: at=info method=POST path="/drive/get/1DqPWaXRA-A2ILYBxCX6NjhZeh2F-go4u" host=companion.uppy.io request_id=12b45127-acd6-4e80-8548-3e339acbf259 fwd="49.228.236.202" dyno=web.1 connect=0ms service=675ms status=200 bytes=705 protocol=https
2021-11-03T10:39:34.023620+00:00 app[web.1]: ::ffff:10.1.80.160 - - [03/Nov/2021:10:39:34 +0000] "POST /drive/get/1DqPWaXRA-A2ILYBxCX6NjhZeh2F-go4u HTTP/1.1" 200 48 "https://uppy.io/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"
2021-11-03T10:39:34.869692+00:00 app[web.1]: companion: 2021-11-03T10:39:34.869Z [info] socket.connect connection received from ae5b8dfd-38dc-4928-9500-fb7255909bee

...

2021-11-03T11:14:18.995061+00:00 app[web.1]: companion: 2021-11-03T11:14:18.994Z [error] ae5b8dfd uploader.error Error: aborted
2021-11-03T11:14:19.271606+00:00 heroku[router]: at=info method=GET path="/api/ae5b8dfd-38dc-4928-9500-fb7255909bee" host=companion.uppy.io request_id=90ea04e5-5a24-471c-8f0a-a8d3a4b42683 fwd="49.228.236.202" dyno=web.1 connect=0ms service=2084401ms status=101 bytes=129 protocol=https

Seen in the uppy client:

{"action":"progress","payload":{"progress":"27.69","bytesUploaded":27417221257,"bytesTotal":99000000000}}
{"action":"progress","payload":{"progress":"27.70","bytesUploaded":27420334217,"bytesTotal":99000000000}}
{"action":"error","payload":{"error":{"code":"ECONNRESET","name":"Error","message":"aborted"}}}

So I think this shows that companion can handle big files now, as it did report progress all the way until the error occurred, and it didn't consume a lot of cpu, memory or disk space. But there is something that was causing the connection to fail after a while. I suspect one of:

• Heroku firewall terminating connection after staying up for too long (I found this and this indicating the "aborted" message is caused by heroku)
• tusd terminating connection after staying up for too long
• Google Drive HTTP terminated after staying up for too long
• Network hiccup anywhere at either google, tus, heroku or the interwebs

Slow transfer

Slow speeds will overall increase the likelyhood of getting a network hiccup or timeout happen. As for why the download/upload is so slow, it might be heroku limiting speed. Avg speed is about 12MByte/s, which is not crazy fast for cloud-to-cloud. I tried uppy with some URLs instead of google drive and I got similar speeds. Need to do some more research or maybe try to run this in AWS or with higher speed heroku (if possible)

[mifi]

I did another couple of tests of a 100gb file and it seems uppy/companion now handles it nicely, so I'll close this.