Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node fails to download entire (large) file via http/https when agent: false is used. Fails on 0.10, 0.12, 4.1.1. #3055

Closed
trentm opened this issue Sep 24, 2015 · 5 comments
Labels
http Issues or PRs related to the http subsystem.

Comments

@trentm
Copy link
Contributor

trentm commented Sep 24, 2015

When attempting to download a (largish) file via https.request and using agent: false, the end of the file will frequently be missing.

https://gist.github.com/trentm/80409ded30b4e0418e9c has a demo script and sample runs showing it failing on node 0.10.40 and node 4.1.1. In my experience I get a failure between 10% - 80% of the time.
There is a sample run with agent: false commented out which shows successful downloads a dozen or more times in a row.

My sample script just does manual writing of 'data' chunks from the res (response) to a file (fs.createWriteStream): https://gist.github.com/trentm/80409ded30b4e0418e9c#file-download-sans-agent-js-L83-L86
This was me trying to eliminate .pipe() usage as a possible cause.

https://gist.github.com/trentm/5d3c4d9dbcb7c3b8dbaa is an equivalent script (and sample runs) using res.pipe(outStream). It fails frequently as well.

trentm added a commit to TritonDataCenter/node-docker-registry-client that referenced this issue Sep 24, 2015
@mscdex mscdex added the http Issues or PRs related to the http subsystem. label Sep 24, 2015
@mscdex
Copy link
Contributor

mscdex commented Sep 24, 2015

I'm not seeing what you're describing. I've ran the non-pipe and pipe versions of your script against the master branch with the same url for ~20 mins (with many, many requests during that time period) and did not see any errors.

@trentm
Copy link
Contributor Author

trentm commented Sep 24, 2015

@mscdex Thanks. You are right. It appears that we (at my office) can only repro this when running over our office network. Yuck. FWIW, I cannot repro when doing (at least as close as I can get) similar with curl:

$ while true; do echo ""; curl -sSf https://nodejs.org/dist/v4.1.1/node-v4.1.1-darwin-x64.tar.gz -o out.file --no-keepalive; \ls -l out.file; done

-rw-r--r--  1 trentm  staff  9544550 Sep 24 15:05 out.file

-rw-r--r--  1 trentm  staff  9544550 Sep 24 15:05 out.file

-rw-r--r--  1 trentm  staff  9544550 Sep 24 15:05 out.file
...

That makes me think it is a node issue, but something to do with a funky interaction with my office network. I'll follow up if I can find something to go on.

@misterdjules
Copy link

Possible duplicate of nodejs/node-v0.x-archive#8488.

@jasnell
Copy link
Member

jasnell commented Apr 9, 2016

is this still an issue?

@jasnell
Copy link
Member

jasnell commented Jun 7, 2016

Closing as there does not appear to be anything further to do. Can reopen if necessary

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
http Issues or PRs related to the http subsystem.
Projects
None yet
Development

No branches or pull requests

4 participants