Skip to content
This repository has been archived by the owner on Aug 10, 2021. It is now read-only.

Downloader will not create multiple connections to download multiple files concurrency #1

Open
ComfyFluffy opened this issue Aug 18, 2020 · 1 comment
Labels
bug Something isn't working

Comments

@ComfyFluffy
Copy link
Owner

ComfyFluffy commented Aug 18, 2020

While downloading multiple *downloader.Task from one website(e.g. i.pximg.net) using *donwloader.Downloader with multi workers, only one connection is transferring data.

By setting the proxy with Proxifier we can see the connections the downloader made. If we set the number of workers to more than 1 (e.g. 5), we can see:

  1. After the tasks being added, the downloader at first made 5 connections to i.pximg.net
  2. 4 connections closed, only one connection left.
  3. Pausing the debugging, 4 workers were blocked at Client.Do(req)

Proxifier Log

[08.19 00:54:38] bowerbird.exe *64 - i.pximg.net:443 open through proxy 127.0.0.1:8888 HTTPS
[08.19 00:54:38] bowerbird.exe *64 - i.pximg.net:443 open through proxy 127.0.0.1:8888 HTTPS
[08.19 00:54:38] bowerbird.exe *64 - i.pximg.net:443 open through proxy 127.0.0.1:8888 HTTPS
[08.19 00:54:38] bowerbird.exe *64 - i.pximg.net:443 open through proxy 127.0.0.1:8888 HTTPS
[08.19 00:54:38] bowerbird.exe *64 - i.pximg.net:443 open through proxy 127.0.0.1:8888 HTTPS
[08.19 00:54:40] bowerbird.exe *64 - i.pximg.net:443 close, 438 bytes sent, 4860 bytes (4.74 KB) received, lifetime 00:02
[08.19 00:54:40] bowerbird.exe *64 - i.pximg.net:443 close, 438 bytes sent, 4860 bytes (4.74 KB) received, lifetime 00:02
[08.19 00:54:40] bowerbird.exe *64 - i.pximg.net:443 close, 438 bytes sent, 4860 bytes (4.74 KB) received, lifetime 00:02
[08.19 00:54:41] bowerbird.exe *64 - i.pximg.net:443 close, 438 bytes sent, 4860 bytes (4.74 KB) received, lifetime 00:03
...
[08.19 00:54:52] bowerbird.exe *64 - i.pximg.net:443 close, 15000 bytes (14.6 KB) sent, 6131444 bytes (5.84 MB) received, lifetime 00:14

Is there anyway to let the *http.Client create 5 connections and use them to transfer data at same time?

@ComfyFluffy ComfyFluffy added the bug Something isn't working label Aug 18, 2020
@ComfyFluffy
Copy link
Owner Author

ComfyFluffy commented Aug 24, 2020

This is because of HTTP/2 multiplexing. Can be fixed by using different client for each worker.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant