Stop ignoring "HTTP Error 429: Too Many Requests" bug reports [discussion]

[jonttm]

YouTube changed something yesterday that automatically blocks requests from youtube-dl depending on the situation. Read about several popular sites going down: https://streamdroid.org/youtube-blocks-popular-mp3-stream-ripping-sites/

My site was affected too and I created several new servers (all with fresh IP addresses) and all of them were blocked as soon as the first request from youtube-dl hit.

This is not just "your server was blocked by YouTube, end of discussion" issue. I can't create a new server and set up the website again.

Please look into these bug reports and discover what YouTube has done to automatically detect youtube-dl. It has to be automatic because it happens the instant you attempt to download a video from a new server.

Here's my relevant error log from my latest server instance:

jonttm@Server:/var/log/apache2$ youtube-dl -v https://www.youtube.com/watch?v=pQioCb4KUHM
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.youtube.com/watch?v=pQioCb4KUHM']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.03.14
[debug] Python version 3.6.8 (CPython) - Linux-4.15.0-52-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.6-0ubuntu0.18.04.1, ffprobe 3.4.6-0ubuntu0.18.04.1, phantomjs ., rtmpdump 2.4
[debug] Proxy map: {}
[youtube] pQioCb4KUHM: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by <HTTPError 429: 'Too Many Requests'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/lib/python3/dist-packages/youtube_dl/extractor/common.py", line 519, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 2199, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.6/urllib/request.py", line 564, in error
result = self._call_chain(*args)
File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/lib/python3.6/urllib/request.py", line 756, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/usr/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.6/urllib/request.py", line 570, in error
return self._call_chain(*args)
File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)

More information:
I later installed youtube-dl.exe on my personal Windows computer and it worked just fine. The only correlation with this bug is that all these instances were running Ubuntu Server 18.04 and using DigitalOcean as a host. I will continue testing with different hosts and other operating systems to see if that matters.

[alejandro-zapeta]

Hi i have the same issue, also put a "new issue" but i have been marked as "duplicated".
I were reading that to fix that you have to use "ipv4" (but the comments are about 2012-2014)
I put:
youtube-dl --force-ipv4 (also the same error).
I put:
youtube-dl --force-ipv6 (and says these: ERROR: Unable to download webpage: <urlopen error [Errno 99] Cannot assign requested address> (caused by URLError(error(99, 'Cannot assign requested address'),)) )

Could someone help us?

[dstftw]

Again: Youtube ban entire IPv6 blocks. You don't even need to ever run youtube-dl to get your "fresh" IP banned. It does not detect youtube-dl, if you run browser you'll get the same 429.

[alejandro-zapeta]

Ok, i don't have clear the solution. Please help me.

[jonttm]

@dstftw I appreciate the mention of "IPv6." I wasn't aware of this being the cause of the problem before. Specifying --force-ipv4 in the command has fixed the problem. I wonder where they got these blocks of addresses? Now I will just have to apply this fix to my website.

[alejandro-zapeta]

Really? i cant fix my problem :( still the same error. I will try to connect via VCN and open a webbrowser hoping appear the "are you a robot" captcha... :(

[jonttm]

Maybe try disabling all IPv6 connectivity to your device?

[alejandro-zapeta]

How can i do that? (i am using ubuntu btw)

[jonttm]

Google it?

…

On Wed, Jul 10, 2019 at 8:39 PM Juan ***@***.***> wrote: How can i do that? (i am using ubuntu btw) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#21729?email_source=notifications&email_token=ADSL3QYSWQITTFGLCFDUOQ3P63IQXA5CNFSM4IAU43XKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZVVLTA#issuecomment-510350796>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADSL3Q7LESDG6SC5UAWN5QDP63IQXANCNFSM4IAU43XA> .

[alejandro-zapeta]

HAHA ok thnks

[Hrxn]

This was the piece of critical information missing from most (if not all) "bug" reports:

all these instances were running Ubuntu Server 18.04 and using DigitalOcean as a host

[LyoSU]

Again: Youtube ban entire IPv6 blocks. You don't even need to ever run youtube-dl to get your "fresh" IP banned. It does not detect youtube-dl, if you run browser you'll get the same 429.

This is not a solution. I have ipv6 disabled on my server, but I still get this error. It appeared relatively recently. There are options?

[dstftw]

There is no special black magic. You either use not banned outgoing IP (be it your direct network interface, proxy, VPN or whatever) or it will not work.

[alejandro-zapeta]

Hi @LyoSU Have you resolved? Please help. I don't understand how to solve it.

[LyoSU]

Hi @LyoSU Have you resolved? Please help. I don't understand how to solve it.

I tried to use a proxy and it helped me. Looks like you just need to change the server.

[alejandro-zapeta]

@LyoSU THanks for your replied. Ok i will try something like Proxy or change server.

[alejandro-zapeta]

Hi! if someone needs. My Problem has been "fixed". I only stop to do "requests" to youtube with YouTube-dl, perhaps for 10 to 16 hours. And now it's working fine.
:)
Have a nice day!

[LyoSU]

@zapetin52 This is not a fix. I have the same and after a couple of dozen requests are blocked again.

[alejandro-zapeta]

Ok i unterstand it's not a fix, but for me it's working again. If youtube will block me in the future i will use another IP or a Proxy. I think i will change the IP ( i don't know )

[sherlock-tez]

Here is my tries:

1. Sleep my server for 24 hours, then come back to use youtube-dl. After 1 hour using, the error occurs again.
2. Set delay time between request is from 60 to 100 seconds. After 2 hours, the error occurs again.

YouTube changed something yesterday that automatically blocks requests from youtube-dl depending on the situation. -> I agree.

[alejandro-zapeta]

My "solution" was paid for another IP. Hoping working for a long long time. Thanks for the support