Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Youtube video download slow #30583

Open
6 tasks
TagAcheron opened this issue Feb 1, 2022 · 29 comments · Fixed by #30582
Open
6 tasks

Youtube video download slow #30583

TagAcheron opened this issue Feb 1, 2022 · 29 comments · Fixed by #30582
Labels
broken-IE problem with existing site extraction fixed

Comments

@TagAcheron
Copy link

Checklist

  • I'm reporting a broken site support issue
  • I've verified that I'm running youtube-dl version 2021.12.17
  • I've checked that all provided URLs are alive and playable in a browser
  • I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • I've searched the bugtracker for similar bug reports including closed ones
  • I've read bugs section in FAQ

Verbose log

T:\Téléchargements\VIDEO_DOWNLOAD>"C:\SCRIPT\YoutubeDL\youtube-dlg\youtube-dl.exe" --verbose --no-playlist -f "best" https://www.youtube.com/watch?v=OylGu7Q2sA0
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '--no-playlist', '-f', 'best', 'https://www.youtube.com/watch?v=OylGu7Q2sA0']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg N-105349-g0a83ecbf48-20220124, ffprobe N-105349-g0a83ecbf48-20220124
[debug] Proxy map: {}
[youtube] OylGu7Q2sA0: Downloading webpage
[youtube] Downloading just video OylGu7Q2sA0 because of --no-playlist
[debug] Invoking downloader on 'https://rr5---sn-25glenez.googlevideo.com/videoplayback?expire=1643756761&ei=eWj5YdyLDJb7xN8P5qSXiA4&ip=90.46.239.137&id=o-ADonWinaIy1ch9uQOkTf0sWb4tx8VgPtn1XKyxexN9Ug&itag=22&source=youtube&requiressl=yes&mh=E2&mm=31%2C29&mn=sn-25glenez%2Csn-25ge7nsk&ms=au%2Crdu&mv=m&mvi=5&pl=17&initcwndbps=1228750&spc=4ocVC2oXTGupjIcIpOuGHDDKbpE_&vprv=1&mime=video%2Fmp4&ns=65YlCMrZ_X4ENWXe4lCtL8UG&cnr=14&ratebypass=yes&dur=336.387&lmt=1643435122563504&mt=1643734872&fvip=5&fexp=24001373%2C24007246&c=WEB&txp=5532434&n=RZidR5G2J4lUeu3OpY&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Cmime%2Cns%2Ccnr%2Cratebypass%2Cdur%2Clmt&sig=AOq0QJ8wRAIgD6Py9YRU8qyhdo1D1TRjwGv6wNjUMzZZj-Lt9TIMsGACIBp4cOeaevQXtUZyHOknna0EmFrsL6lwy3EvTyi-7hX0&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAMlkpNab4w3FSwBV1KGq1RLwTjY8EcOMkbnZLhD0xVjqAiEAxo50AWU5lhM1rsL99UNiZlFi0R25lCMSW7SJUqKbbxQ%3D'
[download] Resuming download at byte 1167647
[download] Destination: Tech We're Excited For! - 2022 Edition-OylGu7Q2sA0.mp4
[download]  12.3% of 18.56MiB at 24.60KiB/s ETA 11:17

Description

I can't understand why my youtube download is so slow, can someone explain me what i'am missing here because my internet is like 45 Mbps

@dirkf
Copy link
Contributor

dirkf commented Feb 1, 2022

YT made the client solve a puzzle in order to get full download speed. We can do that now (again, after YT broke it last night) and an imminent release will fix it. Or you can install from the git HEAD using pip, but as you're using the Windows .exe probably you should wait for the release.

@dirkf dirkf added the fixed label Feb 2, 2022
@sluxzer
Copy link

sluxzer commented Feb 3, 2022

This issue also happened to me.

using the last update, the download stuck at 74.93KiB/s

Good luck team.

@ronyAhmed1200
Copy link

Yeah I was also getting this issue.. Previously it was running fine, now something wrong may be happened with youtube.
Then I updated it to the latest version, but no improvement.

image

@noembryo
Copy link

noembryo commented Feb 4, 2022

Using the "fixed" version of the youtube.py, works now, but there is a problem.
Too much processing power is needed to get the link's streams.
Trying to load a playlist of tracks (not a single playlist link) takes a lot of time (50 tracks ~ 2.5 minutes on a i7)!
I know that it must compute the puzzle, but isn't it possible to cash the first computation result somewhere?

@dirkf Should i open a different issue with this?

@dirkf
Copy link
Contributor

dirkf commented Feb 4, 2022

The results are supposed to be cached. In my original version, I had a local per-item memo so the descramble was only done once. This version is meant to use the player cache as in yt-dlp.

You can see that it does so with -v because the entry like this [debug] [youtube] Decrypted nsig jCeO6nveAEs8dCNvd => Syl79sOxyyy2rQ logs when the actual computation is done, and any other URLs with the n value 'jCeO6nveAEs8dCNvd' are just being looked up.

I've only tested it on a scrap laptop (OK, I upgraded the CPU to T7700 2.4GHz) and I didn't see anything unusual. Equally I didn't timeit this implementation against the VLC regex version.

Would you be able to try your playlist against yt-dlp as well, or reveal the URL anyway?

@noembryo
Copy link

noembryo commented Feb 4, 2022

I use it like a library in an app, so I can't use -v.
Other than that, I also use a different thread for every track of the playlist, so maybe the cache strategy does not work here?
Any playlist have the same (analogous) delay..
If this has to do with cashing the player, then, I might use the same code strategy I send you the last time there where problems.
I'll try that and post the update here..

UPDATE: I couldn't make the older cashing code to work with the current changes.
Also, checking yt-dlp 2021.12.27 with the same links in the same app works OK.

@dirkf
Copy link
Contributor

dirkf commented Feb 4, 2022

The code uses the IV _player_cache which is a dict. I would be happy to consider some patch for a potentially thread-safe global cache. But this SO implies that dict operations are atomic. As the cache usage is either to read a value by key or to insert a value for a key that is always the same, it appears that a module-level cache could work effectively without any locking.

S/t like this:

...
_PLAYER_CACHE = {}
...
class YoutubeIE(YoutubeBaseInfoExtractor):
...
    def __init__(self, *args, **kwargs):
        super(YoutubeIE, self).__init__(*args, **kwargs)
        self._code_cache = {}
        self._player_cache = _PLAYER_CACHE
...

Here I'm assuming you have a different YoutubeIE instance in each thread, but that the module-level variable is shared across threads.

@noembryo
Copy link

noembryo commented Feb 4, 2022

That was a nice idea, but unfortunately it produces an error:

ERROR: Signature extraction failed: Traceback (most recent call last):
  File "...\PYTHON\Python27\lib\site-packages\youtube_dl\extractor\youtube.py", line 1445, in _decrypt_signature
    return func(s)
  File "...\PYTHON\Python27\lib\site-packages\youtube_dl\extractor\youtube.py", line 1427, in <lambda>
    return lambda s: initial_function([s])
  File "...\PYTHON\Python27\lib\site-packages\youtube_dl\jsinterp.py", line 554, in resf
    ret, should_abort = self.interpret_statement(stmt, var_stack)
  File "...\PYTHON\Python27\lib\site-packages\youtube_dl\jsinterp.py", line 157, in interpret_statement
    v = self.interpret_expression(expr, local_vars, allow_recursion)
  File "...\PYTHON\Python27\lib\site-packages\youtube_dl\jsinterp.py", line 470, in interpret_expression
    return eval_method()
  File "...\PYTHON\Python27\lib\site-packages\youtube_dl\jsinterp.py", line 463, in eval_method
    return obj[member](argvals)
  File "...\PYTHON\Python27\lib\site-packages\youtube_dl\jsinterp.py", line 554, in resf
    ret, should_abort = self.interpret_statement(stmt, var_stack)
  File "...\PYTHON\Python27\lib\site-packages\youtube_dl\jsinterp.py", line 157, in interpret_statement
    v = self.interpret_expression(expr, local_vars, allow_recursion)
  File "...\PYTHON\Python27\lib\site-packages\youtube_dl\jsinterp.py", line 470, in interpret_expression
    return eval_method()
  File "...\PYTHON\Python27\lib\site-packages\youtube_dl\jsinterp.py", line 418, in eval_method
    assertion(isinstance(obj, list), 'must be applied on a list')
  File "...\PYTHON\Python27\lib\site-packages\youtube_dl\jsinterp.py", line 370, in assertion
    raise ExtractorError('{0} {1}: {2}'.format(nl.member, msg, expr))

@shellig
Copy link

shellig commented Feb 6, 2022

This says fixed but I am getting 526 B/s downloads - slower than if I watched the video! Realplayer download works at 10 MiS but not what I want. I tried with both youtube-dl and yt-dlp both running that slow.

@dirkf
Copy link
Contributor

dirkf commented Feb 6, 2022

I tried with both youtube-dl and yt-dlp both running that slow.

The master version in the yt-dl repo is fixed. The fix is not in a released version yet.

yt-dlp release 2022.02.04 should run unthrottled.

@dirkf
Copy link
Contributor

dirkf commented Feb 6, 2022

That was a nice idea, but unfortunately it produces an error:

If you can make a small program that reproduces the problem, we might be able to make progress.

@noembryo
Copy link

noembryo commented Feb 6, 2022

If you can make a small program that reproduces the problem, we might be able to make progress.

Unfortunately, I use QThreads in a complicated manner in my app, but I will try to make something minimal with the native Treads, for a simple scripts.

Any already existing, simple threading example, will be greatly appreciated..

P.S. The updated extractor has the same results.. 😢

@dirkf dirkf added the broken-IE problem with existing site extraction label Feb 6, 2022
@dirkf dirkf linked a pull request Feb 6, 2022 that will close this issue
11 tasks
@noembryo
Copy link

noembryo commented Feb 7, 2022

Sorry for the delay, but I was busy yesterday..
Here is some example code:

# coding=utf-8
from __future__ import absolute_import, division, print_function, unicode_literals
from queue import Queue
from threading import Thread, Event
import youtube_dl
# import yt_dlp as youtube_dl

ydl_opts = {'quiet': True, 'no_warnings': True}

link_ids = ["4jduuQh-Uho", "9GNpv7QDvMY", "MbEOR2Flc-4", "ZKUzNF21n9w", "y-JqH1M4Ya8",
            "c5pi-SOlXQc", "pUqfaiUb3l4", "bL5eqSOXMtE", "HyMm4rJemtI", "BU4kGkrrJEw",
            "pFS4zYWxzNA", "aF6hDcAbSoE", "G1ckKDRc69w", "o9_jzBtdMZ4", "AGoQZx8Mn0g",
            "6W-pHCD6Tow", "kszLwBaC4Sw", "mwTd_PzGY-c", "iqLTYD_nhsU", "X335gdcPE7A",
            "z_54vDk8lWw", "8a82arE0JSQ", "tJmzQHWl9kc", "8jPQjjsBbIc", "ENJUB5thpB4",
            "dEhUMvjFuQY", "D6XyJh1tsGI", "tFCfb-Qqdz0", "UkafA6r1caQ", "OO8HtAXnRqQ",
            "--da0m2K4I4", "EOlI0UtLDk4", "r7tQbxTImKw", "s_YLPcW4Tu8", "9wIbhES2UkA",
            "YkX9X4td7j8", "14cHz4ebonY", "saVUUZE50Co", "N1K4NYHqMx4", "iCBL33NKvPA",
            "QPTNS3llm2E", "pFS4zYWxzNA", "wA1v207xlOw"]
infos = []


class Base(object):
    def __init__(self, **kwargs):
        super(Base, self).__init__(**kwargs)
        self.feed_q = Queue()
        self.threads = []
        for i in range(8):
            thread = Worker(self.feed_q)
            thread.daemon = True
            self.threads.append(thread)
            thread.start()


class Node(object):
    pass


class Worker(Thread):
    def __init__(self, feed_q):
        super(Worker, self).__init__()
        self.node = None
        self.feed_q = feed_q
        self.stop = False

    def run(self):
        while not self.stop:
            self.node = self.feed_q.get()
            url = "https://www.youtube.com/watch?v=" + self.node.id
            with youtube_dl.YoutubeDL(ydl_opts) as ydl:
                try:
                    ydl_info = ydl.extract_info(url, download=False)
                except Exception as e:
                    error_text = "A {0} occurred for {2}. Arguments:\n{1!r}"
                    print(error_text.format(type(e).__name__, e.args, url))
                    self.feed_q.task_done()
                    continue
                infos.append(ydl_info)
                print("Got info from {}".format(url))
                self.node.updated.set()
            while True:
                if self.node.updated.wait(timeout=2):
                    break
            self.feed_q.task_done()


if __name__ == "__main__":
    base = Base()
    print("Getting info for {} YouTube videos".format(len(link_ids)))
    for tube_id in link_ids:
        node = Node()
        node.id = tube_id
        node.updated = Event()
        base.feed_q.put(node)

    from timeit import default_timer as timer
    start = timer()
    base.feed_q.join()
    print("Finished in {} seconds".format(round(timer() - start)))

youtube_dl takes 96 sec while yt-dlp takes only 20 sec.
Before the last breakage, youtube_dl was faster than yt-dlp.
There is also more processing with youtube_dl ..

The links are random, you can use whatever links you like..

@dirkf dirkf mentioned this issue Feb 8, 2022
6 tasks
@unityconstruct
Copy link

FYI: Followed advice regarding pulling from 'master'.....

  1. ran 'make'
  2. pushed newly built youtube-dl to my local server
  3. super FAST..
  4. Thanks to the team!

For those wishing to build, here's what I did ( for better or worse & whether correct or not... ):
NOTE: I previously build, so met dependencies referenced in the readme.md

MAKE SESSION

/git/youtube-dl$ make
mkdir -p zip
for d in youtube_dl youtube_dl/downloader youtube_dl/extractor youtube_dl/postprocessor ; do
mkdir -p zip/$d ;
cp -pPR $d/.py zip/$d/ ;
done
touch -t 200001010101 zip/youtube_dl/
.py zip/youtube_dl//.py
mv zip/youtube_dl/main.py zip/
cd zip ; zip -q ../youtube-dl youtube_dl/.py youtube_dl//*.py main.py
rm -rf zip
echo '#!/usr/bin/env python' > youtube-dl
cat youtube-dl.zip >> youtube-dl
rm youtube-dl.zip
chmod a+x youtube-dl
COLUMNS=80 /usr/bin/env python youtube_dl/main.py --help | /usr/bin/env python devscripts/make_readme.py
/usr/bin/env python devscripts/make_contributing.py README.md CONTRIBUTING.md
pandoc -f markdown -t plain README.md -o README.txt
/usr/bin/env python devscripts/prepare_manpage.py youtube-dl.1.temp.md
pandoc -s -f markdown -t man youtube-dl.1.temp.md -o youtube-dl.1
rm -f youtube-dl.1.temp.md
/usr/bin/env python devscripts/bash-completion.py
/usr/bin/env python devscripts/zsh-completion.py
/usr/bin/env python devscripts/fish-completion.py
/usr/bin/env python devscripts/make_supportedsites.py docs/supportedsites.md

LS YOUTUBE-DL

-rwxrwxr-x 1 god god 1849986 Feb 9 01:00 youtube-dl

FAST DL SPEED

[download] 100% of 51.86MiB in 00:04

@ronyAhmed1200
Copy link

Yeah very fast download speed !!

  1. I am using latest version of yt-dl but download speed is too slow for me..
  2. idk what's going wrong in my case.

image

@dirkf
Copy link
Contributor

dirkf commented Feb 9, 2022

If anyone has a Python installation with a recent pip (pip3), you can install the latest master branch (or any other commit level) directly from GitHub or a downloaded tar.gz.

@noembryo
Copy link

noembryo commented Feb 9, 2022

Tested the current latest master with the example above, and the time got improved from 96 to 76 secs.
But unfortunately still way over the 20 sec of yt-dlp or the even faster previous yt-dl.. 😞

@northmatt
Copy link

As a temporary fix you can use Aria2c external downloader to get split file downloading and/or batch file downloading, effectively removing throttle issues.

@noembryo
Copy link

noembryo commented Feb 10, 2022

@dirkf Should I open a different issue with the code for the multi-threading problem?
Opened #30641

@unityconstruct
Copy link

CONFIRMING FIX ON MASTER: had throttling issue on 2022.12.17... pulled master > make = resolved

[download] 43.0% of 274.52MiB at 15.43MiB/s ETA 00:10

@TagAcheron
Copy link
Author

i"m on windows, i don't understant your "fix". can you explain please :)

@noembryo

This comment was marked as resolved.

@dirkf
Copy link
Contributor

dirkf commented May 2, 2022

The fix committed to the repo requires more than one file to be updated.

@TagAcheron, #30583 (comment), or install Python and #30583 (comment).

@noembryo
Copy link

noembryo commented May 2, 2022

@dirkf Yes, I already had that update.
I was referring to the last change ~11-12 days ago..

@rlekni
Copy link

rlekni commented May 18, 2022

CONFIRMING FIX ON MASTER: had throttling issue on 2022.12.17... pulled master > make = resolved

[download] 43.0% of 274.52MiB at 15.43MiB/s ETA 00:10

Building and replacing the binaries, worked for me too

@j-rahman

This comment was marked as off-topic.

@dirkf

This comment was marked as outdated.

@bwalendz

This comment was marked as abuse.

@dirkf

This comment was marked as outdated.

@ytdl-org ytdl-org locked as resolved and limited conversation to collaborators Jul 4, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
broken-IE problem with existing site extraction fixed
Projects
None yet
Development

Successfully merging a pull request may close this issue.