YouTube Livestream Detection

[ben-milanko]

I couldn't get a workable YouTube live stream URL with youtube-dl -g <streamUrl> so I made some minor changes to allow people to provide a standard YouTube URL into the detect function. This works with streams and regular vids.

Example:

python detect.py --device cpu --weights ./weights/yolov5m.pt --source https://www.youtube.com/watch?v=1EiC9bvVGnk

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Updated YOLOv5 to support YouTube video sources, improved stream stability, and enhanced requirement checks.

📊 Key Changes

• 📹 Added support for YouTube video URLs in stream sources.
• 🛠 Modified streaming video capture to use YouTube-dl and Pafy for YouTube sources.
• 📈 Introduced dynamic frame sleep time based on video FPS for smoother streaming.
• ⚙️ Refined the requirements check to handle a list of packages or a requirements file.

🎯 Purpose & Impact

• 🌍 Widens the range of usable video sources for inference, making it more accessible to users.
• 🚀 Enhances user experience with more stable and smooth video stream processing.
• ✅ Helps users ensure they have the necessary dependencies installed for new features and smooth operation.

[github-actions]

👋 Hello @ben-milanko, thank you for submitting a 🚀 PR! To allow your work to be integrated as seamlessly as possible, we advise you to:

• ✅ Verify your PR is up-to-date with origin/master. If your PR is behind origin/master an automatic GitHub actions rebase may be attempted by including the /rebase command in a comment body, or by running the following code, replacing 'feature' with the name of your local branch:

git remote add upstream https://github.com/ultralytics/yolov5.git
git fetch upstream
git checkout feature  # <----- replace 'feature' with local branch name
git rebase upstream/master
git push -u origin -f

• ✅ Verify all Continuous Integration (CI) checks are passing.
• ✅ Reduce changes to the absolute minimum required for your bug fix or feature addition. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." -Bruce Lee

[glenn-jocher]

@ben-milanko thanks for the PR! The idea seems great, though it looks like CI checks on the PR are failing due to your changes. Can you review the CI results and update your code to ensure all checks are passing please?

[ben-milanko]

@glenn-jocher thank you! It looks like the tests are failing because the python module 'pafy' isn't included in the CI requirements. I don't think I have access to modify the actions.

Link to the package page
https://pypi.org/project/pafy/

[glenn-jocher]

@ben-milanko ah I see. I looked over the code, it looks like it introduces a new dependency pafy. We typically try to avoid introducing specialized dependencies outside of the core packages need for basic ops. Is there a way to do this with the existing media loaders (cv2 and PIL)?

If not we might want to put some logic into detect.py that attempts a pafy import on youtube source, with an autoinstall if import fails probably using check_requirements() function with a package list passed instead of a requirements.txt file (function might need updating for this capability). Does this make sense?

We can also add pafy to the requirements.txt extras section, but commented as we don't want to force all users to install it.

[ben-milanko]

@glenn-jocher I've updated the the check requirements function to accept an include tuple which will install along with the regular text file if provided. Seems to pass the tests now.

[glenn-jocher]

@ben-milanko thanks for the updates! This looks a little better. I'll try to clean this up and merge today.

[glenn-jocher]

@ben-milanko when I tried to run your example line above (after enclosing the URL in quotes) I ran intot his error:

Traceback (most recent call last):
  File "detect.py", line 178, in <module>
    detect()
  File "detect.py", line 51, in detect
    dataset = LoadStreams(source, img_size=imgsz, stride=stride)
  File "/Users/glennjocher/PycharmProjects/yolov5_youtube/utils/datasets.py", line 281, in __init__
    import pafy
  File "/Users/glennjocher/PycharmProjects/yolov5_youtube/venv/lib/python3.8/site-packages/pafy/__init__.py", line 7, in <module>
    from .pafy import new
  File "/Users/glennjocher/PycharmProjects/yolov5_youtube/venv/lib/python3.8/site-packages/pafy/pafy.py", line 51, in <module>
    raise ImportError(
ImportError: pafy: youtube-dl not found; you can use the internal backend by setting the environmental variable PAFY_BACKEND to "internal". It is not enabled by default because it is not as well maintained as the youtube-dl backend.
(venv) (base) glennjocher@Glenns-iMac yolov5_youtube % 

Is this a second dependency we should add to the list?

[ben-milanko]

@glenn-jocher youtube-dl also has to be present on the machine to run, I've included it in the checkRequirements call

[glenn-jocher]

@ben-milanko I'm testing it out and it looks good, except for some reason videos stream super quickly, i.e.:

python detect.py --source 'https://www.youtube.com/watch?v=VMp55KH_3wo'

I think this might have to do with skipping frames to avoid building up queues in this line, but even when I set this line to n == 1 everything still streams too fast.

yolov5/utils/datasets.py

Line 302 in 6dd1083

if n == 4: # read every 4th frame

[glenn-jocher]

@ben-milanko it also looks like the loop never ends. I tested a video of a ten second clock and the loop is still going minutes later:

python detect.py --source 'https://www.youtube.com/watch?v=PPu1ekDSKw8' --img 320

I think the cv2 cap object created by the pafy package is unable to determine the endpoint of the video.

[glenn-jocher]

@ben-milanko ok I've cleaned up the check_requirements update to minimize the modifications and to allow for passing of package lists in addition to text files. Everything looks good on my end, though it would be great if you could provide any insight into the two issues: 1) framerate too fast, 2) loop never ends.

[ben-milanko]

@glenn-jocher I've managed to fix the framerate issue, the thread needs to wait according to the framerate to grab the next frame.

yolov5/utils/datasets.py

Line 315 in 3d57d56

time.sleep(1/self.fps) # wait time

The stream ending issue looks trickier, we should be able to raise StopIteration like in other classes when frames can't be grabbed

yolov5/utils/datasets.py

Lines 309 to 312 in 3d57d56

	success, im = cap.retrieve()
	if not success:
	cap.release()
	raise StopIteration # This does not correctly throw in a thread

But because the exception is thrown from a thread the main program can't handle it. Could potentially do something such as below but that might add a fair bit more complexity.
https://stackoverflow.com/questions/2829329/catch-a-threads-exception-in-the-caller-thread-in-python/31614591

[glenn-jocher]

@ben-milanko thanks, the fix looks much better! I've updated the Colab notebook separately just now and the README in this PR to display the new inference option. The only outstanding item is the loop termination. If you find a resolution for this in the future please submit another PR with a fix.

I think this is PR is good merge on our side now. If you are all done updating let me know so I can get it done. Thanks!

[glenn-jocher]

@ben-milanko I've removed the cap retrieve failure code you put in, as the current failure mode is to pass a black image. The assumption is failure may be due to a temporary network issue etc. for RTSP streaming. The cap.isOpened() boolean should switch to false at the end of the file in theory without any extra code (this is how local video files work).

Merging this PR now. Thank you for your contributions!

[hixio-mh]

Pull request and change #2752

[Haebuk]

@ben-milanko I appreciate your PR, I'm trying to inference via Youtube html like python detect.py --weights yolov5s.pt --source https://www.youtube.com/watch?v=Q19R4Nppqo0 , works fine.
But I think the displayed video is quite slow.. Is there way to decrease video latency?

[glenn-jocher]

@Haebuk maybe inference is a bottleneck, so try a smaller model or a smaller --img size?

python detect.py --weights yolov5n.pt --img 320

[Haebuk]

@glenn-jocher thank you for answering. I tried several experiments and the results are below:

youtube link: https://www.youtube.com/watch?v=uyB_ZYin4ew

- yolov5n cuda
Speed: 4.4ms pre-process, 7.8ms inference, 0.1ms NMS per image at shape (1, 3, 640, 640)
- yolov5n cpu
Speed: 0.2ms pre-process, 15.2ms inference, 0.3ms NMS per image at shape (1, 3, 640, 640)
- yolov5n-fp16
Speed: 0.2ms pre-process, 35.5ms inference, 0.4ms NMS per image at shape (1, 3, 640, 640)
- yolov5n --img 320
Speed: 0.1ms pre-process, 4.2ms inference, 0.4ms NMS per image at shape (1, 3, 320, 320)

- yolov5s cuda
Speed: 0.2ms pre-process, 4.4ms inference, 0.5ms NMS per image at shape (1, 3, 640, 640)
- yolov5s cpu
Speed: 0.2ms pre-process, 28.8ms inference, 0.3ms NMS per image at shape (1, 3, 640, 640)
- yolov5s-fp16
Speed: 0.2ms pre-process, 76.9ms inference, 0.5ms NMS per image at shape (1, 3, 640, 640)
- yolov5s --img 320
Speed: 0.1ms pre-process, 4.5ms inference, 0.4ms NMS per image at shape (1, 3, 320, 320)

I had a few questions the results.

1. Is T(ms) inference means 1000/T fps? (ex. 5 ms inference = 200FPS I'm not sure :( )
2. I think that quantization model(tflite fp-16) should be perform better the cuda model, but it's not. why ?
3. Is there better way to check model performance?

Thank you :)

[glenn-jocher]

@Haebuk FPS definition is up to you, the times per image displayed above per image are accurate. Results will vary by hardware, software and drivers. README speeds are reproducible using

python val.py --data coco.yaml --img 640 --task speed

[Haebuk]

@glenn-jocher Thank you so much 💯